[uom][math][numbers] Request for feedback on Units of Measurement library

[Thomas Neidhart]

Hi all,

recently I had more time to work on open source and I always wanted to
work on a fully typed Units of Measurements library.

My initial results can be found here:

https://github.com/netomi/uom

Its already fully functional, I am working on more unit tests,
documentation and as many units / quantities as possible.

It uses the BigFraction / Fraction class of commons-math /
commons-numbers internally (I copied the source for now as I had to make
some small modifications).

The main page of the project explain some basic uses of the library.

In general it allows to do things like that:

Length l1 = Length.of(1, Units.SI.METRE);
Length l2 = Length.of(4, Units.Imperial.YARD);

Length l3 = l1.add(l2);

System.out.println(l3.to(Units.Imperial.INCH));

It supports using quantities in double or decimal precision using the
BigDecimal class if this is needed.

I would be very grateful if somebody with an interest in this topic
could take a look and provide feedback.

Thanks & Regards,
Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collections] An order statistic tree

[Thomas Neidhart]

On Fri, Mar 18, 2016 at 2:23 PM, Rodion Efremov <
rodio...@cs.helsinki.fi.invalid> wrote:

> Hello, all.
> I would like to announce that I have a java.util.Set implementation that
> is an order statistic tree (all non-bulk operations + select + rank in
> O(log n) time) [1]. However, it seems like the team on behalf of the order
> statistic tree issue is rather idle [2], hence the message.
>

Hi Rodion,

sorry, but I stopped working on collections and it looks like nobody else
is interested either.

The datastructure seems interesting, I would propose to contribute it
either to guava or Eclipse Collections (descendent of GS collections).

Regards,

Thomas

Re: [math] Staging site

[Thomas Neidhart]

On 03/21/2016 07:32 PM, Evan Ward wrote:
> Hi,
> 
> I'm on step 19 in Release.howto.txt, I've committed the updated site
> using svn, but I get a 404 when I try to view it at
> http://commons.staging.apache.org/proper/commons-math
> 
> Any ideas what I'm doing wrong? I've noticed that the staging URL for
> other components such as [lang] or [io] also result in a 404. Has the
> URL changed?

you do not need to copy to the staging site, the howto is wrong there.

Once the commit to the site-content repository is done, the site is
immediately synced and visible at http://commons.apache.org/math. You
might need to refresh your browser though.

btw. running mvn site-deploy should do the same as described in the howto.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] questions on GA

[Thomas Neidhart]

On 03/18/2016 02:12 PM, Семенов Кирилл wrote:
>>
>> In effect, some time ago we evoked the possibility to drop GA support
>> altogether since the code seemed little used and a lot of work was
>> anticipated for making it useful beyond demo applications.
> 
> 
> It is rather surprising, that ASF doesn't have any complex and extensive AI
> instruments. I was looking for ANN some time ago and found only Mahout,
> which seemed rather idle.
> But Commons projects, IMO, are a good place for GA, while there is no
> activity on AI merging/separating.

take a loot at http://jenetics.io which contains already everything you
were asking for and is a very mature and well designed library. On top
of that it is also under the Apache license and actually uses CM for
testing.

Thomas

> but lacking human resources
>> it's unlikely to become a reality any time soon.
>>
> 
>  You can always point to lacking features by opening JIRA reports, but
>> unless
>> you intend to work on them yourself, I wouldn't bet on having them fixed
>> rapidly.
>>
> 
> If someone would want to start a large overhaul of the GA code, that is
>> worth considering.
>>
> 
> There surely can be a big scope. Btw, I'm still a student (last year of
> BS).
> So, I can do things listed and some more as a GSoC project (I was planning
> to participate anyway).
> https://docs.google.com/document/d/1RT_zNfBdf8rX2p5Qo0bIQ28SqwWUVPwfKmjStwK8SCw/edit#
> - I wrote my vision and ready to refine it,
> if you care to comment.
> 
> I find it strange, that ASF list for GSoC doesn't include any Commons
> projects or widely known (Cassandram Solr, Kafka, etc.) Is it even possible
> to participate on behalf of Commons Math this year?
> 
> Do you mean using an existing library, or do you suggest implementing the
>> functionality specifically for CM?
>>
>> I'm not sure, which way is preferred. There is Commons Pool, that can
> become a dependency, but is it okay to get any dependencies beside Junit?
> And implementing It would definitely enlarge codebase. Which is less of two
> evils?
> 
> 
> 
> 2016-03-17 21:35 GMT+03:00 Gilles :
> 
>> Hello.
>>
>> On Thu, 17 Mar 2016 19:11:16 +0300, Семенов Кирилл wrote:
>>
>>> Hi,
>>>
>>> I've been using genetic algorithm for some pet projects. And I'd like to
>>> shed some light on a number of topics.
>>>
>>
>> Thanks for you interest.
>>
>> Given that there exist Java softwares that seem to provide a more complete
>> features set, I'd be interested to know a user's opinion on how the
>> CM implementation compares with those.
>> In effect, some time ago we evoked the possibility to drop GA support
>> altogether since the code seemed little used and a lot of work was
>> anticipated for making it useful beyond demo applications.
>>
>> 1. Am I correct to think, that now GA is working in a single thread?
>>>
>>
>> Certainly.
>>
>> Very few CM codes are multi-thread ready.  It was one of the task to
>> be tackled for future versions of the library, but lacking human resources
>> it's unlikely to become a reality any time soon.
>>
>> In
>>> such case, was there any discussions on the subject (I didn't find within
>>> a
>>> quick check of Jira).
>>>
>>
>> There were discussions (cf. "dev" ML archive).
>>
>> If not, could you provide some API reference. The
>>> subject is important, because the ability to be distributed is one of the
>>> key features of the GA.
>>>
>>> 2. Was there talks about implementing Pool for chromosomes? I found
>>> enhancement proposal https://issues.apache.org/jira/browse/MATH-1219 -
>>> which is aimed to solve the same problem - creating an enormous amount of
>>> chromosomes in each generation. Chromosomes after each generation hangs in
>>> a heap waiting for GC. Also, object pool can be implemented, supposing
>>> that
>>> chromosome would consist of List.
>>>
>>
>> If someone would want to start a large overhaul of the GA code, that is
>> worth considering.
>> Do you mean using an existing library, or do you suggest implementing the
>> functionality specifically for CM?
>>
>> 3. Examples of using getRepresentation method of AbstractListChromosome
>>> seem misleading. Because getRepresentation  is protected method and
>>> writing
>>> classes that implement MutationPolicy/CrossoverPolicy can't use it. For
>>> rapid development one could implement public overriding method, but can't
>>> it be defined public in AbstractListChromosome? If one is to write some
>>> particular policy, he must override getRepresentation method in
>>> CustomChromosome. But if one wants to write some common genetic policy
>>> (e.g., some reordering crossover), he would face an obstacle mentioned.
>>>
>>> I'd like to create tasks for those in Jira. Just want to make sure, that
>>> these topics would be useful and gather some information, other
>>> devepoler's
>>> opinions on a matter.
>>>
>>
>> You can always point to lacking features by opening JIRA reports, but
>> unless
>> you intend to work on them yourself, I wouldn't bet on having them fi

Re: [all] apologies

[Thomas Neidhart]

On Mon, Feb 8, 2016 at 9:13 PM, Phil Steitz  wrote:

> I am sorry for the bad tone of my recent posts here.  Not the nicest
> way to leave and I am sorry for that.
>

Hi Phil,

nothing that you have written is worth an apology, although I can
understand your disappointment about the way (or motive) you are leaving.

In fact, as Niall pointed out, we as community have failed as we have
accepted unhealthy behavior for too long a time resulting in people just
being frustrated up to the point where they break and leave.
Sometimes it does not take a flame war but this constant little nagging
about every little thing to make people feel uncomfortable and dissapointed.

For me you have been a role model how to handle conflicts and it makes me
really sad to see to leave because of one.

Thomas

Re: [math] [POLL] new TLP name

[Thomas Neidhart]

On Thu, Feb 4, 2016 at 2:24 PM, Phil Steitz  wrote:

> On 2/2/16 11:44 AM, Thomas Neidhart wrote:
> > On 02/01/2016 06:06 PM, Phil Steitz wrote:
> >> Please select your top choice among the following suggested names
> >> for the new [math]-based TLP.  All are welcome and encouraged to
> >> respond.  This POLL will be open for 72 hours, at which time two
> >> tallies will be presented:  one among those who have volunteered for
> >> the new PMC and a second among all respondents.  Hopefully, one name
> >> will emerge as consensus winner.  If not, I will kick off another
> >> runoff poll among the top choices.   Please respond with your top
> >> choice for the name.
> >>
> >> AjaMa
> >> Epsilon
> >> Erdos
> >> Euclid
> >> Euler
> >> Gauss
> >> JAML
> >> Math
> >> MathBlocks
> >> MathComponents (or Math Components)
> >> Mathelactica
> >> MathModules
> >> Megginson
> >> modMath
> >> Nash
> >> Newton
> >> Pythagoras
> > I am split between Epsilon and Math, both seem quite appealing to me.
> > Both choices are fine for me.
>
> I need a first choice for the tally.  Can I assume that is "Epsilon?"
>

I would vote for Math as first choice.

Thomas

Re: [math] [POLL] new TLP name

[Thomas Neidhart]

On 02/01/2016 06:06 PM, Phil Steitz wrote:
> Please select your top choice among the following suggested names
> for the new [math]-based TLP.  All are welcome and encouraged to
> respond.  This POLL will be open for 72 hours, at which time two
> tallies will be presented:  one among those who have volunteered for
> the new PMC and a second among all respondents.  Hopefully, one name
> will emerge as consensus winner.  If not, I will kick off another
> runoff poll among the top choices.   Please respond with your top
> choice for the name.
> 
> AjaMa
> Epsilon
> Erdos
> Euclid
> Euler
> Gauss
> JAML
> Math
> MathBlocks
> MathComponents (or Math Components)
> Mathelactica
> MathModules
> Megginson
> modMath
> Nash
> Newton
> Pythagoras

I am split between Epsilon and Math, both seem quite appealing to me.
Both choices are fine for me.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Fwd: Re: svn commit: r12178 - in /release/commons/collections: binaries/ source/

[Thomas Neidhart]

 Forwarded Message 
Subject: Re: svn commit: r12178 - in /release/commons/collections:
binaries/ source/
Date: Tue, 2 Feb 2016 19:07:08 +0100
From: Thomas Neidhart 
To: comm...@commons.apache.org

On 02/02/2016 07:03 PM, s...@apache.org wrote:
> Author: sebb
> Date: Tue Feb  2 18:03:09 2016
> New Revision: 12178
> 
> Log:
> Archive collections release 4.1
> 
> Removed:
> release/commons/collections/binaries/commons-collections4-4.1-bin.tar.gz
> 
> release/commons/collections/binaries/commons-collections4-4.1-bin.tar.gz.asc
> 
> release/commons/collections/binaries/commons-collections4-4.1-bin.tar.gz.md5
> 
> release/commons/collections/binaries/commons-collections4-4.1-bin.tar.gz.sha1
> release/commons/collections/binaries/commons-collections4-4.1-bin.zip
> release/commons/collections/binaries/commons-collections4-4.1-bin.zip.asc
> release/commons/collections/binaries/commons-collections4-4.1-bin.zip.md5
> release/commons/collections/binaries/commons-collections4-4.1-bin.zip.sha1
> release/commons/collections/source/commons-collections4-4.1-src.tar.gz
> release/commons/collections/source/commons-collections4-4.1-src.tar.gz.asc
> release/commons/collections/source/commons-collections4-4.1-src.tar.gz.md5
> 
> release/commons/collections/source/commons-collections4-4.1-src.tar.gz.sha1
> release/commons/collections/source/commons-collections4-4.1-src.zip
> release/commons/collections/source/commons-collections4-4.1-src.zip.asc
> release/commons/collections/source/commons-collections4-4.1-src.zip.md5
> release/commons/collections/source/commons-collections4-4.1-src.zip.sha1

-1, you just removed the current release.

Thomas



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] Name of the new TLP

[Thomas Neidhart]

On 01/25/2016 09:27 PM, Gary Gregory wrote:
> On Jan 25, 2016 10:11 AM, "Emmanuel Bourg"  wrote:
>>
>> Le 25/01/2016 18:52, Gilles a écrit :
>>
>>> AFAICT, the real issue is one of policy: Commons is supposed to be
> stable,
>>> stable, stable and stable (IIUC).
>>>
>>> And CM is far from being mature as a programming project, when
> considering
>>> design and scope, and not only the quality of its results and
> performance
>>> (which are both good in many cases).
>>> So stability (as in using JDK 5 only) is not a good perspective (surely
> not
>>> developers and probably not for users either IMO).
>>>
>>> If this does not change, what's the point indeed?
>>
>> I hope that a motivation behind the TLP isn't to break the compatibility
>> on every release, otherwise this will quickly turn into a nightmare for
>> the users. Bouncycastle plays this game and it isn't really fun to follow
> :(
> 
> WRT compatibility, the only thing that matters is not creating jar hell for
> users. You can break compatibility if you change package and maven
> coordinates. It's up to the project to create enough alphas and betas to
> get to a stable public API before a release. That's just basic project
> management IMO. Anything less will leave a lot users unhappy.

What you describe is the mantra of Commons and while I perfectly agree
with it for certain wide-spread libraries like lang, collections or
logging, the same can not be applied to any other type of library in
existence.

A library like CM is much less used and the danger of creating a jar
hell because of it does not justify such a strict policy. In fact the
Commons policy is one of the reasons why the vote to move to a TLP was
started originally.

There are also other, very respected and mature libraries (like
joda-time) that allow non-compatible changes in major version without
changing package / artefact ids, and the world did not collapse because
of it.

The key point is to be reasonable about the audience of a library, and
CM does not play in the same league as lang or collections for example.
Also a better modularization of the project might help in this respect,
as certain modules might have different maturity levels and users can
expect them to not change much over time, whereas others are more likely
to change but their use is mainly in very specific applications (think
about the optimization package in CM).

We certainly need to think about and express the way we want to organize
CM as a TLP, which really needs to be different to the status quo. I
really hope that the people willing to contribute to CM as a TLP take
this into account, as otherwise there is no point in doing it as Gilles
pointed out correctly.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] Volunteer for the new TLP PMC

[Thomas Neidhart]

On Sun, Jan 24, 2016 at 9:54 PM, Phil Steitz  wrote:

> Please respond to this thread if you are a Commons Committer
> interested in joining the PMC for the new TLP based on [math].
>

Hi Phil,

thanks for pushing this whole issue!

I am happy to volunteer for the new PMC to be formed.

Thomas

Re: [Math] Revamping the "random" package or ...

[Thomas Neidhart]

On 01/17/2016 02:34 PM, Gilles wrote:
> On Sun, 17 Jan 2016 10:56:38 +0100, Luc Maisonobe wrote:
>> Le 16/01/2016 16:51, Gilles a écrit :
>>> Hi.
>>>
>>> Context: nobody gave an opinion on the arguments which I put
>>> forward in these posts:
>>>   http://markmail.org/message/uiljlf63uucnfyy2
>>>   http://markmail.org/message/ifwuijbgjytne6w2
>>>
>>> As a consequence, the lack of any development policy, rather
>>> than being the touted advantage of the "free world" of Apache,
>>> is, objectively, a quite efficient way to push in the direction
>>> of the stronger voice, not necessarily backed with the stronger
>>> arguments (especially when those are not "technical" but, in
>>> reality, "political"!).
>>> This has been the subject of another post, that also was not
>>> followed by a constructive debate in order to change this
>>> community's ways, so that it would not discourage proposals
>>> for code evolutions towards a modern use of the Java language.
>>>
>>> Thus, in this context, I obviously can't know whether "silence
>>> is consent" or if people will continue raising objections to my
>>> experimenting with the contents of the "random" package, even
>>> after not raising concern and/or not engaging in the practical
>>> discussions about the proposals.
>>>
>>> Also, it is disrespectful to let people think that they could
>>> work on some part of the library, and then voice an opinion
>>> akin to the hidden policy that there exists, in CM, codes
>>> that are deemed too sensitive to be ever touched again.
>>>
>>> My first idea was to make incremental changes in "random".
>>> The first few, and little, steps unexpectedly implied a huge
>>> amount of work, mainly due to the disproportionate
>>> justifications that were being required.
>>>
>>> It is a fact that even tiny, even no-op, changes meet
>>> infinitely more opposition than adding even very large chunks
>>> of new code.
>>>
>>> Hence, I propose that all my recent changes to the "random"
>>> package be reverted so that it will match the contents of the
>>> 3.6 release (modulo the changes which were explicitly agreed
>>> on like those in "RandomGeneratorAbstractTest").
>>
>> I did answer to at least part of your proposals, and suggested
>> this experimentation is done on a branch.
>> At the same time, you also proposed to adopt another branching
>> policy, and this was seen positively by anyone.
>>
>> So I would suggest that rather than adding a parallel rng package,
>> which reminds me of the difficulties we get with the two optim and
>> optimization packages, you continue doing your changes directly
>> in the random package as you started to do, but in a feature branch.
> 
> Sorry, but I don't agree.
> I've explained that I want to propose as a *replacement* to "random".
> Almost every file will be changed, and a basic requirement is to have
> the RNGs, and only the RNGs, in their own package/module.
> 
> So for example, "RandomDataGenerator" and "ValueServer", as "users"
> of the RNGs, should not be in the "rng" package (but but stay in
> "random" whatever else changed or delete there).
> 
> This situation here cannot be more different than for the "optim"
> package!
> First, the old "optimization" _has_ been deleted in "master"; we
> had to keep it in the 3.x line.
> The code in "optim" has been been criticized but until now nobody
> came up with a better proposal, so the only working code must
> obviously stay.

Afaicr we all agreed on going forward with the design as implemented for
the least-squares optimizer by Evan?

Thomas

> For "rng", I'll propose a working remplacement, and we'll be able
> to immediately decide whether to keep "random" as is or adapt it
> in order to remove the redundancy with the new "rng" and/or write
> some adaptation layers from "random" to "rng".
> 
> Best regards,
> Gilles
> 
>>>
>>> Is that possible?  [Luc, as the most experienced "git" user,
>>> would you mind managing this, perhaps delicate, operation?]
>>
>> Reverting is not difficult. Remember the trick discussed on
>> this list to port commits between math3 and math4? It was
>> basically doing a "git diff -p some-commit~1 some-commit",
>> then patching the commit with a sed and applying it later on.
>>
>> Here is it even simpler because we don't have to patch the commit.
>> The trick is to do the git diff the other way round, i.e.
>> "git diff -p some-commit some-commit~1".
>>
>> Also rather than reverting them and restarting again, in
>> order not to lose your work I'll cut a new feature branch
>> first, then revert on master only. You will be able to
>> continue your work on the feature branch.
>>
>> On a related subject, I also read on another list that infra
>> now allows deleting branches again. The concerns I had with
>> short-lived hotfix branches are therefore not realistic
>> anymore, we can do as many brnahces and as short-lived as we want.
>>
>>>
>>> I would then pursue my refactoring in a new package named
>>>   org.apache.commons.math4.rng
>>> whe

Re: [VOTE] Form a separate TLP based on [math]

[Thomas Neidhart]

On 01/16/2016 04:18 PM, Phil Steitz wrote:
> The discussion has thus far been generally favorable.  I would like
> therefore to put the proposal to split [math] out into a separate
> TLP to a VOTE.  Assuming a favorable vote, we can discuss how to go
> about doing it.  Votes, please.  All are welcome to vote.
> 
[X] +1 I am in favor of this action

Thomas

> [ ] +0 I am OK with this
> [ ] -0 OK, but...
> [ ] -1 I oppose this action because...
> 
> This VOTE will run a little longer than usual - closing at 20 Jan
> 13:00 UTC.
> 
> Thanks!
> 
> Phil
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] TLP

[Thomas Neidhart]

On Thu, Jan 14, 2016 at 1:50 AM, Phil Steitz  wrote:

> I would like to propose that we split [math] out into a top level
> project at the ASF.  This has been proposed before, and I have
> always come down on the side of staying in Commons, but I am now
> convinced that it is a good step for us to take for the following
> reasons:
>
> 0) We have several committers who are really only interested in
> [math], so being on the Commons PMC does not really make sense for them
> 1) The code base has swollen in size to well beyond the "small sharp
> tools" that make up the bulk of Commons
> 2) We are probably at the point where we should consider splitting
> [math] itself into separately released subcomponents (could be done
> in Commons, but starts smelling a little Jakarta-ish when Commons
> has components with subcomponents).
>
> The downsides are
> a) [newPMC] loses Commons eyeballs / contributors who would not find
> us otherwise
> b) Migration / repackaging pain
> c) Overhead of starting and managing a PMC
> d) Other Commons components lose some eyeballs
>
> Personally, I think the benefits outweigh the downsides at this
> point.  New better tools and ASF processes have made b) and c) a
> little less onerous.  I don't think d) is really a big problem for
> Commons, as those of us who work on other stuff here could continue
> to do so.  It is possible that a) actually works in the reverse
> direction - i.e., we are easier to find as a TLP.
>
> What do others think about this?
>

I think it would be the right move.

+1

Thomas

Re: [Math] Add 64-bits based RNG (Was: [Math] New base class for all RNGs)

[Thomas Neidhart]

On Mon, Jan 11, 2016 at 1:10 PM, Gilles 
wrote:

> On Mon, 11 Jan 2016 07:47:40 +0100, Thomas Neidhart wrote:
>
>> On 01/10/2016 05:09 AM, Gilles wrote:
>>
>>> Hi.
>>>
>>> Relevant excerpt of previous posts:
>>>
>>> [...]
>>>>>
>>>>> Something implicit in "BitStreamGenerator": the maximum number of
>>>>> bits is 32 (cf. return type of "next(int)" and the ubiquitous use
>>>>> of hard-coded "32".
>>>>>
>>>>> What about the possibility of using a "long" as a building block
>>>>> for another family of RNG?
>>>>>
>>>>
>>>> Why?  We don't have contributions of other RNGs implemented using
>>>> 64-bit blocks of data.  In theory, I guess you could do that, but I
>>>> don't know of any and all the ones we have use 32-bit words.
>>>>
>>>> Phil
>>>>
>>>>>
>>>>> Gilles
>>>>>
>>>>
>>> (1)
>>> CPUs and OS are nowadays commonly 64-bits based.
>>> Thus one could legitimately wonder whether, on such systems, generating
>>>   one "long" value
>>> would not be faster than generating
>>>   two "int" values,
>>> especially when 64 bits are necessary in order to produce the next random
>>> "long" or "double" value.
>>>
>>> (2)
>>> There indeed exist 64-bits implementations of RNGs:
>>>
>>>
>>>
>>> http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/VERSIONS/C-LANG/mt19937-64.c
>>>
>>>   http://burtleburtle.net/bob/rand/isaacafa.html
>>>
>>>
>>> I thus propose to create a feature branch that will contain
>>> * a new interface, for when "nextLong()" is the "random bits" producer
>>>
>>
>> be aware that (at least some of) the 64-bit variants rely on unsigned
>> long arithmetic operations which can't be done in java without losing a
>> *lot* performance.
>>
>
> What do you mean?
>
> Wasn't there the same problem when implementing RNGs that rely on
> unsigned int arithmetic, with Java's signed int?
>

in which case you can perform the operation in longs and convert back to
ints.
This is actually done for some of the RNG implementations in CM.

Normally, the addition, subtraction and multiplication should be
unaffected. Problematic are division (modulo) and comparison,
which need to be treated specially. At least the Well type rngs do modulo
operations, but I did not look into detail if this would create problems.

I just wanted to raise this issue before any decision is being made.


>
> But IIUC, the RNG codes generally rely on bit operations (and bit
> operations disguised as arithmetic).[1]
> Fortunately, the representation is the same in C and Java, which
> allows us to mostly copy source code.
> There *are* caveats, though:
>   * ">>" in C code must (in general) be replaced with ">>>"
>   * Some constants (that exceed MAX_VALUE) cannot be written using
> their decimal representation (hexadecimal must be used instead).
>

the ">>" operator in C is implementation specific for signed values (see
http://stackoverflow.com/questions/7622/shift-operator-in-c).
Replacing it blindly with ">>>" is not correct imho.

Thomas


> Gilles
>
> [1] This is related to the "next(int bits)" debate: Ideally, an
> RNG indeed provides a "bit stream" but, as we are now all
> hopefully convinced, those that concern us in CM actually
> manipulate "int" (or "long") values, not "boolean" sequences.
>
>
> Thomas
>>
>> * the (adapted) "xorshift1024*" RNG implementation from
>>> http://xorshift.di.unimi.it/
>>> * an "adapter" to the existing generic methods
>>> * benchmarking code
>>>
>>>
>>> Gilles
>>>
>>>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [Math] Add 64-bits based RNG (Was: [Math] New base class for all RNGs)

[Thomas Neidhart]

On 01/10/2016 05:09 AM, Gilles wrote:
> Hi.
> 
> Relevant excerpt of previous posts:
> 
>>> [...]
>>>
>>> Something implicit in "BitStreamGenerator": the maximum number of
>>> bits is 32 (cf. return type of "next(int)" and the ubiquitous use
>>> of hard-coded "32".
>>>
>>> What about the possibility of using a "long" as a building block
>>> for another family of RNG?
>>
>> Why?  We don't have contributions of other RNGs implemented using
>> 64-bit blocks of data.  In theory, I guess you could do that, but I
>> don't know of any and all the ones we have use 32-bit words.
>>
>> Phil
>>>
>>> Gilles
> 
> (1)
> CPUs and OS are nowadays commonly 64-bits based.
> Thus one could legitimately wonder whether, on such systems, generating
>   one "long" value
> would not be faster than generating
>   two "int" values,
> especially when 64 bits are necessary in order to produce the next random
> "long" or "double" value.
> 
> (2)
> There indeed exist 64-bits implementations of RNGs:
>  
> http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/VERSIONS/C-LANG/mt19937-64.c
> 
>   http://burtleburtle.net/bob/rand/isaacafa.html
> 
> 
> I thus propose to create a feature branch that will contain
> * a new interface, for when "nextLong()" is the "random bits" producer

be aware that (at least some of) the 64-bit variants rely on unsigned
long arithmetic operations which can't be done in java without losing a
*lot* performance.

Thomas

> * the (adapted) "xorshift1024*" RNG implementation from
> http://xorshift.di.unimi.it/
> * an "adapter" to the existing generic methods
> * benchmarking code
> 
> 
> Gilles
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE][RC2] Release Commons Math 3.6

[Thomas Neidhart]

On 01/02/2016 09:15 PM, Luc Maisonobe wrote:
> This is a [VOTE] for releasing Apache Commons Math 3.6 from release
> candidate 2.
> 
> Tag name:
>   MATH_3_6_RC2 (signature can be checked from git using 'git tag -v')
> 
> Tag URL:
> 
> 
> 
> Commit ID the tag points at:
>   95a9d35e77f70ffc9bd5143880c236a760b42005
> 
> Site:
>   
> 
> Distribution files:
>   https://dist.apache.org/repos/dist/dev/commons/math/
> 
> Distribution files hashes (SHA1):
>  8b0b724b91102f63c7211f8de60dec19f94c4af7  commons-math3-3.6-bin.tar.gz
>  f3f2b3974e5ecd368aa3294f8c59a7aeae4c5d90  commons-math3-3.6-bin.zip
>  213985b076b344178cd2fb07e7257cb52b65e3b9  commons-math3-3.6-src.tar.gz
>  0c66a1aaf878aa7e6c0d9bbf917b25efa463b313  commons-math3-3.6-src.zip
> 
> KEYS file to check signatures:
>   http://www.apache.org/dist/commons/KEYS
> 
> Maven artifacts:
> 
> https://repository.apache.org/content/repositories/orgapachecommons-1138/org/apache/commons/commons-math3/3.6/
> 
[X] +1 Release it.

tested with

Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17
17:22:22+0200)
Maven home: /home/tn/bin/apache-maven-3.1.1
Java version: 1.7.0_91, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-i386/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-71-generic", arch: "i386", family: "unix"


Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] releasing 3.6

[Thomas Neidhart]

On 12/29/2015 07:39 PM, Luc Maisonobe wrote:
> Hi all,
> 
> A few weeks ago, I proposed to release 3.6. There were two
> points I wanted to address before that, both related to
> ODE. These points are now completed: the Adams methods
> stability issues have been fixed, and a bunch a field-based
> integrators are available.
> 
> There are 3 issues in JIRA that are tagged for 3.6:
> 
>  - MATH-1281 "Median" should not extend "Percentile"
>  - MATH-1285 Description API ZipfDistribution
>  - MATH-1308 Deprecate and remove "AbstractRandomGenerator"
> 
> MATH-1281 could probably not be solved in 3.6 (or in any 3.X),
> so I suggest to simply retag it for 4.0 only.
> 
> MATH-1285 seems complete to me, so I suggest to resolve it.

+1

> MATH-1308 is more 4.0-oriented and subject to experimentation.
> As we can do what we want in 4.0, it seems possible to *not*
> deprecate the class in 3.6, even if it completely disappears
> in 4.0 (furthermore as users probably are more concerned with
> the upper RandomGenerator interface or the lower specific
> implementations than with the intermedaite abstract class).

Any objection to also in apply the patch for MATH-1196?

> If you agree with this, I could cut an RC as soon as tomorrow,
> targeting a release for 2016-01-01 (so I would also change the
> copyright years throughout the library).
> 
> What do you think?

I fixed some compilation problems with java 1.5. The site reports look
all fine.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [Math] About the refactoring of RNGs

[Thomas Neidhart]

On 12/29/2015 05:10 PM, Gilles wrote:
> On Tue, 29 Dec 2015 10:33:15 +0100, Luc Maisonobe wrote:
>> Hi all,
>>
>> Le 29/12/2015 09:21, Thomas Neidhart a écrit :
>>> On 12/29/2015 04:33 AM, Phil Steitz wrote:
>>>> On 12/28/15 8:08 PM, Gilles wrote:
>>>>> On Mon, 28 Dec 2015 11:08:56 -0700, Phil Steitz wrote:
>>>>>> The significant refactoring to eliminate the (standard) next(int)
>>>>>> included in these changes has the possibility of introducing subtle
>>>>>> bugs or performance issues.  Please run some tests to verify that
>>>>>> the same sequences are generated by the 3_X code
>>>>>
>>>>> IIUC our unit tests of the RNGs, this is covered.
>>>>
>>>> No.  Not sufficient.  What you have done is changed the internal
>>>> implementation of all of the Bitstream generators.  I am not
>>>> convinced that you have not broken anything.  I will have to do the
>>>> testing myself.  I see no point in fiddling with the internals of
>>>> this code that has had a lot of eyeballs and testing on it.  I was
>>>> not personally looking forward to researching the algorithms to make
>>>> sure any invariants may be broken by these changes; but I am now
>>>> going to have to do this.  I have to ask why.  Please at some point
>>>> read [1], especially the sections on "Avoid Flexibility Syndrom" and
>>>> "Value Laziness as a Virtue."  Gratuitous refactoring drains
>>>> community energy.
>>>
>>> +1, on top of that I think we should aim to refactor the parts that
>>> really need refactoring
> 
> Though I could have liked to say as much on the parts of the library
> were my changes were much criticized because they failed to produce
> a perfect design (which the previous one wasn't either), I would have
> refrained to tell volunteers what they should do or not.

when I have seen the commit I was already thinking of writing a comment
about it but refrained until Phil was doing so. Afaict, your refactoring
moved the bit-shifting logic from one method (next(int)) to others
(nextDouble(), ...) in a slightly different way. On top of that, some
classes have been removed and added and I do not clearly see how this
has improved something, but maybe it was a change in the right direction
as Luc pointed out.

Nobody blames anybody for something, I believe, instead I have the
impression that people are focused on good technical solutions and this
sometimes means that you get criticism (and I got this as well a lot,
which can be quite frustrating).

I just got the impression lately that some people want to completely
refactor CM for the 4.0 release and I wonder if this will do any good to
the project, especially in areas where there is really no need to
further refactor anything.

>>> and try to keep the number of incompatibilities
>>> to the 3_X branch as minimal as possible.
> 
> I clearly and not surprisingly do not subscribe to that goal.
> And the recent discussions about RERO and "experimental" releases
> certainly were getting to a completely different consensus.

I am not sure what this has to do with my comment. RERO just means that
we should release more often, but you can only do this if you are not
introducing incompatibilities.

The discussion about "experimental" releases was just brain-storming
imho. There was no decision or real consensus on how to achieve this,
just some ideas that are at best very non-standard and in some cases
impractical.

I fear that Apache Commons is not the right place for doing such things.

btw. I proposed to use the scheme from guava to explicitly mark new
features with a Beta annotation to allow possible incompatible changes
later on, which I find much more practical, especially considering the
limited man-power and release cycles.

Thomas

>>>>>> and the refactored
>>>>>> code and benchmarks to show there is no loss in performance.
>>>>>
>>>>> Given that there are exactly two operations _less_ (a subtraction
>>>>> and a shift), it would be surprising.
>>>>>
>>>>>> It
>>>>>> would also be good to have some additional review of this code by
>>>>>> PRNG experts.
>>>>>
>>>>> The "nextInt()" code is exactly the same as the "next(int)" modulo
>>>>> the little change above (in the last line of the "nextInt/next"
>>>>> code).
>>>>>
>>>>> That change in "nextInt/next" implied similarly tiny recodings in
>>>>&g

Re: [Math] About the refactoring of RNGs (Was: [01/18] [math] MATH-1307)

[Thomas Neidhart]

On 12/29/2015 04:33 AM, Phil Steitz wrote:
> On 12/28/15 8:08 PM, Gilles wrote:
>> On Mon, 28 Dec 2015 11:08:56 -0700, Phil Steitz wrote:
>>> The significant refactoring to eliminate the (standard) next(int)
>>> included in these changes has the possibility of introducing subtle
>>> bugs or performance issues.  Please run some tests to verify that
>>> the same sequences are generated by the 3_X code
>>
>> IIUC our unit tests of the RNGs, this is covered.
> 
> No.  Not sufficient.  What you have done is changed the internal
> implementation of all of the Bitstream generators.  I am not
> convinced that you have not broken anything.  I will have to do the
> testing myself.  I see no point in fiddling with the internals of
> this code that has had a lot of eyeballs and testing on it.  I was
> not personally looking forward to researching the algorithms to make
> sure any invariants may be broken by these changes; but I am now
> going to have to do this.  I have to ask why.  Please at some point
> read [1], especially the sections on "Avoid Flexibility Syndrom" and
> "Value Laziness as a Virtue."  Gratuitous refactoring drains
> community energy. 

+1, on top of that I think we should aim to refactor the parts that
really need refactoring and try to keep the number of incompatibilities
to the 3_X branch as minimal as possible.

Thomas

>>> and the refactored
>>> code and benchmarks to show there is no loss in performance.
>>
>> Given that there are exactly two operations _less_ (a subtraction
>> and a shift), it would be surprising.
>>
>>> It
>>> would also be good to have some additional review of this code by
>>> PRNG experts.
>>
>> The "nextInt()" code is exactly the same as the "next(int)" modulo
>> the little change above (in the last line of the "nextInt/next"
>> code).
>>
>> That change in "nextInt/next" implied similarly tiny recodings in
>> the generic methods "nextDouble()", "nextBoolean()", ... which, apart
>> from that, were copied from "BitsStreamGenerator".
>>
>> [However tiny a change, I had made a mistake... and dozens of tests
>> started to fail. Found the typo and all was quiet again...]
>>
>> About "next(int)" being standard, it would be interesting to know
>> what that means.
> 
> Have a look at the source code for the JDK generators, for example.
>> As I indicated quite clearly in one of my first posts about this
>> refactoring
>> 1. all the CM implementations generate random bits in batches
>>of 32 bits, and
>> 2. before returning, the "next(int bits)" method was truncating
>>the generated "int":
>>  return x >>> (32 - bits);
>>
>> In all implementations, that was the only place where the "bits"
>> parameter was used, from which I concluded that the randomness
>> provider does not care if the request was to create less than 32
>> random bits.
>> Taking "nextBoolean()" for example, it looks like a waste of 31
>> bits (or am I missing something?).
> 
> Quite possibly, yes, you are missing something.
>>
>> Of course, if some implementation were able to store the bits not
>> requested by the last call to "next(int)", then I'd understand that
>> we must really provide access to a "next(int)" method.
>>
>> Perhaps that the overhead of such bookkeeping is why the practical
>> algorithms chose to store integers rather than bits (?).
>>
>> As you dismissed my request about CM being able to care for a RNG
>> implementation based on a "long", I don't quite understand the
>> caring for a "next(int)" that serves no more purpose (as of current
>> CM).
>>
> This change is
>>
>> Gilles
>>
>>
>>> Phil
>>>
>>> On 12/28/15 10:23 AM, er...@apache.org wrote:
 Repository: commons-math
 Updated Branches:
   refs/heads/master 7b62d0155 -> 81585a3c4


 MATH-1307

 New base class for RNG implementations.
 The source of randomness is provided through the "nextInt()"
 method (to be defined in subclasses).


 [...]
>>
> [1] http://www.apachecon.com/eu2007/materials/ac2006.2.pdf
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>>
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [Math] Exceptions from "JDKRandomGenerator"

[Thomas Neidhart]

On 12/23/2015 05:39 PM, Gilles wrote:
> On Wed, 23 Dec 2015 16:26:52 +0100, Thomas Neidhart wrote:
>> On 12/21/2015 04:41 AM, Gilles wrote:
>>> On Sat, 19 Dec 2015 11:35:26 -0700, Phil Steitz wrote:
>>>> On 12/19/15 9:02 AM, Gilles wrote:
>>>>> Hi.
>>>>>
>>>>> While experimenting on
>>>>>   https://issues.apache.org/jira/browse/MATH-1300
>>>>> I created a new
>>>>>   JDKRandomGeneratorTest
>>>>> that inherits from
>>>>>   RandomGeneratorAbstractTest
>>>>> similarly to the classes for testing all the other RNG implemented
>>>>> in CM.
>>>>>
>>>>> The following tests (implemented in the base class) failed:
>>>>> 1.
>>>>>
>>>>>
>>>>> testNextIntNeg(org.apache.commons.math4.random.JDKRandomGeneratorTest)
>>>>> Time elapsed: 0.001 sec  <<< ERROR!
>>>>>java.lang.Exception: Unexpected exception,
>>>>>
>>>>>
>>>>> expected
>>>>>
>>>>>
>>>>> but was
>>>>> 2.
>>>>>
>>>>>
>>>>> testNextIntIAE2(org.apache.commons.math4.random.JDKRandomGeneratorTest)
>>>>>
>>>>> Time elapsed: 0.015 sec  <<< ERROR!
>>>>>java.lang.IllegalArgumentException: bound must be positive
>>>>>
>>>>> This is caused by try/catch clauses that expect a
>>>>> "MathIllegalArgumentException"
>>>>> but "JDKRandomGenerator" extends "java.util.Random" that for those
>>>>> methods throws
>>>>> "IllegalArgumentException".
>>>>>
>>>>> What to do?
>>>>
>>>> I would change the test to expect IllegalArgumentException.  Most
>>>> [math] generators actually throw NotStrictlyPositiveException here,
>>>> which extends MIAE, which extends IAE, so this should work.
>>>
>>> It turns out that, in the master branch, the hierarchy is
>>>
>>>RuntimeException
>>>  |
>>>  MathRuntimeException
>>>  |
>>>MathIllegalArgumentException
>>>
>>> as per
>>>   https://issues.apache.org/jira/browse/MATH-853
>>>
>>> [And the Javadoc and "throws" clauses are not yet consistent with this
>>> in all the code base (e.g. the "RandomGenerator" interface).]
>>>
>>> So, in 4.0, "JDKRandomGenerator" should probably not inherit from
>>> "java.util.Random" but delegate to it, trap standard exceptions raised,
>>> and rethrow CM ones.
>>
>> which is probably a good indication that the current situation in CM4
>> (as per MATH-853) was not a good design decision.
> 
> It was consensual.
> What you express below is far more controversial.
> 
>> I applied the changes as I thought the issue was settled, but it turns
>> out that some of its implications were not fully taken into account.
>>
>> From my POV, we should stick to the existing exceptions were applicable,
>> as this is what people usually expect and is good practice. This means
>> we should not create our own MathIAE but instead throw a standard IAE. I
>> understand that the MIAE was created to support the localization of
>> exception messages, but I wonder if this is really needed in that case.
>> Usually, when an IAE is thrown it indicates a bug, as the developer did
>> not provide proper arguments as indicated per javadoc. Now I do not see
>> the value of being able to localize such exceptions as *only* developers
>> should ever see them.
> 
> This is a point I made a long time ago (not "in a far, far away galaxy").
> To which I got the answer that CM must provide a
>  * detailed,
>  * localizable,
>  * textual
> error message.
> 
> IMO, for a bug pointed to by an IAE, all the developer has to know is the
> stack trace.
> 
> If we want to be "standard", we shouldn't even have to check for null or
> array length on caller's input data as we know that the JVM will do the
> checks and trivially throw standard exceptions on these programming errors!

Checking input parameters and explicitly throwing exceptions *is*
considered to be good practice.

Some of the commons libraries that exist for a very long time do it
differently (especially lang and collections), but this might lead to
inconsistent behavior (check th

Re: [Math] Exceptions from "JDKRandomGenerator"

[Thomas Neidhart]

On 12/21/2015 04:41 AM, Gilles wrote:
> On Sat, 19 Dec 2015 11:35:26 -0700, Phil Steitz wrote:
>> On 12/19/15 9:02 AM, Gilles wrote:
>>> Hi.
>>>
>>> While experimenting on
>>>   https://issues.apache.org/jira/browse/MATH-1300
>>> I created a new
>>>   JDKRandomGeneratorTest
>>> that inherits from
>>>   RandomGeneratorAbstractTest
>>> similarly to the classes for testing all the other RNG implemented
>>> in CM.
>>>
>>> The following tests (implemented in the base class) failed:
>>> 1.
>>>
>>> testNextIntNeg(org.apache.commons.math4.random.JDKRandomGeneratorTest)
>>> Time elapsed: 0.001 sec  <<< ERROR!
>>>java.lang.Exception: Unexpected exception,
>>>
>>> expected
>>>
>>> but was
>>> 2.
>>>
>>> testNextIntIAE2(org.apache.commons.math4.random.JDKRandomGeneratorTest)
>>> Time elapsed: 0.015 sec  <<< ERROR!
>>>java.lang.IllegalArgumentException: bound must be positive
>>>
>>> This is caused by try/catch clauses that expect a
>>> "MathIllegalArgumentException"
>>> but "JDKRandomGenerator" extends "java.util.Random" that for those
>>> methods throws
>>> "IllegalArgumentException".
>>>
>>> What to do?
>>
>> I would change the test to expect IllegalArgumentException.  Most
>> [math] generators actually throw NotStrictlyPositiveException here,
>> which extends MIAE, which extends IAE, so this should work.
> 
> It turns out that, in the master branch, the hierarchy is
> 
>RuntimeException
>  |
>  MathRuntimeException
>  |
>MathIllegalArgumentException
> 
> as per
>   https://issues.apache.org/jira/browse/MATH-853
> 
> [And the Javadoc and "throws" clauses are not yet consistent with this
> in all the code base (e.g. the "RandomGenerator" interface).]
> 
> So, in 4.0, "JDKRandomGenerator" should probably not inherit from
> "java.util.Random" but delegate to it, trap standard exceptions raised,
> and rethrow CM ones.

which is probably a good indication that the current situation in CM4
(as per MATH-853) was not a good design decision.

I applied the changes as I thought the issue was settled, but it turns
out that some of its implications were not fully taken into account.

>From my POV, we should stick to the existing exceptions were applicable,
as this is what people usually expect and is good practice. This means
we should not create our own MathIAE but instead throw a standard IAE. I
understand that the MIAE was created to support the localization of
exception messages, but I wonder if this is really needed in that case.
Usually, when an IAE is thrown it indicates a bug, as the developer did
not provide proper arguments as indicated per javadoc. Now I do not see
the value of being able to localize such exceptions as *only* developers
should ever see them.

For any other exceptions (typically converge exceptions or similar)
which are clearly algorithm dependent, I fully support the design of 1
base MathRuntimeException with localization support.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] Refactored Precision

[Thomas Neidhart]

On Mon, Dec 14, 2015 at 9:17 AM, Ole Ersoy  wrote:

> Hi,
>
> Just a heads up for those of you interested or have nothing better to do
> at 2 am :).  I refactored the Precision class into classes PrecisionAssert
> and RoundDouble.
>
> https://github.com/firefly-numbers/firefly-numbers
>
> I created a new github organization for the package, since it deals less
> with math and more with number precision in general.  I also removed
> support for float.  It seems like most of the code in CM uses double, and
> if float is needed then it should be provided via it's own module.  I also
> replaced calls to FastMath with Math.  Most of the calls were for abs() and
> ulp()...functions that I would think would have similar performance
> regardless.
>
> Probably moving onto FastMath next.  I plan on only including functions
> that have a performance benefit, and delegating to Math for everything else.
>

Hi Ole,

what is the motivation for posting these questions on the math ML?
Do you intend to contribute some new functionality or propose changes back
to commons-math?

Or is this project intended to be a fork of commons-math?

Thomas

Re: [collections][site] publish security-report? (was: svn commit: r1719350 - /commons/proper/collections/trunk/src/site/xdoc/security-reports.xml)

[Thomas Neidhart]

On 12/11/2015 01:16 PM, Bernd Eckenfels wrote:
> Hello,
> 
> I added a second CVE to the de-serialisation security-report for
> ACC, I dont have the infrastrcuture handy to push the site, can
> somebody do that, please?

fixed the respective html directly on the site repository.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] MATH 4 requires at least java 7, build also the MATH_3_X branch.

[Thomas Neidhart]

On Thu, Dec 3, 2015 at 9:31 AM, Luc Maisonobe  wrote:

> Hi Thomas,
>
> Le 02/12/2015 21:08, t...@apache.org a écrit :
> > Repository: commons-math
> > Updated Branches:
> >   refs/heads/master 7afc1c34f -> 25de9b780
> >
> >
> > MATH 4 requires at least java 7, build also the MATH_3_X branch.
>
> Do you mean MATH_3_X does not compile with Java 6 anymore?
>

no. The MATH4 branch requires java 7.

The MATH3_X branch should still work with java 6, but I need to create a
separate travis config for it.

Thomas


>
> best regards,
> Luc
>
> >
> >
> > Project: http://git-wip-us.apache.org/repos/asf/commons-math/repo
> > Commit:
> http://git-wip-us.apache.org/repos/asf/commons-math/commit/25de9b78
> > Tree: http://git-wip-us.apache.org/repos/asf/commons-math/tree/25de9b78
> > Diff: http://git-wip-us.apache.org/repos/asf/commons-math/diff/25de9b78
> >
> > Branch: refs/heads/master
> > Commit: 25de9b7800887c12365f6a19b13cf32baf5bfe2f
> > Parents: 7afc1c3
> > Author: Thomas Neidhart 
> > Authored: Wed Dec 2 21:08:41 2015 +0100
> > Committer: Thomas Neidhart 
> > Committed: Wed Dec 2 21:08:41 2015 +0100
> >
> > --
> >  .travis.yml | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> > --
> >
> >
> >
> http://git-wip-us.apache.org/repos/asf/commons-math/blob/25de9b78/.travis.yml
> > --
> > diff --git a/.travis.yml b/.travis.yml
> > index bd60a9f..3177040 100644
> > --- a/.travis.yml
> > +++ b/.travis.yml
> > @@ -2,14 +2,14 @@ language: java
> >  sudo: false
> >
> >  jdk:
> > -  - openjdk6
> >- openjdk7
> >- oraclejdk8
> >
> > -# only build trunk
> > +# only build master and the MATH_3_X branch
> >  branches:
> >only:
> >  - master
> > +- MATH_3_X
> >
> >  after_success:
> >- mvn clean test jacoco:report coveralls:report
> >
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [VOTE] Release Configuration 2.0-beta2 based on RC1

[Thomas Neidhart]

On 12/01/2015 10:25 PM, Oliver Heger wrote:
> Hi all,
> 
> to continue the current series of releases, this is a vote for the
> second beta version of [configuration] 2.0 based on the first release
> candidate. After beta1 was released in June there has been some feedback
> which lead to some minor changes on the API. Details are available in
> the release notes.
> 
> Because this is a beta and not yet an official release I did not deploy
> the artifacts to Nexus; only the distributions were created.
> 
> Configuration 2.0-beta1 RC2 is available for review here:
> https://dist.apache.org/repos/dist/dev/commons/configuration
> (revision 11367)
> 
> Details of changes since 1.10 and the previous beta version are in the
> release notes:
> 
> https://dist.apache.org/repos/dist/dev/commons/configuration/RELEASE-NOTES.txt
> 
> http://people.apache.org/~oheger/configuration-2.0-beta2-rc1/changes-report.html
> 
> Here is the tag:
> 
> http://svn.apache.org/repos/asf/commons/proper/configuration/tags/CONFIGURATION_2_0_beta2_RC1
> (revision 1717329)
> 
> Site:
>http://people.apache.org/~oheger/configuration-2.0-beta2-rc1/
> (note some links in the menu are not yet working)
> 
> KEYS:
>   http://www.apache.org/dist/commons/KEYS
> 
> Please review the release candidate and vote.
> This vote will close no sooner than 72 hours from now, i.e. after 2100
> GMT 04-Dec 2015

Hi Oliver,

first I looked at the site which reported many test errors.

Building the source tag / source dist myself using

Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17
17:22:22+0200)
Maven home: /home/tn/bin/apache-maven-3.1.1
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-i386/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-63-generic", arch: "i386", family: "unix"

results in no errors:

Tests run: 2697, Failures: 0, Errors: 0, Skipped: 0

so I guess the site build is corrupted.


The rest looks fine, so considering that this is a beta release

[X] +1 Release these artifacts

Thomas


>   [ ] +0 OK, but...
>   [ ] -0 OK, but really should fix...
>   [ ] -1 I oppose this release because...
> 
> Thanks!
> Oliver
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] Adding badges

[Thomas Neidhart]

On 12/02/2015 08:29 PM, Luc Maisonobe wrote:
> Le 02/12/2015 09:57, Thomas Neidhart a écrit :
>> Hi,
>>
>> recently I added some badges (building on travis, code coverage with
>> coveralls, license tag, latest available version from maven) to
>> collections, which can be seen here:
>> https://github.com/apache/commons-collections
>>
>> Any objection to add the same for math?
> 
> Go for it if you think it is useful.
> 
> For the record, there is a pending pull request about travis here:
> 
>  <https://github.com/apache/commons-math/pull/11>

ok done. I have also closed the pull request at the same time.

The infra ticket [1] for travis is pending, it might take a few days.

Thomas

[1] https://issues.apache.org/jira/browse/INFRA-10883

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[math] Adding badges

[Thomas Neidhart]

Hi,

recently I added some badges (building on travis, code coverage with
coveralls, license tag, latest available version from maven) to
collections, which can be seen here:
https://github.com/apache/commons-collections

Any objection to add the same for math?

The travis integration can be quite useful for a RM as you can specify
which jdks shall be automatically tested.

Thomas

[collections] Next Steps after 4.1 release

[Thomas Neidhart]

Hi all,

we have just released a new version of the collections component and I
wanted to start a discussion about the future of this component.

In the past years I spent some time to clean up the component and also
add a few things that I found worth the effort (which I tend to use in
my day jobs as well). As of today, there are no open bugs (only feature
requests) so the component can be considered to be in good shape, but
apart from myself, there is no real community around the component anymore.

I had planned to step back from maintaining this component after the 4.1
release and so the time has finally come. If needed I will probably fix
bugs that will be reported for the latest release, but I have no
interest in further driving this component and catching up wrt latest
JDK features and guava. Maybe somebody else sees some potential and
wants to take over, and I would be happy to help out if needed.

Thomas


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[ANNOUNCEMENT] Apache Commons Collections 4.1 Released

[Thomas Neidhart]

The Apache Commons team is pleased to announce the release of Apache
Commons Collections 4.1. The release is available for download at


http://commons.apache.org/proper/commons-collections/download_collections.cgi

Apache Commons Collections is a project to develop and maintain
collection classes based on and inspired by the JDK collection framework.

This Collections 4.1 release is a security and minor release, adding new
features and fixing several bugs present in the previous release of the
4.X branch.

Additionally, this release provides a fix for a known remote code
exploitation via the standard java object serialization mechanism.
Serialization support for unsafe classes in the functor package
has been removed. For more details, please refer to COLLECTIONS-580.

All users are strongly encouraged to updated to this release.

See the release-notes at

http://commons.apache.org/proper/commons-collections/release_4_1.html
http://www.apache.org/dist/commons/collections/RELEASE-NOTES.txt

for a full list of changes.

Please verify signatures using the KEYS file available at the above
location when downloading the release.

For complete information on collections, including instructions on how
to submit bug reports, patches, or suggestions for improvement, see the
Apache Commons Collections website:

http://commons.apache.org/proper/commons-collections/

Thomas, on behalf of the Apache Commons team

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[VOTE][RESULT] Release Commons Collections 4.1 Based on RC2 (24h vote)

[Thomas Neidhart]

Voting was as follows:

+1 (binding)

Luc Maisonobe
Joerg Schaible
Oliver Heger
Phil Steitz
Gary Gregory
Thomas Neidhart

There were no other votes.

The vote therefore passes.

Thanks to all who voted.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [CANCEL][VOTE] Release Commons Collections 4.1 Based on RC1

[Thomas Neidhart]

On 11/25/2015 09:04 PM, Gary Gregory wrote:
> On Wed, Nov 25, 2015 at 11:40 AM, Thomas Neidhart > wrote:
> 
>> On 11/22/2015 11:26 PM, Thomas Neidhart wrote:
>>> Hi all,
>>>
>>> we have accumulated enough changes since the last 4.0 release as well as
>>> we need to provide a fix for the known remote code exploit via java
>>> de-serialization. Therefore, I would like to start a vote to release
>>> Commons Collections 4.1 based on RC1.
>>
>> Thanks for all the people that voted.
>> Some glitches have been found and I also have found myself a few minor
>> issues that I would like to correct as this will be my last release.
>>
> 
> What do you mean by "this will be my last release"?

that means that I will use my evening spare time differently in the
future, but I will send a separate email to the ml after this release.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[VOTE] Release Commons Collections 4.1 Based on RC2 (24h vote)

[Thomas Neidhart]

Hi all,

we have accumulated enough changes since the last 4.0 release as well as
we need to provide a fix for the known remote code exploit via java
de-serialization. Therefore, I would like to start a vote to release
Commons Collections 4.1 based on RC2.

Note:

 * The fix for the security related issue results in Clirr errors as
   unsafe classes in the functor package do not implement the
   Serializable interface anymore. This is mentioned in the release
   notes.

 * There are 2 test failures with the IBM 6 JDK. The same failures were
   reported for the 4.0 release and are related to a buggy Map
   implementation in this JDK

 * Commons Collections 4.X does not successfully compile with JDK 9 EA
   This will be tackled in a later release.

Changes since RC1:

 * fixed compilation problems of test classes with some Java 8 compilers
 * fixed some javadoc in IterableUtils and MultiValuedMap
 * added travis configuration (only in the repository, not part of the
   release) to help a RM by building with different jdks

Collections 4.1 RC2 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/collections/
(svn revision 11307)

Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1129/org/apache/commons/commons-collections4/4.1/

Details of changes since 4.0 are in the release notes:

https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt

http://people.apache.org/builds/commons/collections/4.1/RC2/changes-report.html

The tag is here:

https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_4_1_RC2
(svn revision 1716550)

Site:
http://people.apache.org/builds/commons/collections/4.1/RC2/

Clirr Report (compared to 4.0):

http://people.apache.org/builds/commons/collections/4.1/RC2/clirr-report.html

RAT Report:

http://people.apache.org/builds/commons/collections/4.1/RC2/rat-report.html

KEYS:
https://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.

This vote will close no sooner than 24 hours from now, i.e. after 2400
GMT 26-November 2015

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[CANCEL][VOTE] Release Commons Collections 4.1 Based on RC1

[Thomas Neidhart]

On 11/22/2015 11:26 PM, Thomas Neidhart wrote:
> Hi all,
> 
> we have accumulated enough changes since the last 4.0 release as well as
> we need to provide a fix for the known remote code exploit via java
> de-serialization. Therefore, I would like to start a vote to release
> Commons Collections 4.1 based on RC1.

Thanks for all the people that voted.
Some glitches have been found and I also have found myself a few minor
issues that I would like to correct as this will be my last release.

So my plan is to create a new RC and call a new vote that will expire in
24h as no critical issues have been found so far.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Collections 4.1 Based on RC1

[Thomas Neidhart]

On 11/25/2015 11:30 AM, Jörg Schaible wrote:
> Hi Thomas,
> 
> Thomas Neidhart wrote:
> 
>> On 11/24/2015 11:30 PM, Jörg Schaible wrote:
>>> Hi Thomas,
>>>
>>> Thomas Neidhart wrote:
> 
> [snip]
> 
>> These test failures exist since the 4.0 release, quoting your vote for
>> Collection 4.0 based on RC5:
>>
>>> +1, builds for all but one JDK flawlessly from source. I still have 2
>>> failing tests for IBM JDK 1.6:
>>>
>>> Failed tests:
>>>   
> AbstractMapTest$TestMapEntrySet.testMapEntrySetIteratorEntrySetValue:1656
>>> expected: but was:
>>>   
> AbstractMapTest$TestMapEntrySet.testMapEntrySetIteratorEntrySetValue:1656
>>> expected: but was:
>>>
>>> However, since we already blamed that JDK, it does not influence the
>>> release.
>>
>> I can not remember anymore why these have not been worked-around though,
>> but I suspect that it was not so simple in this case.
> 
> OK, at least it's no regression.
> 
>>>  %< ===
>>> $ mvn-3.0 -version
>>> Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19
>>> 14:51:28+0100)
>>> Maven home: /usr/share/maven-bin-3.0
>>> Java version: 1.6.0, vendor: IBM Corporation
>>> Java home: /opt/ibm-jdk-bin-1.6.0.9_p2/jre
>>> Default locale: en_US, platform encoding: UTF-8
>>> OS name: "linux", version: "4.1.12-gentoo", arch: "amd64", family: "unix"
>>>  %< ===
>>>
>>> It fails to compile with Java 8:
>>>
>>>  %< ===
>>> [INFO] -
>>> [ERROR] COMPILATION ERROR :
>>> [INFO] -
>>> [ERROR] /home/joehni/tmp/download/commons-collections4-4.1-
>>>
> src/src/test/java/org/apache/commons/collections4/FluentIterableTest.java:
>>> [242,41] reference to forEach is ambiguous
>>>   both method forEach(java.util.function.Consumer) in
>>> java.lang.Iterable and method
>>> forEach(org.apache.commons.collections4.Closure) in
>>> org.apache.commons.collections4.FluentIterable match
>>> [INFO] 1 error
>>> [INFO] -
>>
>> ok, this error has been already spotted by Oliver and fixed in trunk. It
>> is only in a test case where the type inference fails when passing a
>> null value to a method, as you usually do in tests.
> 
> OK, fine with me.
> 
> [snip]
> 
>>> nor Java 9:
>>
>> ok. I have to investigate these errors, but I really wonder what we
>> should do about it. Java 1.9 is not released yet, and these might be
>> compiler related bugs that will be fixed, and adding work-arounds for
>> things that perfectly compile with all previous Java versions does not
>> sound right.
> 
> I've update Java 9 now to the latest available (b93), but the problem stays. 
> However, we have the same errors with 4.0.0, so it's actually no regression, 
> but an indication that we need probably collections5 when Java 9 is final.

I did some further investigation into the compile errors with Java 9.

Some errors depend on the source/target version: they disappear for
version 1.8+ (there is at least one bugreport related to that:
https://bugs.openjdk.java.net/browse/JDK-8075793)

Some errors are not understandable at all:

Iterator[] iterators = new Iterator[others.length + 1];
iterators[0] = first.iterator();
for (int i = 0; i < others.length; i++) {
iterators[i + 1] = others[i].iterator();
}
return IteratorUtils.zippingIterator(iterators);

The signature of IteratorUtils.zippingIterator is as follows:

public static  ZippingIterator zippingIterator(final Iterator... iterators)

it fails with:

[ERROR]
/home/tn/workspace/apache/commons-collections/src/main/java/org/apache/commons/collections4/IterableUtils.java:[554,36]
error: no suitable method found for zippingIterator(Iterator[])


changing the call to

return IteratorUtils.zippingIterator(iterators);

makes it pass.

Anyway, this will require some more time to investigate and fix. For a
4.1 release I will note that the project can not be compiled with Java 9
yet, and the next release (4.2) should improve on that imho.

Thanks for testing,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Collections 4.1 Based on RC1

[Thomas Neidhart]

On 11/24/2015 11:30 PM, Jörg Schaible wrote:
> Hi Thomas,
> 
> Thomas Neidhart wrote:
> 
>> Hi all,
>>
>> we have accumulated enough changes since the last 4.0 release as well as
>> we need to provide a fix for the known remote code exploit via java
>> de-serialization. Therefore, I would like to start a vote to release
>> Commons Collections 4.1 based on RC1.
>>
>> Note:
>>
>> The fix for the security related issue results in Clirr errors as unsafe
>> classes in the functor package do not implement the Serializable
>> interface anymore. This is mentioned in the release notes.
>>
>>
>> Collections 4.1 RC1 is available for review here:
>> https://dist.apache.org/repos/dist/dev/commons/collections/
>> (svn revision 11263)
>>
>> Maven artifacts are here:
>>
>>
>>
> https://repository.apache.org/content/repositories/orgapachecommons-1122/org/apache/commons/commons-collections4/4.1/
>>
>> Details of changes since 4.0 are in the release notes:
>>
>>
>> https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt
>>
>>
>> http://people.apache.org/builds/commons/collections/4.1/RC1/changes-report.html
>>
>> The tag is here:
>>
>>
>>
> https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_4_1_RC1
>> (svn revision 1715703)
>>
>> Site:
>> http://people.apache.org/builds/commons/collections/4.1/RC1/
>>
>> Clirr Report (compared to 4.0):
>>
>>
>> http://people.apache.org/builds/commons/collections/4.1/RC1/clirr-report.html
>>
>> RAT Report:
>>
>>
>> http://people.apache.org/builds/commons/collections/4.1/RC1/rat-report.html
>>
>> KEYS:
>> https://www.apache.org/dist/commons/KEYS
>>
>> Please review the release candidate and vote.
>>
>> This vote will close no sooner that 72 hours from now, i.e. after 2400
>> GMT 25-November 2015
>>
>>   [ ] +1 Release these artifacts
>>   [ ] +0 OK, but...
>>   [ ] -0 OK, but really should fix...
>>   [ ] -1 I oppose this release because...
>>
>> Thanks,
>>
>> Thomas
> 
> It fails for IBM JDK 6:
> 
>  %< ===
> Failed tests: 
> org.apache.commons.collections4.map.AbstractMapTest$TestMapEntrySet.testMapEntrySetIteratorEntrySetValue(org.apache.commons.collections4.map.AbstractMapTest$TestMapEntrySet)
>   Run 1: PASS
>   Run 2: PASS
>   Run 3: PASS
>   Run 4: PASS
>   Run 5: PASS
>   Run 6: PASS
>   Run 7: PASS
>   Run 8: PASS
>   Run 9: PASS
>   Run 10: PASS
>   Run 11: PASS
>   Run 12: PASS
>   Run 13: PASS
>   Run 14: PASS
>   Run 15: PASS
>   Run 16: PASS
>   Run 17: PASS
>   Run 18: PASS
>   Run 19: PASS
>   Run 20: PASS
>   Run 21: PASS
>   Run 22: PASS
>   Run 23: PASS
>   Run 24: PASS
>   Run 25: PASS
>   Run 26: PASS
>   Run 27: PASS
>   Run 28: PASS
>   Run 29: PASS
>   Run 30: PASS
>   Run 31: PASS
>   Run 32: PASS
>   Run 33: PASS
>   Run 34: PASS
>   Run 35: PASS
>   Run 36: PASS
>   Run 37: 
> AbstractMapTest$TestMapEntrySet.testMapEntrySetIteratorEntrySetValue:1665 
> expected: but was:
>   Run 38: PASS
>   Run 39: PASS
>   Run 40: PASS
>   Run 41: PASS
>   Run 42: 
> AbstractMapTest$TestMapEntrySet.testMapEntrySetIteratorEntrySetValue:1665 
> expected: but was:
>   Run 43: PASS

These test failures exist since the 4.0 release, quoting your vote for
Collection 4.0 based on RC5:

> +1, builds for all but one JDK flawlessly from source. I still have 2 
> failing tests for IBM JDK 1.6:
> 
> Failed tests: 
>   AbstractMapTest$TestMapEntrySet.testMapEntrySetIteratorEntrySetValue:1656 
> expected: but was:
>   AbstractMapTest$TestMapEntrySet.testMapEntrySetIteratorEntrySetValue:1656 
> expected: but was:
> 
> However, since we already blamed that JDK, it does not influence the 
> release.

I can not remember anymore why these have not been worked-around though,
but I suspect that it was not so simple in this case.


>  %< ===
> $ mvn-3.0 -version
> Apache Maven 3.0.5 (r01de14724cdef164cd33c7c8c2fe155faf9602da; 2013-02-19 
> 14:51:28+0100)
> Maven home: /usr/share/maven-bin-3.0
> Java version: 1.6.0, vendor: IBM Corporation
> Java home: /opt/ibm-jdk-bin-1.6.0.9_p2/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.1.12-gentoo", arch: "amd64", family: "unix"
>  %< ===

Re: [VOTE] Release Commons Collections 4.1 Based on RC1

[Thomas Neidhart]

On 11/22/2015 11:26 PM, Thomas Neidhart wrote:
> Hi all,
> 
> we have accumulated enough changes since the last 4.0 release as well as
> we need to provide a fix for the known remote code exploit via java
> de-serialization. Therefore, I would like to start a vote to release
> Commons Collections 4.1 based on RC1.
> 
> Note:
> 
> The fix for the security related issue results in Clirr errors as unsafe
> classes in the functor package do not implement the Serializable
> interface anymore. This is mentioned in the release notes.
> 
> 
> Collections 4.1 RC1 is available for review here:
> https://dist.apache.org/repos/dist/dev/commons/collections/
> (svn revision 11263)
> 
> Maven artifacts are here:
> 
> 
> https://repository.apache.org/content/repositories/orgapachecommons-1122/org/apache/commons/commons-collections4/4.1/
> 
> Details of changes since 4.0 are in the release notes:
> 
> 
> https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt
> 
> 
> http://people.apache.org/builds/commons/collections/4.1/RC1/changes-report.html
> 
> The tag is here:
> 
> 
> https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_4_1_RC1
> (svn revision 1715703)
> 
> Site:
> http://people.apache.org/builds/commons/collections/4.1/RC1/
> 
> Clirr Report (compared to 4.0):
> 
> 
> http://people.apache.org/builds/commons/collections/4.1/RC1/clirr-report.html
> 
> RAT Report:
> 
> 
> http://people.apache.org/builds/commons/collections/4.1/RC1/rat-report.html
> 
> KEYS:
> https://www.apache.org/dist/commons/KEYS
> 
> Please review the release candidate and vote.
> 
> This vote will close no sooner that 72 hours from now, i.e. after 2400
> GMT 25-November 2015
> 
>   [ ] +1 Release these artifacts
>   [ ] +0 OK, but...
>   [ ] -0 OK, but really should fix...
>   [ ] -1 I oppose this release because...

I would like to remind all PMC members that this is also a security
related release that several people have requested.

If someone is not happy with the release as is, please speak up and vote
so that we at least can move forward.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [JXPATH] Java Version

[Thomas Neidhart]

On 11/24/2015 09:55 PM, Uwe Barthel wrote:
>> I've updated JXPATH to Java 7. There is a lot of work to update the code
>> base to use Java 7 languages features and APIs. I invite everybody to join
>> me here…
> 
> Do you like to start these changes before or after the release 1.4?
> I prefer to create the release as soon as possible and start rework on that 
> baseline.

If the idea is to roll out a bugfix release that people are awaiting
then I do not see the point in updating the minimum java version and
changing the code to use new language / jdk features.

This will just limit the use of the bugfix release, as not every
application using it will have been upgraded already. Also, every change
might introduce another problem, but this is up to the judgement of the
maintainer(s).

For the release afterwards (i.e. 1.5), I think it is fine to apply all
the changes, and updating the minimum java version in minor releases is
ok and has been done for other components already.

Just my 2 cents

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: svn commit: r1716090 - in /commons/proper/collections/trunk: .travis.yml README.md pom.xml

[Thomas Neidhart]

Actually, it does not work yet, as I can not enable commons-collections on
travis.

I already sent an email to them, but did not get an answer yet. How did you
manage to do it for commons-lang?

Thomas

On Tue, Nov 24, 2015 at 5:36 PM, Benedikt Ritter  wrote:

> Nice!
>
> 2015-11-24 10:25 GMT+01:00 :
>
> > Author: tn
> > Date: Tue Nov 24 09:25:40 2015
> > New Revision: 1716090
> >
> > URL: http://svn.apache.org/viewvc?rev=1716090&view=rev
> > Log:
> > Add travis configuration for jacoco and coveralls. Add badges to
> Readme.md.
> >
> > Modified:
> > commons/proper/collections/trunk/.travis.yml
> > commons/proper/collections/trunk/README.md
> > commons/proper/collections/trunk/pom.xml
> >
> > Modified: commons/proper/collections/trunk/.travis.yml
> > URL:
> >
> http://svn.apache.org/viewvc/commons/proper/collections/trunk/.travis.yml?rev=1716090&r1=1716089&r2=1716090&view=diff
> >
> >
> ==
> > --- commons/proper/collections/trunk/.travis.yml (original)
> > +++ commons/proper/collections/trunk/.travis.yml Tue Nov 24 09:25:40 2015
> > @@ -5,3 +5,6 @@ jdk:
> >- openjdk6
> >- openjdk7
> >- oraclejdk8
> > +
> > +after_success:
> > +  - mvn clean test jacoco:report coveralls:report
> >
> > Modified: commons/proper/collections/trunk/README.md
> > URL:
> >
> http://svn.apache.org/viewvc/commons/proper/collections/trunk/README.md?rev=1716090&r1=1716089&r2=1716090&view=diff
> >
> >
> ==
> > --- commons/proper/collections/trunk/README.md (original)
> > +++ commons/proper/collections/trunk/README.md Tue Nov 24 09:25:40 2015
> > @@ -43,6 +43,11 @@
> >  Apache Commons Collections
> >  ===
> >
> > +[![Build Status](
> >
> https://travis-ci.org/apache/commons-collections.svg?branch=master)](https://travis-ci.org/apache/commons-collections
> > )
> > +[![Coverage Status](
> >
> https://coveralls.io/repos/apache/commons-collections/badge.svg?branch=master)](https://coveralls.io/r/apache/commons-collections
> > )
> > +[![Maven Central](
> >
> https://maven-badges.herokuapp.com/maven-central/org.apache.commons/commons-collections4/badge.svg)](https://maven-badges.herokuapp.com/maven-central/org.apache.commons/commons-collections4/
> > )
> > +[![License](
> >
> http://img.shields.io/:license-apache-blue.svg)](http://www.apache.org/licenses/LICENSE-2.0.html
> > )
> > +
> >  The Apache Commons Collections package contains types that extend and
> > augment the Java Collections Framework.
> >
> >  Documentation
> >
> > Modified: commons/proper/collections/trunk/pom.xml
> > URL:
> >
> http://svn.apache.org/viewvc/commons/proper/collections/trunk/pom.xml?rev=1716090&r1=1716089&r2=1716090&view=diff
> >
> >
> ==
> > --- commons/proper/collections/trunk/pom.xml (original)
> > +++ commons/proper/collections/trunk/pom.xml Tue Nov 24 09:25:40 2015
> > @@ -690,6 +690,38 @@
> >  
> >
> >  
> > +
> > +
> > +  travis
> > +  
> > +
> > +  env.TRAVIS
> > +  true
> > +
> > +  
> > +  
> > +
> > +  
> > +org.jacoco
> > +jacoco-maven-plugin
> > +${commons.jacoco.version}
> > +
> > +  
> > +prepare-agent
> > +
> > +  prepare-agent
> > +
> > +  
> > +
> > +  
> > +  
> > +org.eluder.coveralls
> > +coveralls-maven-plugin
> > +3.1.0
> > +  
> > +
> > +  
> > +
> >
> >
> >  
> >
> >
> >
>
>
> --
> http://people.apache.org/~britter/
> http://www.systemoutprintln.de/
> http://twitter.com/BenediktRitter
> http://github.com/britter
>

Re: [VOTE] Release Commons Collections 4.1 Based on RC1

[Thomas Neidhart]

On 11/23/2015 09:53 PM, Oliver Heger wrote:
> The Maven build worked fine with Java 1.6 and 1.7 on Windows 10. With
> Java 1.8 I get the following compilation error in test classes:
> 
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /c:/data/dev/projects/OpenSource/collections/commons-collections4-4.1-src/src/test/java/org/apache/commons/collections4/FluentIterableTest.java:[242,41]
> reference to forEach is ambiguous
>   both method forEach(java.util.function.Consumer) in
> java.lang.Iterable and method
> forEach(org.apache.commons.collections4.Closure) in
> org.apache.commons.collections4.FluentIterable match
> [INFO] 1 error

Hi Oliver,

thanks for the test. Actually, I did test it with JDK 1.8, but with an
earlier version and the build/test worked fine.

Using the latest JDK (1.8.0_65) showed the same error as you reported.
Additionally, the eclipse compiler (included in release Luna 4.4.1)
showed some more compile errors which are mostly related to tests that
pass null as argument.

I will fix them in trunk, but I agree that these are not critical for
the release.

> Artifacts and site look good. As the compilation error is only in a test
> class, this is not blocking.
> 
> +1

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release NET 3.4 based on RC2 - resend with corrected tag

[Thomas Neidhart]

On 11/22/2015 04:58 PM, sebb wrote:
> It's probably about time to release NET.
> There have been quite a few improvements and fixes since the last version.
> 
> [This is a repeat of the original mail, but using a tag that actually
> exists this time]
> 
> ==
> 
> Net 3.4 RC2 is available for review here:
> https://dist.apache.org/repos/dist/dev/commons/net/ (svn revision 11241)
> 
> ./binaries/commons-net-3.4-bin.tar.gz.sha1:88411522395b4000b038a94ab007be89ebda2ec3
> ./binaries/commons-net-3.4-bin.zip.sha1:f252a45b1610346116c9dd966fec9a15171223d1
> ./source/commons-net-3.4-src.tar.gz.sha1:abfba84427a06341e113d59d8f75855e67093087
> ./source/commons-net-3.4-src.zip.sha1:f3b38dfcccd8fcdc9ac500a5f8580a19817b
> 
>   Maven artifacts are here:
> 
> https://repository.apache.org/content/repositories/orgapachecommons-1120/commons-net/commons-net/3.4/
> 
> These are the artifacts and their hashes
> 
> /commons-net/commons-net/3.4/commons-net-3.4-test-sources.jar
> (SHA1: fdb74119054a2aacc134c56137660d7a0a40e4a8)
> /commons-net/commons-net/3.4/commons-net-3.4-javadoc.jar
> (SHA1: b882750c8dbd480e2b9afd357dcf71d962f2fa03)
> /commons-net/commons-net/3.4/commons-net-3.4-ftp.jar
> (SHA1: 6fc585e5f3dc8333b20110af22a8bae59f5246cb)
> /commons-net/commons-net/3.4/commons-net-3.4-tests.jar
> (SHA1: ce44ebc0e7be56c3bd650700be93c5b377b47573)
> /commons-net/commons-net/3.4/commons-net-3.4.pom
> (SHA1: d1790447a41c848af8cba0919fae7a577fbc744a)
> /commons-net/commons-net/3.4/commons-net-3.4.jar
> (SHA1: 5e984db9554728564d58e90da5d90eff8ae8cf2d)
> /commons-net/commons-net/3.4/commons-net-3.4-sources.jar
> (SHA1: 08439b8f20d7bdec502423732798a639501732c8)
> /commons-net/commons-net/3.4/commons-net-3.4-examples.jar
> (SHA1: 33abb13d790501fc9e4e77a40425bc381d39b9ca)
> 
>   Details of changes since 3.3 are in the release notes:
> https://dist.apache.org/repos/dist/dev/commons/net/RELEASE-NOTES.txt
> http://people.apache.org/~sebb/net-3.4-RC2/changes-report.html
> 
>   I have tested this with JDK 1.6, 1.7, 1.8 using maven3.
> 
>   The tag is here:
> http://svn.apache.org/repos/asf/commons/proper/net/tags/NET_3_4_RC2/
> (svn 1715635)
> 
>   N.B. the SVN revision is required because SVN tags are not immutable.
> 
>   Site:
> http://people.apache.org/~sebb/net-3.4-RC2/
>   (note some *relative* links are broken and the 3.4 directories are
>   not yet created - these will be OK once the site is deployed)
> 
>   download_net.cgi does not work, but the template can be checked at
> http://people.apache.org/~sebb/net-3.4-RC2/download_net.html
> 
>   Clirr Report (compared to 3.3):
> http://people.apache.org/~sebb/net-3.4-RC2/clirr-report.html
> 
>   Note that adding methods to an interface is binary-compatible, but
> not source-compatible
>   This change is documented in the Release Notes
> 
>   RAT Report:
> http://people.apache.org/~sebb/net-3.4-RC2/rat-report.html
> 
>   KEYS:
>   https://www.apache.org/dist/commons/KEYS
> 
>   Please review the release candidate and vote.
>   This vote will close no sooner that 72 hours from now, i.e. after
> 14:00 GMT 22-Nov 2015

Tested with JDK 1.7. Everything was running fine.

There are a few findbugs errors that could be fixed as well.

[X] +1 Release these artifacts

>   [ ] +0 OK, but...
>   [ ] -0 OK, but really should fix...
>   [ ] -1 I oppose this release because...
> 
>   Thanks!
> 
>   Sebb
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Validator 1.5.0 based on RC1

[Thomas Neidhart]

On 11/19/2015 10:48 AM, sebb wrote:
> It's probably about time to release VALIDATOR.
> There have been quite a few improvements and fixes since the last version.
> 
> ==
> 
> Validator 1.5.0 RC1 is available for review here:
> https://dist.apache.org/repos/dist/dev/commons/validator/ (svn
> revision 11236)
> 
> ./binaries/commons-validator-1.5.0-bin.tar.gz.sha1:d1ec43eee78475675787db4ce478f03e9089a38f
> ./binaries/commons-validator-1.5.0-bin.zip.sha1:de3664516fdf68667d10b8d060eebd0cf1e349c5
> ./source/commons-validator-1.5.0-src.tar.gz.sha1:737f4df692f3d64bd8c0c6ba6c39e1a7533309c0
> ./source/commons-validator-1.5.0-src.zip.sha1:690c353b52359e75a6b5f0bd1b0cf0f34067c544
> 
>   Maven artifacts are here:
> 
> https://repository.apache.org/content/repositories/orgapachecommons-1118/commons-validator/commons-validator/1.5.0/
> 
> These are the artifacts and their hashes
> 
> /commons-validator/commons-validator/1.5.0/commons-validator-1.5.0-tests.jar
> (SHA1: 371f755d033b4a5d4a8b799e453e27e8211179a7)
> /commons-validator/commons-validator/1.5.0/commons-validator-1.5.0.pom
> (SHA1: 4050c881a79cd5e621339c749ef03b47eb851a4a)
> /commons-validator/commons-validator/1.5.0/commons-validator-1.5.0.jar
> (SHA1: 26b98c21975c5caf6bf34bacdeb7183be7a65970)
> /commons-validator/commons-validator/1.5.0/commons-validator-1.5.0-javadoc.jar
> (SHA1: f3f3364cb4b3e9d9b0362b2bb2770a579ec76a64)
> /commons-validator/commons-validator/1.5.0/commons-validator-1.5.0-sources.jar
> (SHA1: 96c77b61d4ed9ca3b58a7a40bac7fa98a09f5c2f)
> /commons-validator/commons-validator/1.5.0/commons-validator-1.5.0-test-sources.jar
> (SHA1: 68e7c52affe4fdc98d4431e6b9ffcc30969279a1)
> 
>   Details of changes since 1.4.1 are in the release notes:
> https://dist.apache.org/repos/dist/dev/commons/validator/RELEASE-NOTES.txt
> http://people.apache.org/~sebb/validator-1.5.0-RC1/changes-report.html
> 
>   I have tested this with JDK 1.6, 1.7, 1.8 using maven3.
> 
>   The tag is here:
> 
> http://svn.apache.org/repos/asf/commons/proper/validator/tags/VALIDATOR_1_5_0_RC1/
> (svn 1715135)
>   N.B. the SVN revision is required because SVN tags are not immutable.
> 
>   Site:
> http://people.apache.org/~sebb/validator-1.5.0-RC1/
>   (note some *relative* links are broken and the 1.5.0 directories are
>   not yet created - these will be OK once the site is deployed)
> 
>   Clirr Report (compared to 1.4.1):
> http://people.apache.org/~sebb/validator-1.5.0-RC1/clirr-report.html
> 
>   RAT Report:
> http://people.apache.org/~sebb/validator-1.5.0-RC1/rat-report.html
> 
>   KEYS:
>   https://www.apache.org/dist/commons/KEYS
> 
>   Please review the release candidate and vote.
>   This vote will close no sooner that 72 hours from now, i.e. after
> 10:00 GMT 22-Nov 2015
> 

I have first tested to build the site from the source archive using JDK
1.6, but it failed due to a problem with the used checkstyle plugin
(6.2). It obviously is compiled for Java 7 (class version 51).

This will certainly affect other components as well, but not a blocker
for the release imho.

Afterwards tested with JDK 1.7 and everything was fine.

There are a few checkstyle and findbugs errors which could be fixed as well.

[X] +1 Release these artifacts

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[VOTE] Release Commons Collections 4.1 Based on RC1

[Thomas Neidhart]

Hi all,

we have accumulated enough changes since the last 4.0 release as well as
we need to provide a fix for the known remote code exploit via java
de-serialization. Therefore, I would like to start a vote to release
Commons Collections 4.1 based on RC1.

Note:

The fix for the security related issue results in Clirr errors as unsafe
classes in the functor package do not implement the Serializable
interface anymore. This is mentioned in the release notes.


Collections 4.1 RC1 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/collections/
(svn revision 11263)

Maven artifacts are here:


https://repository.apache.org/content/repositories/orgapachecommons-1122/org/apache/commons/commons-collections4/4.1/

Details of changes since 4.0 are in the release notes:


https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt


http://people.apache.org/builds/commons/collections/4.1/RC1/changes-report.html

The tag is here:


https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_4_1_RC1
(svn revision 1715703)

Site:
http://people.apache.org/builds/commons/collections/4.1/RC1/

Clirr Report (compared to 4.0):


http://people.apache.org/builds/commons/collections/4.1/RC1/clirr-report.html

RAT Report:


http://people.apache.org/builds/commons/collections/4.1/RC1/rat-report.html

KEYS:
https://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.

This vote will close no sooner that 72 hours from now, i.e. after 2400
GMT 25-November 2015

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [COLLECTIONS] Release road map for 4.1

[Thomas Neidhart]

On 11/21/2015 11:45 AM, Uwe Barthel wrote:
> Hi,
> 
> How is the road map for the 4.1 release or is there a plan to provide 4.0.1?

the plan is to start the release process tomorrow.

There is only one issue open that needs to be resolved for a 4.1
release, and I am working on it.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[collections] Review of MultiValuedMap interface & implementations

[Thomas Neidhart]

Hi,

we had to remove the use of reflection in the MultiValuedMap
implementations (due to COLLECTIONS-580), which required a huge effort
to refactor the various implementations.

Now, there are only strongly typed implementations:

  * ArrayListValuedHashMap
  * HashSetValuedHashMap

that do not rely on reflection to create value collections.
This means we loose some flexibility that we had before (there was a
MultiValuedHashMap implementation that would take a Factory to create
value collections), but it is also somehow simpler and cleaner that way.

I would like to ask people to review this interface/package and give
already feedback on class names, ...

There are still a few todos (mostly in the AbstractMultiValuedMap class)
that I need to resolve before I can make a new release, but hopefully
this should not take long anymore.

This is the last issue to resolve before I cut a new 4.1 release that
will also include the fixed for COLLECTIONS-580.

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ANNOUNCMENT] Apache Commons Collections 3.2.2 Released

[Thomas Neidhart]

The problem is that the resulting manifest will be different for the ant
and maven builds (as for most other components).

Thomas

On Mon, Nov 16, 2015 at 12:18 PM, Emmanuel Bourg  wrote:

> Le 16/11/2015 12:07, Thomas Neidhart a écrit :
> > Would it be much effort to switch to a maven-style build as it is done
> > already for the commons-lang3 package in debian?
>
> It isn't difficult but the issue is the backport to the previous Debian
> releases, the delta with the released versions should be minimal and a
> change of the build system is usually frowned upon.
>
> I'll just add back the manifest to the Debian package, but if another
> 3.2.x release has to be cut it would be nice to have the manifest back
> in the Apache Commons release.
>
> Emmanuel Bourg
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [ANNOUNCMENT] Apache Commons Collections 3.2.2 Released

[Thomas Neidhart]

Would it be much effort to switch to a maven-style build as it is done
already for the commons-lang3 package in debian?
This release of commons-collections 3.2.2 depends on the latest
commons-parent, so the build should be pretty much the same as for lang
(especially as there are no special plugins configured).

Thomas

On Mon, Nov 16, 2015 at 11:46 AM, Thomas Neidhart  wrote:

> Hi,
>
> I did not test the ant build, but I explicitly removed the MANIFEST.MF as
> this has to be created by the relevant plugins as setup in the
> commons-parent.pom (especially for proper OSGI metadata).
>
> I will take a look this evening how to mitigate this.
>
> Thomas
>
> On Mon, Nov 16, 2015 at 11:19 AM, Emmanuel Bourg 
> wrote:
>
>> Thank you for driving this update Thomas. I noticed too late that
>> src/conf/MANIFEST.MF is missing from the sources, this breaks the Ant
>> build (which is still used in Debian).
>>
>> Emmanuel Bourg
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>>
>

Re: [ANNOUNCMENT] Apache Commons Collections 3.2.2 Released

[Thomas Neidhart]

Hi,

I did not test the ant build, but I explicitly removed the MANIFEST.MF as
this has to be created by the relevant plugins as setup in the
commons-parent.pom (especially for proper OSGI metadata).

I will take a look this evening how to mitigate this.

Thomas

On Mon, Nov 16, 2015 at 11:19 AM, Emmanuel Bourg  wrote:

> Thank you for driving this update Thomas. I noticed too late that
> src/conf/MANIFEST.MF is missing from the sources, this breaks the Ant
> build (which is still used in Debian).
>
> Emmanuel Bourg
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [ANNOUNCMENT] Apache Commons Collections 3.2.2 Released

[Thomas Neidhart]

On 11/15/2015 11:23 PM, Thomas Neidhart wrote:

should have been an ANNOUNCEMENT, but it's late, sorry.

Thomas


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[ANNOUNCMENT] Apache Commons Collections 3.2.2 Released

[Thomas Neidhart]

The Apache Commons team is pleased to announce the release of Apache
Commons Collections 3.2.2. The release is available for download at


http://commons.apache.org/proper/commons-collections/download_collections.cgi

Apache Commons Collections is a project to develop and maintain
collection classes based on and inspired by the JDK collection framework.

This Collections 3.2.2 release is a security and bugfix release, fixing
several bugs present in the previous releases of the 3.2 branch.

Additionally, this release provides a mitigation for a known remote code
exploitation via the standard java object serialization mechanism. By
default, serialization support for unsafe classes in the functor package
is disabled and will result in an exception when either trying to
serialize or de-serialize an instance of these classes. For more
details, please refer to COLLECTIONS-580.

All users are strongly encouraged to updated to this release.

See the release-notes at

 http://commons.apache.org/proper/commons-collections/release_3_2_2.html
 http://www.apache.org/dist/commons/collections/RELEASE-NOTES-3.2.2.txt

for a full list of changes.

Please verify signatures using the KEYS file available at the above
location when downloading the release.

For complete information on collections, including instructions on how
to submit bug reports, patches, or suggestions for improvement, see the
Apache Commons Collections website:

 http://commons.apache.org/proper/commons-collections/

Thomas, on behalf of the Apache Commons team

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [RESULT][VOTE] Release Commons Collections 3.2.2 Based on RC3

[Thomas Neidhart]

On 11/14/2015 04:20 PM, Uwe Barthel wrote:
> Thx Thomas.
> 
> The fix for the Java serialization vulnerability is on the way.
> Now should we add some information on
> http://commons.apache.org/security.html like Commons Compress did?

yes, we will do something similar.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[RESULT][VOTE] Release Commons Collections 3.2.2 Based on RC3

[Thomas Neidhart]

On 11/13/2015 12:31 AM, Thomas Neidhart wrote:
> Hi all,

[snip]

> Considering that this is a security related release and that RC2 did not
> show any functional problems with the release, I plan to close this vote
> in 24h from now, i.e. after 0100 GMT 14-November 2015

Here is a tally of the VOTE

Commons PMC:
 +1 from Luc, Joerg, Gary, Stefan, Thomas

No other votes have been recorded.

This VOTE, therefore, passes.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Collections 3.2.2 Based on RC3

[Thomas Neidhart]

On 11/13/2015 08:26 PM, Gary Gregory wrote:
> +1
> 
> Tested with src zip.
> 
> BUT:
> 
> - The site Javadoc link is labeled "3.2.1" (fixed in
> https://svn.apache.org/repos/asf/commons/proper/collections/branches/COLLECTIONS_3_2_X
> )
> - The site history does not mentioned (fixed in svn)

as I said the site will not be published from the 3.2.2 release but from
the 4.X branch.

> ASC OK, MD5 OK, SHA1 OK. Everyone's checking these, right?
> 
> Reports OK.
> 
> Tested building with:
> 
> Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06;
> 2015-04-22T04:57:37-07:00)
> Maven home: C:\Java\apache-maven-3.3.3\bin\..
> Java version: 1.7.0_79, vendor: Oracle Corporation
> Java home: C:\Program Files\Java\jdk1.7.0_79\jre
> Default locale: en_US, platform encoding: Cp1252
> OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"
> 
> and:
> 
> Apache Ant(TM) version 1.9.6 compiled on June 29 2015
> 
> Gary
> 
> On Thu, Nov 12, 2015 at 3:31 PM, Thomas Neidhart 
> wrote:
> 
>> Hi all,
>>
>> in order to provide a work-around for the known remote code exploit via
>> java de-serialization of malicious InvokerTransformer instances, I would
>> like to start a vote to release Commons Collections 3.2.2 based on RC3.
>>
>> Notes:
>>
>>  * the site will not be published, it just serves as a reference to
>> access the various reports. After a successful vote, the current 4.X
>> branch site will be updated with relevant information and published.
>>
>>  * some tests might fail with various IBM JDK 6 JREs, these are known
>> issues and have been worked-around in the 4.X branch but are not
>> back-ported to this release.
>>
>>  * Collections 3.2.2 can not be compiled with JDK 8 due to a name clash
>> with a newly introduced default method in the Map interface.
>>
>>  * the collections-testframework.jar that has been published in previous
>> versions is not included in this release
>>
>> Changes from RC2:
>>
>>  * fixed false positives in RAT report
>>  * fixed test execution and compilation problems with JDK 1.4 and 1.5
>>
>> Changes from RC1:
>>
>>  * fixed RAT report
>>  * fixed NOTICE file
>>  * improve the security fix: it has been made symmetric in the sense
>>that also the serialization of an unsafe class is disabled by
>>default and will result in an exception
>>  * changed the system property to re-enable serialization of unsafe
>>classes. It is now
>>"org.apache.commons.collections.enableUnsafeSerialization"
>>  * all classes in the functor package which (based on current
>>knowledge) have to be considered unsafe cannot be serialized/
>>de-serialized any more by default. This includes the following
>>classes:
>>
>>  ** CloneTransformer
>>  ** PrototypeFactory (inner classes
>>   PrototypeCloneFactory and
>>   PrototypeSerializationFactory)
>>  ** InstantiateFactory
>>  ** InstantiateTransformer
>>  ** ForClosure
>>  ** WhileClosure
>>  ** InvokerTransformer
>>
>>
>>
>> Collections 3.2.2 RC3 is available for review here:
>> https://dist.apache.org/repos/dist/dev/commons/collections/
>> (svn revision 11167)
>>
>> Maven artifacts are here:
>>
>>
>> https://repository.apache.org/content/repositories/orgapachecommons-1117/commons-collections/commons-collections/3.2.2/
>>
>> Details of changes since 3.2.1 are in the release notes:
>>
>>
>> https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt
>>
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC3/changes-report.html
>>
>> The tag is here:
>>
>>
>> https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC3
>> (svn revision 1714131)
>>
>> Site:
>> http://people.apache.org/builds/commons/collections/3.2.2/RC3/
>>
>> Clirr Report (compared to 3.2.1):
>>
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC3/clirr-report.html
>>
>> RAT Report:
>>
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC3/rat-report.html
>>
>> KEYS:
>>   https://www.apache.org/dist/commons/KEYS
>>
>> Please review the release candidate and vote.
>>
>>
>> Considering that this is a security related release and that RC2 did not
>> show any functional problems with the release, I plan to close this vote
>> in 24h from now, i.e. after 0100 GMT 14-November 2015
>>
>>   [ ] +1 Release these artifacts
>>   [ ] +0 OK, but...
>>   [ ] -0 OK, but really should fix...
>>   [ ] -1 I oppose this release because...
>>
>> Thanks,
>>
>> Thomas
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>>
> 
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[VOTE] Release Commons Collections 3.2.2 Based on RC3

[Thomas Neidhart]

Hi all,

in order to provide a work-around for the known remote code exploit via
java de-serialization of malicious InvokerTransformer instances, I would
like to start a vote to release Commons Collections 3.2.2 based on RC3.

Notes:

 * the site will not be published, it just serves as a reference to
access the various reports. After a successful vote, the current 4.X
branch site will be updated with relevant information and published.

 * some tests might fail with various IBM JDK 6 JREs, these are known
issues and have been worked-around in the 4.X branch but are not
back-ported to this release.

 * Collections 3.2.2 can not be compiled with JDK 8 due to a name clash
with a newly introduced default method in the Map interface.

 * the collections-testframework.jar that has been published in previous
versions is not included in this release

Changes from RC2:

 * fixed false positives in RAT report
 * fixed test execution and compilation problems with JDK 1.4 and 1.5

Changes from RC1:

 * fixed RAT report
 * fixed NOTICE file
 * improve the security fix: it has been made symmetric in the sense
   that also the serialization of an unsafe class is disabled by
   default and will result in an exception
 * changed the system property to re-enable serialization of unsafe
   classes. It is now
   "org.apache.commons.collections.enableUnsafeSerialization"
 * all classes in the functor package which (based on current
   knowledge) have to be considered unsafe cannot be serialized/
   de-serialized any more by default. This includes the following
   classes:

 ** CloneTransformer
 ** PrototypeFactory (inner classes
  PrototypeCloneFactory and
  PrototypeSerializationFactory)
 ** InstantiateFactory
 ** InstantiateTransformer
 ** ForClosure
 ** WhileClosure
 ** InvokerTransformer



Collections 3.2.2 RC3 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/collections/
(svn revision 11167)

Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1117/commons-collections/commons-collections/3.2.2/

Details of changes since 3.2.1 are in the release notes:

https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt

http://people.apache.org/builds/commons/collections/3.2.2/RC3/changes-report.html

The tag is here:

https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC3
(svn revision 1714131)

Site:
http://people.apache.org/builds/commons/collections/3.2.2/RC3/

Clirr Report (compared to 3.2.1):

http://people.apache.org/builds/commons/collections/3.2.2/RC3/clirr-report.html

RAT Report:

http://people.apache.org/builds/commons/collections/3.2.2/RC3/rat-report.html

KEYS:
  https://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.


Considering that this is a security related release and that RC2 did not
show any functional problems with the release, I plan to close this vote
in 24h from now, i.e. after 0100 GMT 14-November 2015

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Collections 3.2.2 Based on RC2

[Thomas Neidhart]

On 11/12/2015 07:14 PM, Jörg Schaible wrote:
> Hi Thomas,
> 
> Thomas Neidhart wrote:
> 
>> Hi all,
>>
>> in order to provide a work-around for the known remote code exploit via
>> java de-serialization of malicious InvokerTransformer instances, I would
>> like to start a vote to release Commons Collections 3.2.2 based on RC2.
>>
>> Notes:
>>
>>  * the site will not be published, it just serves as a reference to
>> access the various reports. After a successful vote, the current 4.X
>> branch site will be updated with relevant information and published.
>>
>>  * some tests might fail with various IBM JDK 6 JREs, these are known
>> issues and have been worked-around in the 4.X branch but are not
>> back-ported to this release.
>>
>>  * Collections 3.2.2 can not be compiled with JDK 8 due to a name clash
>> with a newly introduced default method in the Map interface.
>>
>>  * the collections-testframework.jar that has been published in previous
>> versions is not included in this release
>>
>>
>> Changes from RC1:
>>
>>  * fixed RAT report
>>  * fixed NOTICE file
>>  * improve the security fix: it has been made symmetric in the sense
>>that also the serialization of an unsafe class is disabled by
>>default and will result in an exception
>>  * changed the system property to re-enable serialization of unsafe
>>classes. It is now
>>"org.apache.commons.collections.enableUnsafeSerialization"
>>  * all classes in the functor package which (based on current
>>knowledge) have to be considered unsafe cannot be serialized/
>>de-serialized any more by default. This includes the following
>>classes:
>>
>>  ** CloneTransformer
>>  ** PrototypeFactory (inner classes
>>   PrototypeCloneFactory and
>>   PrototypeSerializationFactory)
>>  ** InstantiateFactory
>>  ** InstantiateTransformer
>>  ** ForClosure
>>  ** WhileClosure
>>  ** InvokerTransformer
>>
>>
>>
>> Collections 3.2.2 RC2 is available for review here:
>> https://dist.apache.org/repos/dist/dev/commons/collections/
>> (svn revision 11147)
>>
>> Maven artifacts are here:
>>
>>
> https://repository.apache.org/content/repositories/orgapachecommons-1116/commons-collections/commons-collections/3.2.2/
>>
>> Details of changes since 3.2.1 are in the release notes:
>>
>> https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC2/changes-report.html
>>
>> The tag is here:
>>
>>
> https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC2
>> (svn revision 1713883)
>>
>> Site:
>> http://people.apache.org/builds/commons/collections/3.2.2/RC2/
>>
>> Clirr Report (compared to 3.2.1):
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC2/clirr-report.html
>>
>> RAT Report:
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC2/rat-report.html
>>
>> KEYS:
>>   https://www.apache.org/dist/commons/KEYS
>>
>> Please review the release candidate and vote.
>>
>>
>> Considering that this is a security related release and that RC1 did not
>> show any functional problems with the release, I plan to close this vote
>> in 24 from now, i.e. after 1800 GMT 12-November 2015
>>
>>   [ ] +1 Release these artifacts
>>   [ ] +0 OK, but...
>>   [ ] -0 OK, but really should fix...
>>   [ ] -1 I oppose this release because...
> 
> -1,
> 
> sorry, but there's a regression
> 
> The package claims to be compatible with Java 1.3. Well, I don't have 1.3 
> anymore, but 1.4. And I can build CC-3.2.1 and run all tests with Blackdown 
> JDK 1.4 and Maven 2.0.11.
> 
> For CC-3.2.2 I have to use at least Java 5 and Maven 3.0(.5):
> 
> - Using java-1.4 profile: Build fails, because tests no longer compile
> - Sun JDK 1.5: TestAllPackages fails due to SecurityException:
> == %< ==
> Running org.apache.commons.collections.TestAllPackages
> java.lang.SecurityException
> at 
> org.apache.commons.collections.TestExtendedProperties$1.checkPropertyAccess(TestExtendedProperties.java:322)
> at java.lang.System.getProperty(System.java:628)
> at 
> sun.security.action.GetPropertyAction.run(GetPropertyAction.java:66)
> at java.security.AccessController.doPrivileged(Na

[CANCEL][VOTE] Release Commons Collections 3.2.2 Based on RC2

[Thomas Neidhart]

On 11/11/2015 05:27 PM, Thomas Neidhart wrote:
> Hi all,
> 
> in order to provide a work-around for the known remote code exploit via
> java de-serialization of malicious InvokerTransformer instances, I would
> like to start a vote to release Commons Collections 3.2.2 based on RC2.
> 
> Notes:
> 
>  * the site will not be published, it just serves as a reference to
> access the various reports. After a successful vote, the current 4.X
> branch site will be updated with relevant information and published.
> 
>  * some tests might fail with various IBM JDK 6 JREs, these are known
> issues and have been worked-around in the 4.X branch but are not
> back-ported to this release.
> 
>  * Collections 3.2.2 can not be compiled with JDK 8 due to a name clash
> with a newly introduced default method in the Map interface.
> 
>  * the collections-testframework.jar that has been published in previous
> versions is not included in this release
> 
> 
> Changes from RC1:
> 
>  * fixed RAT report
>  * fixed NOTICE file
>  * improve the security fix: it has been made symmetric in the sense
>that also the serialization of an unsafe class is disabled by
>default and will result in an exception
>  * changed the system property to re-enable serialization of unsafe
>classes. It is now
>"org.apache.commons.collections.enableUnsafeSerialization"
>  * all classes in the functor package which (based on current
>knowledge) have to be considered unsafe cannot be serialized/
>de-serialized any more by default. This includes the following
>classes:
> 
>  ** CloneTransformer
>  ** PrototypeFactory (inner classes
>   PrototypeCloneFactory and
>   PrototypeSerializationFactory)
>  ** InstantiateFactory
>  ** InstantiateTransformer
>  ** ForClosure
>  ** WhileClosure
>  ** InvokerTransformer
> 
> 
> 
> Collections 3.2.2 RC2 is available for review here:
> https://dist.apache.org/repos/dist/dev/commons/collections/
> (svn revision 11147)
> 
> Maven artifacts are here:
> 
> https://repository.apache.org/content/repositories/orgapachecommons-1116/commons-collections/commons-collections/3.2.2/
> 
> Details of changes since 3.2.1 are in the release notes:
> 
> https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt
> 
> http://people.apache.org/builds/commons/collections/3.2.2/RC2/changes-report.html
> 
> The tag is here:
> 
> https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC2
> (svn revision 1713883)
> 
> Site:
> http://people.apache.org/builds/commons/collections/3.2.2/RC2/
> 
> Clirr Report (compared to 3.2.1):
> 
> http://people.apache.org/builds/commons/collections/3.2.2/RC2/clirr-report.html
> 
> RAT Report:
> 
> http://people.apache.org/builds/commons/collections/3.2.2/RC2/rat-report.html
> 
> KEYS:
>   https://www.apache.org/dist/commons/KEYS
> 
> Please review the release candidate and vote.
> 
> 
> Considering that this is a security related release and that RC1 did not
> show any functional problems with the release, I plan to close this vote
> in 24 from now, i.e. after 1800 GMT 12-November 2015
> 
>   [ ] +1 Release these artifacts
>   [ ] +0 OK, but...
>   [ ] -0 OK, but really should fix...
>   [ ] -1 I oppose this release because...

The vote is cancelled to fix the test errors on some java versions.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[VOTE] Release Commons Collections 3.2.2 Based on RC2

[Thomas Neidhart]

Hi all,

in order to provide a work-around for the known remote code exploit via
java de-serialization of malicious InvokerTransformer instances, I would
like to start a vote to release Commons Collections 3.2.2 based on RC2.

Notes:

 * the site will not be published, it just serves as a reference to
access the various reports. After a successful vote, the current 4.X
branch site will be updated with relevant information and published.

 * some tests might fail with various IBM JDK 6 JREs, these are known
issues and have been worked-around in the 4.X branch but are not
back-ported to this release.

 * Collections 3.2.2 can not be compiled with JDK 8 due to a name clash
with a newly introduced default method in the Map interface.

 * the collections-testframework.jar that has been published in previous
versions is not included in this release


Changes from RC1:

 * fixed RAT report
 * fixed NOTICE file
 * improve the security fix: it has been made symmetric in the sense
   that also the serialization of an unsafe class is disabled by
   default and will result in an exception
 * changed the system property to re-enable serialization of unsafe
   classes. It is now
   "org.apache.commons.collections.enableUnsafeSerialization"
 * all classes in the functor package which (based on current
   knowledge) have to be considered unsafe cannot be serialized/
   de-serialized any more by default. This includes the following
   classes:

 ** CloneTransformer
 ** PrototypeFactory (inner classes
  PrototypeCloneFactory and
  PrototypeSerializationFactory)
 ** InstantiateFactory
 ** InstantiateTransformer
 ** ForClosure
 ** WhileClosure
 ** InvokerTransformer



Collections 3.2.2 RC2 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/collections/
(svn revision 11147)

Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1116/commons-collections/commons-collections/3.2.2/

Details of changes since 3.2.1 are in the release notes:

https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt

http://people.apache.org/builds/commons/collections/3.2.2/RC2/changes-report.html

The tag is here:

https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC2
(svn revision 1713883)

Site:
http://people.apache.org/builds/commons/collections/3.2.2/RC2/

Clirr Report (compared to 3.2.1):

http://people.apache.org/builds/commons/collections/3.2.2/RC2/clirr-report.html

RAT Report:

http://people.apache.org/builds/commons/collections/3.2.2/RC2/rat-report.html

KEYS:
  https://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.


Considering that this is a security related release and that RC1 did not
show any functional problems with the release, I plan to close this vote
in 24 from now, i.e. after 1800 GMT 12-November 2015

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[CANCEL][VOTE] Release Commons Collections 3.2.2 Based on RC1

[Thomas Neidhart]

On 11/09/2015 11:37 PM, Thomas Neidhart wrote:
> Hi all,
> 
> in order to provide a work-around for the known remote code exploit via
> java de-serialization of malicious InvokerTransformer instances, I would
> like to start a vote to release Commons Collections 3.2.2 based on RC1.
> 
> I would kindly ask people to review the RC especially wrt the following
> topics:
> 
>  * OSGI compatibility
>  * reproducing the exploits and verifying that it provides protection
>  * any kind of regression that this release might create with existing
> applications
> 
> Notes:
> 
>  * the site will not be published, it just serves as a reference to
> access the various reports. After a successful vote, the current 4.X
> branch site will be updated with relevant information and published.
> 
>  * some tests might fail with various IBM JDK 6 JREs, these are known
> issues and have been worked-around in the 4.X branch but are not
> back-ported to this release.
> 
> 
> Collections 3.2.2 RC1 is available for review here:
> https://dist.apache.org/repos/dist/dev/commons/collections/
> (svn revision 11092)
> 
> Maven artifacts are here:
> 
> https://repository.apache.org/content/repositories/orgapachecommons-1115/commons-collections/commons-collections/3.2.2/
> 
> Details of changes since 3.2.1 are in the release notes:
> 
> https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt
> 
> http://people.apache.org/builds/commons/collections/3.2.2/RC1/changes-report.html
> 
> The tag is here:
> 
> https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC1
> (svn revision 1713561)
> 
> Site:
> http://people.apache.org/builds/commons/collections/3.2.2/RC1/
> 
> Clirr Report (compared to 3.2.1):
> 
> http://people.apache.org/builds/commons/collections/3.2.2/RC1/clirr-report.html
> 
> RAT Report:
> 
> http://people.apache.org/builds/commons/collections/3.2.2/RC1/rat-report.html
> 
> KEYS:
>   https://www.apache.org/dist/commons/KEYS
> 
> Please review the release candidate and vote.
> 
> This vote will close no sooner that 72 hours from now, i.e. after 2300
> GMT 12-November 2015
> 
>   [ ] +1 Release these artifacts
>   [ ] +0 OK, but...
>   [ ] -0 OK, but really should fix...
>   [ ] -1 I oppose this release because...

Hi,

after careful consideration, I decided to cancel this vote to apply the
fix for InvokerTransformer to all classes in the functor package that
are serializable and use reflection.

Additionally, the fix will be applied in a symmetric way, i.e.
serialization of an unsafe class will be blocked as well.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Collections 3.2.2 Based on RC1

[Thomas Neidhart]

On 11/10/2015 11:41 PM, Gary Gregory wrote:
> On Tue, Nov 10, 2015 at 2:22 PM, Thomas Neidhart 
> wrote:
> 
>> On 11/10/2015 10:52 PM, Gary Gregory wrote:
>>> Hi all:
>>>
>>> -1
>>>
>>> Sorry, the RAT failure needs to be handled one way or another: exclude
>> the
>>> files or add headers:
>>>
>>> Unapproved licenses:
>>>
>>>   data/test/NullComparator.version2.obj1
>>>   data/test/NullComparator.version2.obj2
>>>   xdocs/style/project.css
>>>
>>>
>>> I imagine the obj files can be excluded but the CSS file can just have a
>>> header added, just like
>>>
>> https://svn.apache.org/repos/asf/commons/proper/daemon/trunk/src/docs/daemon.css
>>>
>>> It's just messy to rush this through without dotting the i's and so on.
>>
>> yeah, I did not see the 2 NullComparator files as the problem appears
>> only on Windows. The same happened for the Collections 4 release, and I
>> forgot about it.
>>
>> @css: wtf, are you serious to vote with -1 because of that and complain
>> about the RC being messy? I mean, I can handle it if there are real
>> issues to be fixed, and I had planned to cancel the VOTE anyways to make
>> some more adjustments but something like that is just ridiculous. Just
>> take a look at some other published commons releases and count the
>> number of RAT errors, even for source files.
>>
> 
> Sorry, two wrongs to do make a right. If other Commons components have made
> a mess of specific releases in the past, then that's sad. Either the RAT
> report is clean or it is not. If it is clean, I have to assume that
> exclusions in the POM for specific files or types of files have been done
> with careful consideration and that I can always go digging in the commit
> log to see a hopefully useful comment as to why the exclusion was made.
> 
> Since this is a release to address a security issue, I would have hoped
> that all details would have been handled with extra care.
> 
> I'd never get away with a sloppy release at work, and I hope I won't have
> to here either.
> 
> In any case, a -1 is not a veto on a vote thread like it is on a commit, so
> this vote may yet pass. It's up to you as the RM to decide what to do.
> 
> I know that cutting releases is still a pain, we have a lot of gymnastics,
> it's not like pushing a button, but that' what we're stuck with for now.

you complain about false positives in a code base that has not been
released in 8 years and call my work messy. I have seen the css alert,
but thought I can safely ignore it, as it is anyway obsolete (pointing
to a non-existing css on the apache homepage).

People blame Apache for not providing a fix in 9 months for a known
exploit and we are arguing about totally unimportant issues.

I explicitly asked for review in areas that *are* important, e.g. OSGI
compatibility, as the build/release chain has changed quite a lot in the
last 8 years, and I wanted verification that the 3.2.2 release can
really be used in all areas. But no, we talk about a missing AL header
in a one line css file.

Frankly, I am pissed because I spent the last days working on this while
my baby is teething and would have certainly better things to do.

I will continue with the release as it is too important, but I am not
sure any more that I want to make another release for commons in the future.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Collections 3.2.2 Based on RC1

[Thomas Neidhart]

On 11/10/2015 10:52 PM, Gary Gregory wrote:
> Hi all:
> 
> -1
> 
> Sorry, the RAT failure needs to be handled one way or another: exclude the
> files or add headers:
> 
> Unapproved licenses:
> 
>   data/test/NullComparator.version2.obj1
>   data/test/NullComparator.version2.obj2
>   xdocs/style/project.css
> 
> 
> I imagine the obj files can be excluded but the CSS file can just have a
> header added, just like
> https://svn.apache.org/repos/asf/commons/proper/daemon/trunk/src/docs/daemon.css
> 
> It's just messy to rush this through without dotting the i's and so on.

yeah, I did not see the 2 NullComparator files as the problem appears
only on Windows. The same happened for the Collections 4 release, and I
forgot about it.

@css: wtf, are you serious to vote with -1 because of that and complain
about the RC being messy? I mean, I can handle it if there are real
issues to be fixed, and I had planned to cancel the VOTE anyways to make
some more adjustments but something like that is just ridiculous. Just
take a look at some other published commons releases and count the
number of RAT errors, even for source files.

Thomas

> 
> There is also the issue of the possibly wrong revision being tagged or
> being used in the VOTE email thread. That can be fixed for RC2 as well.
> 
> Gary
> 
> On Mon, Nov 9, 2015 at 2:37 PM, Thomas Neidhart 
> wrote:
> 
>> Hi all,
>>
>> in order to provide a work-around for the known remote code exploit via
>> java de-serialization of malicious InvokerTransformer instances, I would
>> like to start a vote to release Commons Collections 3.2.2 based on RC1.
>>
>> I would kindly ask people to review the RC especially wrt the following
>> topics:
>>
>>  * OSGI compatibility
>>  * reproducing the exploits and verifying that it provides protection
>>  * any kind of regression that this release might create with existing
>> applications
>>
>> Notes:
>>
>>  * the site will not be published, it just serves as a reference to
>> access the various reports. After a successful vote, the current 4.X
>> branch site will be updated with relevant information and published.
>>
>>  * some tests might fail with various IBM JDK 6 JREs, these are known
>> issues and have been worked-around in the 4.X branch but are not
>> back-ported to this release.
>>
>>
>> Collections 3.2.2 RC1 is available for review here:
>> https://dist.apache.org/repos/dist/dev/commons/collections/
>> (svn revision 11092)
>>
>> Maven artifacts are here:
>>
>>
>> https://repository.apache.org/content/repositories/orgapachecommons-1115/commons-collections/commons-collections/3.2.2/
>>
>> Details of changes since 3.2.1 are in the release notes:
>>
>>
>> https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt
>>
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC1/changes-report.html
>>
>> The tag is here:
>>
>>
>> https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC1
>> (svn revision 1713561)
>>
>> Site:
>> http://people.apache.org/builds/commons/collections/3.2.2/RC1/
>>
>> Clirr Report (compared to 3.2.1):
>>
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC1/clirr-report.html
>>
>> RAT Report:
>>
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC1/rat-report.html
>>
>> KEYS:
>>   https://www.apache.org/dist/commons/KEYS
>>
>> Please review the release candidate and vote.
>>
>> This vote will close no sooner that 72 hours from now, i.e. after 2300
>> GMT 12-November 2015
>>
>>   [ ] +1 Release these artifacts
>>   [ ] +0 OK, but...
>>   [ ] -0 OK, but really should fix...
>>   [ ] -1 I oppose this release because...
>>
>> Thanks,
>>
>> Thomas
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>>
> 
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Collections 3.2.2 Based on RC1

[Thomas Neidhart]

On 11/10/2015 09:59 PM, Luc Maisonobe wrote:
> Le 09/11/2015 23:37, Thomas Neidhart a écrit :
>> Hi all,
>>
>> in order to provide a work-around for the known remote code exploit via
>> java de-serialization of malicious InvokerTransformer instances, I would
>> like to start a vote to release Commons Collections 3.2.2 based on RC1.
>>
>> I would kindly ask people to review the RC especially wrt the following
>> topics:
>>
>>  * OSGI compatibility
>>  * reproducing the exploits and verifying that it provides protection
>>  * any kind of regression that this release might create with existing
>> applications
>>
>> Notes:
>>
>>  * the site will not be published, it just serves as a reference to
>> access the various reports. After a successful vote, the current 4.X
>> branch site will be updated with relevant information and published.
>>
>>  * some tests might fail with various IBM JDK 6 JREs, these are known
>> issues and have been worked-around in the 4.X branch but are not
>> back-ported to this release.
>>
>>
>> Collections 3.2.2 RC1 is available for review here:
>> https://dist.apache.org/repos/dist/dev/commons/collections/
>> (svn revision 11092)
>>
>> Maven artifacts are here:
>>
>> https://repository.apache.org/content/repositories/orgapachecommons-1115/commons-collections/commons-collections/3.2.2/
>>
>> Details of changes since 3.2.1 are in the release notes:
>>
>> https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC1/changes-report.html
>>
>> The tag is here:
>>
>> https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC1
>> (svn revision 1713561)
> 
> It seems the revision is 1713556 rather than 1713561.
> It is both what I see in svn and what was used to generate the
> binaries in dist (according to the Implementation-Build element
> in the MANIFEST.MF embedded within the jar).
> 
>>
>> Site:
>> http://people.apache.org/builds/commons/collections/3.2.2/RC1/
>>
>> Clirr Report (compared to 3.2.1):
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC1/clirr-report.html
>>
>> RAT Report:
>>
>> http://people.apache.org/builds/commons/collections/3.2.2/RC1/rat-report.html
> 
> The single file with unknown license in this report is
> xdocs/style.project.css. It is a one line file that imports
> commons-maven.css). The file has been unchanged since April 2008.
> 
> Certainly not a blocker.
> 
>>
>> KEYS:
>>   https://www.apache.org/dist/commons/KEYS
>>
>> Please review the release candidate and vote.
> 
> I first got a compilation error when attempting a compilation with maven
> 3.3.3 and the default JVM on my system (java8 openJDK, on a Debian
> stretch machine)
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR]
> /home/luc/tmp/downloaded-RC/dist/source/commons-collections-3.2.2-src-t/src/java/org/apache/commons/collections/map/MultiValueMap.java:[156,19]
> remove(java.lang.Object,java.lang.Object) in
> org.apache.commons.collections.map.MultiValueMap cannot implement
> remove(java.lang.Object,java.lang.Object) in java.util.Map
>   return type java.lang.Object is not compatible with boolean
> [ERROR]
> /home/luc/tmp/downloaded-RC/dist/source/commons-collections-3.2.2-src-t/src/java/org/apache/commons/collections/MultiMap.java:[69,19]
> remove(java.lang.Object,java.lang.Object) in
> org.apache.commons.collections.MultiMap clashes with
> remove(java.lang.Object,java.lang.Object) in java.util.Map
>   return type java.lang.Object is not compatible with boolean
> [ERROR]
> /home/luc/tmp/downloaded-RC/dist/source/commons-collections-3.2.2-src-t/src/java/org/apache/commons/collections/map/MultiKeyMap.java:[200,19]
> remove(java.lang.Object,java.lang.Object) in
> org.apache.commons.collections.map.MultiKeyMap cannot implement
> remove(java.lang.Object,java.lang.Object) in java.util.Map
>   return type java.lang.Object is not compatible with boolean
> [ERROR]
> /home/luc/tmp/downloaded-RC/dist/source/commons-collections-3.2.2-src-t/src/java/org/apache/commons/collections/MultiHashMap.java:[334,19]
> remove(java.lang.Object,java.lang.Object) in
> org.apache.commons.collections.MultiHashMap cannot implement
> remove(java.lang.Object,java.lang.Object) in java.util.Map
>   return type java.lang.Object is not compatible with boolean
> 
> Then I forced maven to run using java7 and it was fine. The pom does
> specify maven.c

[VOTE] Release Commons Collections 3.2.2 Based on RC1

[Thomas Neidhart]

Hi all,

in order to provide a work-around for the known remote code exploit via
java de-serialization of malicious InvokerTransformer instances, I would
like to start a vote to release Commons Collections 3.2.2 based on RC1.

I would kindly ask people to review the RC especially wrt the following
topics:

 * OSGI compatibility
 * reproducing the exploits and verifying that it provides protection
 * any kind of regression that this release might create with existing
applications

Notes:

 * the site will not be published, it just serves as a reference to
access the various reports. After a successful vote, the current 4.X
branch site will be updated with relevant information and published.

 * some tests might fail with various IBM JDK 6 JREs, these are known
issues and have been worked-around in the 4.X branch but are not
back-ported to this release.


Collections 3.2.2 RC1 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/collections/
(svn revision 11092)

Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1115/commons-collections/commons-collections/3.2.2/

Details of changes since 3.2.1 are in the release notes:

https://dist.apache.org/repos/dist/dev/commons/collections/RELEASE-NOTES.txt

http://people.apache.org/builds/commons/collections/3.2.2/RC1/changes-report.html

The tag is here:

https://svn.apache.org/repos/asf/commons/proper/collections/tags/COLLECTIONS_3_2_2_RC1
(svn revision 1713561)

Site:
http://people.apache.org/builds/commons/collections/3.2.2/RC1/

Clirr Report (compared to 3.2.1):

http://people.apache.org/builds/commons/collections/3.2.2/RC1/clirr-report.html

RAT Report:

http://people.apache.org/builds/commons/collections/3.2.2/RC1/rat-report.html

KEYS:
  https://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.

This vote will close no sooner that 72 hours from now, i.e. after 2300
GMT 12-November 2015

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: invoker-defender Java agent

[Thomas Neidhart]

On 11/09/2015 12:34 PM, Eirik Bjørsnøs wrote:
> Hi,
> 
> Following the "recent" "news" about Java deserialization security issues, I
> decided to create:
> 
> https://github.com/kantega/invoker-defender/
> 
> This is a Java Agent which removes java.io.Serializable from classes known
> to be vulnerable to deserialization attacks. (Including InvokerTransformer)
> 
> I do not in any way consider this a complete solution to the problem since
> it only "fixes" a few well known classes.
> 
> But it might be something people could consider as a mitigation effort
> while vendors/projects work on more long-term fixes.
> 
> Feedback is welcome.

Thanks for sharing your work here.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collections] Review of proposed fix for InvokerTransformer exploit

[Thomas Neidhart]

On Mon, Nov 9, 2015 at 10:37 AM, Emmanuel Bourg  wrote:

> Le 08/11/2015 23:21, Thomas Neidhart a écrit :
>
> > please review the proposed fix for this issue here:
>
> The exception message ends with a comma, is this a typo? I suggest
> mentioning the system property in the message, such that someone
> hitting the exception immediately knows how to work around it.
>

yes, I wanted to add this information, but forgot to finish.
I also need to add a paragraph to the class javadoc outlining the change.


>
> Also:
>
> !"true".equalsIgnoreCase(deserializeProperty)
>
> is shorter than:
>
> deserializeProperty == null ||
> !deserializeProperty.equalsIgnoreCase("true")
>

yes indeed, will change too.

Thanks,

Thomas

[collections] Review of proposed fix for InvokerTransformer exploit

[Thomas Neidhart]

Hi all,

please review the proposed fix for this issue here:

http://svn.apache.org/viewvc?view=revision&revision=1713307

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/08/2015 09:36 PM, James Carman wrote:
> Oh nasty! I must've met, this is quite a fascinating exploit. I'm going to
> do some digging later today when I am at my computer.

I just figured that the xalan code already does have a system property
to prevent translets from being de-serialized:

public final static String DESERIALIZE_TRANSLET =
"jdk.xml.enableTemplatesImplDeserialization";

so a similar solution what we are going to do for collections.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/08/2015 09:11 PM, James Carman wrote:
> How did we get to the point where someone could invoke arbitrary bytecode?

Take a look at class TemplatesImpl in
com.sun.org.apache.xalan.internal.xsltc.trax which is part of the oracle
and openjdk jre.

It is serializable and can load so called Translets which are stored as
byte[] and will be loaded once the newTransformer method is invoked.

So an attacker can store byte code in the array of a serialized
TemplatesImpl object and force its execution via the InvokerTransformer.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/08/2015 08:20 PM, James Carman wrote:
> I think this entire thing can be prevented with a security manager and a
> proper policy in place. Nobody does that, though

You cannot prevent the use of reflection for public methods via a
SecurityManager.

If you then look at the different provided payloads you can see that an
attacker can inject arbitrary bytecode that is being loaded.

How would you prevent that such code is able to do anything harmful,
especially considering that it is being executed in the security context
of some trusted component?

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/08/2015 07:51 PM, James Carman wrote:
> Couldn't they use the same attack vector to set a system property also? I
> do believe that would be possible

for this you need a way to execute code via a de-serialized class.
Right now, the simplest way to do so is via the InvokerTransformer.

There are surely other ways to do so, but if the only available way is
blocked (i.e. InvokerTransformer can not be deserialized), a remote
attacker cannot set a system property via this attack vector.

btw. setting a system property can also be restricted by a SecurityManager.

I am -1 on a programmatic interface, and for the 4.X branch I propose to
remove the serialization support completely.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/08/2015 01:32 PM, Mark Thomas wrote:
> On 08/11/2015 10:18, Thomas Neidhart wrote:
>> On 11/07/2015 11:19 AM, Mark Thomas wrote:
>>> On 07/11/2015 10:13, Thomas Neidhart wrote:
>>>> On 11/07/2015 04:25 AM, Bernd Eckenfels wrote:
>>>>> Hello,
>>>>>
>>>>> I tried to raise that concern in the message already, but it is probably
>>>>> worth repeating it explicitly: this is not a real bug
>>>>> in the Commons-Collection class, and it might not be worse fixing it, as
>>>>> there are possibly tons of other vectors. This was also addressed by the
>>>>> original authors in the talk and even here on Twitter:
>>>>>
>>>>> https://twitter.com/gebl/status/662754611304996866
>>>>>
>>>>> however, as the "foxglove" article shows, people still point at the
>>>>> apache project, and after all it is good pratice to reduce footprints
>>>>> and attack surfaces.
>>>>
>>>> it is clear that the InvokerTransformer by itself does not have a bug,
>>>> but due to the way how java serialization is applied and considering the
>>>> fact that at least collections-3.2.1 is used *a lot* it would make sense
>>>> to provide a hardened version of collections to give people a chance to
>>>> easily avoid this line of attack in their application.
>>>>
>>>> Instead of removing the class we could prevent de-serialization of it in
>>>> the hardened jar. This would not break b/c and it is very unlikely that
>>>> the InvokerTransformer is serialized in legit ways.
>>>
>>> Rather than having hardened vs unhardened JARs, it would probably be
>>> better to use a system property to enable/disable the behaviour. I don't
>>> know the code or the vulnerability well enough to know exactly where to
>>> put this switch so it prevents the attack but has minimal impact on
>>> other uses.
>>
>> my idea was to have a binary compatible drop-in replacement that does
>> not require any configuration, so that people that happen to have
>> commons-collections 3.2.1 in their classpath can replace it with a
>> hardened version.
>>
>> But I am open to other suggestions, in the end it is important to do
>> what affected users would like to have to mitigate the problem.
> 
> My main concern with a hardened JAR is that, while with just this
> vulnerability, we end up with two JARs but how many JARs will we end up
> with 3 or 4 vulnerabilities down the line. Particularly when fixing a
> vulnerability means breaking functionality. I think system properties
> scale better.

with the default being: do not de-serialize InvokerTransformer?
Then I would be ok going that route.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/07/2015 11:19 AM, Mark Thomas wrote:
> On 07/11/2015 10:13, Thomas Neidhart wrote:
>> On 11/07/2015 04:25 AM, Bernd Eckenfels wrote:
>>> Hello,
>>>
>>> I tried to raise that concern in the message already, but it is probably
>>> worth repeating it explicitly: this is not a real bug
>>> in the Commons-Collection class, and it might not be worse fixing it, as
>>> there are possibly tons of other vectors. This was also addressed by the
>>> original authors in the talk and even here on Twitter:
>>>
>>> https://twitter.com/gebl/status/662754611304996866
>>>
>>> however, as the "foxglove" article shows, people still point at the
>>> apache project, and after all it is good pratice to reduce footprints
>>> and attack surfaces.
>>
>> it is clear that the InvokerTransformer by itself does not have a bug,
>> but due to the way how java serialization is applied and considering the
>> fact that at least collections-3.2.1 is used *a lot* it would make sense
>> to provide a hardened version of collections to give people a chance to
>> easily avoid this line of attack in their application.
>>
>> Instead of removing the class we could prevent de-serialization of it in
>> the hardened jar. This would not break b/c and it is very unlikely that
>> the InvokerTransformer is serialized in legit ways.
> 
> Rather than having hardened vs unhardened JARs, it would probably be
> better to use a system property to enable/disable the behaviour. I don't
> know the code or the vulnerability well enough to know exactly where to
> put this switch so it prevents the attack but has minimal impact on
> other uses.

my idea was to have a binary compatible drop-in replacement that does
not require any configuration, so that people that happen to have
commons-collections 3.2.1 in their classpath can replace it with a
hardened version.

But I am open to other suggestions, in the end it is important to do
what affected users would like to have to mitigate the problem.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/07/2015 04:25 AM, Bernd Eckenfels wrote:
> Hello,
> 
> I tried to raise that concern in the message already, but it is probably
> worth repeating it explicitly: this is not a real bug
> in the Commons-Collection class, and it might not be worse fixing it, as
> there are possibly tons of other vectors. This was also addressed by the
> original authors in the talk and even here on Twitter:
> 
> https://twitter.com/gebl/status/662754611304996866
> 
> however, as the "foxglove" article shows, people still point at the
> apache project, and after all it is good pratice to reduce footprints
> and attack surfaces.

it is clear that the InvokerTransformer by itself does not have a bug,
but due to the way how java serialization is applied and considering the
fact that at least collections-3.2.1 is used *a lot* it would make sense
to provide a hardened version of collections to give people a chance to
easily avoid this line of attack in their application.

Instead of removing the class we could prevent de-serialization of it in
the hardened jar. This would not break b/c and it is very unlikely that
the InvokerTransformer is serialized in legit ways.

Other ways to fix this issue seem to be quite complex, i.e. change the
way serialization is performed (use whitelisting, ...), and will most
likely not be executed in a timely manner. I wonder if some people
already ship a collections version with this class being
removed/manipulated, or use a runtime weaving technique to remove the
serialization support from it?

Setting up a reasonable security policy for this line of attack also
seems to be futile as it would have to be so restrictive that most
applications probably wont work anymore.

> So it seems to be a good idea to discuss some hardening. Unfortunatelly
> having a hardened distribution with only this one class removed might
> be a bit overkill. Are there other candidates to be left out in such a
> more specific distribution? Maybe everything proxy/reflection related?

I checked the codebase, and there are other Invoker* type classes but
they all use internally the InvokerTransformer.

> Both sample payloads have "gadget chains" which do start (readObject())
> in JCL classes and then use pretty generic interfaces like Annotations
> or Comparators, so there is really no link between the types and the
> specific weakness.

I did not see JCL (commons logging?) used in the gadget chains.

btw. the technique using the TransformerImpl from xalan (which is part
of the oracle/openjdk jre) is really scary as it allows an attacker to
inject arbitrary bytecode.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/07/2015 12:56 AM, Thomas Neidhart wrote:
> On 11/06/2015 10:25 PM, Bernd Eckenfels wrote:
>> ello,
>>
>> I came across this article:
>>
>> http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
>>
>> It describes attacks against common Java applications with
>> pre-authentication requests using malicious Java Object serialisation.
>> It builds upon the work of Gabriel Lawrence (@gebl) and Chris Frohoff
>> (@frohoff) (presented on January 28th, 2015, “Marshalling Pickles”
>> given at AppSecCali)
>>
>> http://www.slideshare.net/frohoff1/appseccali-2015-marshalling-pickles
>>
>> The ysoserial tool has some sample payloads, two use
>> commons-collection oac.collections.functors.InvokerTransformer. * 
>>
>> https://github.com/frohoff/ysoserial/tree/master/src/main/java/ysoserial/payloads
>>
>> The class itself is rather handy to break out of the readObject()
>> chains to execute arbitrary methods.
>>
>> I do'nt recall any discussion here about this
>> class. Is this currently handled/reported? Of course the more general
>> problem is using serialisation with untusted peers, and if
>> commons-collection fixes this, there might still be other vectors, but
>> still I think it would be good to do something against that "bad press"?
> 
> I was not aware of this yet, thanks for the pointers.
> 
> If we would remove the problematic classes and release a new collections
> version (for the 3.x or 4.x branch) we would break source and binary
> compatibility.
> 
> It might be acceptable/doable to release a collections version with an
> additional maven classifier (e.g. -hardened) that removes the relevant
> classes and explain the compatibility issues in detail in the release
> notes. What do others think about something like that?
> 
> btw. with Java 8 you can do similar things by using serialized method
> references. I did create a very simple example to illustrate the issue:
> 
> public class MyTest {
> 
>   public static void main(String[] args) throws Exception {
>   final SAM1 m1 = (SAM1 & Serializable) Runtime::getRuntime;
>   final SAM2 m2 = (SAM2 & Serializable) Runtime::exec;
>   
>   Transformer t1 = (Transformer & Serializable) (input) -> m1.action();
>   Transformer t2 = (Transformer & Serializable) (input) -> {
>   try {
>   Process p = m2.action((Runtime) input, "ls");
>   
>   String line;
>   StringBuilder output = new StringBuilder();
>   BufferedReader reader = new BufferedReader(new
> InputStreamReader(p.getInputStream()));
>   while ((line = reader.readLine()) != null) {
>   output.append(line);
>   output.append("\n");
>   }
>   reader.close();
>   return output.toString();
>   } catch (IOException e) {
>   return null;
>   }
>   };
> 
>   ChainedTransformer chain = new ChainedTransformer(t1, t2);
>   TransformedList list = TransformedList.transformedList(new
> ArrayList(), chain);
> 
> FileOutputStream fos = new FileOutputStream("test.ser");
> ObjectOutputStream os = new ObjectOutputStream(fos);
> os.writeObject(list);
> os.close();
> 
> FileInputStream fis = new FileInputStream("test.ser");
> ObjectInputStream ois = new ObjectInputStream(fis);
> 
> List l2 = (List)ois.readObject();
> 
> l2.add(1);
> 
> //Print the result
> System.out.println(l2);
> ois.close();
> }
> 
> interface SAM1 {
>   Runtime action();
> }
> 
> interface SAM2 {
> Process action(Runtime r, String s) throws IOException;
> }
> 

I was a bit too fast with this. This example will only work if the
remote site has the same code otherwise the lambda functions can not be
de-serialized.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collection][security] InvokerTransformer missused in java object serialisation exploits

[Thomas Neidhart]

On 11/06/2015 10:25 PM, Bernd Eckenfels wrote:
> ello,
> 
> I came across this article:
> 
> http://foxglovesecurity.com/2015/11/06/what-do-weblogic-websphere-jboss-jenkins-opennms-and-your-application-have-in-common-this-vulnerability/
> 
> It describes attacks against common Java applications with
> pre-authentication requests using malicious Java Object serialisation.
> It builds upon the work of Gabriel Lawrence (@gebl) and Chris Frohoff
> (@frohoff) (presented on January 28th, 2015, “Marshalling Pickles”
> given at AppSecCali)
> 
> http://www.slideshare.net/frohoff1/appseccali-2015-marshalling-pickles
> 
> The ysoserial tool has some sample payloads, two use
> commons-collection oac.collections.functors.InvokerTransformer. * 
> 
> https://github.com/frohoff/ysoserial/tree/master/src/main/java/ysoserial/payloads
> 
> The class itself is rather handy to break out of the readObject()
> chains to execute arbitrary methods.
> 
> I do'nt recall any discussion here about this
> class. Is this currently handled/reported? Of course the more general
> problem is using serialisation with untusted peers, and if
> commons-collection fixes this, there might still be other vectors, but
> still I think it would be good to do something against that "bad press"?

I was not aware of this yet, thanks for the pointers.

If we would remove the problematic classes and release a new collections
version (for the 3.x or 4.x branch) we would break source and binary
compatibility.

It might be acceptable/doable to release a collections version with an
additional maven classifier (e.g. -hardened) that removes the relevant
classes and explain the compatibility issues in detail in the release
notes. What do others think about something like that?

btw. with Java 8 you can do similar things by using serialized method
references. I did create a very simple example to illustrate the issue:

public class MyTest {

public static void main(String[] args) throws Exception {
final SAM1 m1 = (SAM1 & Serializable) Runtime::getRuntime;
final SAM2 m2 = (SAM2 & Serializable) Runtime::exec;

Transformer t1 = (Transformer & Serializable) (input) -> m1.action();
Transformer t2 = (Transformer & Serializable) (input) -> {
try {
Process p = m2.action((Runtime) input, "ls");

String line;
StringBuilder output = new StringBuilder();
BufferedReader reader = new BufferedReader(new
InputStreamReader(p.getInputStream()));
while ((line = reader.readLine()) != null) {
output.append(line);
output.append("\n");
}
reader.close();
return output.toString();
} catch (IOException e) {
return null;
}
};

ChainedTransformer chain = new ChainedTransformer(t1, t2);
TransformedList list = TransformedList.transformedList(new
ArrayList(), chain);

FileOutputStream fos = new FileOutputStream("test.ser");
ObjectOutputStream os = new ObjectOutputStream(fos);
os.writeObject(list);
os.close();

FileInputStream fis = new FileInputStream("test.ser");
ObjectInputStream ois = new ObjectInputStream(fis);

List l2 = (List)ois.readObject();

l2.add(1);

//Print the result
System.out.println(l2);
ois.close();
}

interface SAM1 {
Runtime action();
}

interface SAM2 {
Process action(Runtime r, String s) throws IOException;
}

}


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] releasing 3.6 ?

[Thomas Neidhart]

On 11/04/2015 10:13 AM, luc wrote:
> Hi all,
> 
> I would like to release 3.6 in the upcoming weeks.
> There have been a bunch of bug fixes and a few evolutions that are
> important to me.
> 
> I am still working on two things both related to ode: first trying
> to stabilize the Adams-BAshforth and Adams-Moulton integrators which
> sometime get stuck in an infinite loop reducing step size, and second
> adding a Field integrator, in exactly the same spirit we have added
> a field version of other algorithms (geometry and linear algebra come
> to my mind, as well as a specialized version of solver for Dfp).
> 
> Of course, once 3.6 is out the MATH_3_X branch will remain alive and
> we could also release other versions later on in the 3.x series.
> 
> What do other developers want to have in the 3.6 release?
> 
> Of course, I volunteer to be the release manager.

No objections, please let me know if you need some help.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [RESULT] [VOTE] Accept Naomi

[Thomas Neidhart]

On 11/04/2015 03:37 AM, Phil Steitz wrote:
> Here is a tally of the VOTE
> 
> Commons PMC:
> +1 from Dave Brosius, Luc Maisonobe, Phil Steitz, Joerg Schaibl,
> Oliver Heger, Gary Gregory, Niall Pemberton
> +0 from BenediKt Ritter
> -0 from
> -1 from Emmanuel Bourg, Mark Thomas, James Carman
> 
> Others:
> -1 from Uwe Barthel
> 
> This VOTE, therefore, passes.
> 
> I am not happy about proceeding, though, in the presence of the -1
> votes.  We like to make decisions by consensus and while this is a
> procedural decision and therefore subject to majority approval, I
> would like to ask those who case negative votes to reconsider.  I
> would also like to allow any others who have not registered an
> opinion to chime in so we make the right decision. 

I would not have a problem committing the code as is to the sandbox and
let people work on it.

In the end, I do not see much difference between any committer starting
to work on a new component in the sandbox and this case where already 2
committers expressed their interest in cleaning up / maintaining the
codebase and Dave already doing some work in this direction.

I am more concerned about the *existing* proper components that are
basically dead (and have been for a long time), while people are
regularly asking for progress or new releases.

Thomas

> Thanks,
> 
> Phil
> 
> On 10/29/15 5:42 PM, Phil Steitz wrote:
>> This is a VOTE to accept the code discussed in [1] and available for
>> review using the git commands below.  All are welcome to vote, votes
>> from PMC members are binding.  Assuming a positive vote, we will
>> execute a software grant with the authors and use the code as the
>> basis for a new Commons Sandbox component. 
>>
>> This VOTE will close in 72 hours.  More discussion on the code and
>> its fit in Commons is always welcome, but please do not reply to
>> this thread with discussion, other than embedded justification for
>> negative VOTES.  Use the thread from [1] instead.
>>
>> Git commands to grab the code:
>>
>> git clone g...@github.com:NormanShapiro/Naomi.git
>> git checkout gh-pages
>>
>> Thanks!
>>
>> Phil
>> [1] http://markmail.org/message/imoi5aipf63f7rsa
>>
>> [ ] +1 Yes!
>> [ ] +0 OK...
>> [ ] -0 OK, but...
>> [ ] -1 We should not do this, because...
>>
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [Math] Utilization of Lombok

[Thomas Neidhart]

On 09/28/2015 06:14 AM, venkatesha murthy wrote:
> Do we know if lombok is supported on all flavours of java for instance IBM
> JDK, Open JDK , java 8 etc...
> 
> Was just thinking of the future proof readiness.
> 
> Iam absolutely interested in lombok and even today use it for most demo
> purposes and have been stressing for its use where Oracle JDK is the
> primary use.
> 
> however i have always been cautioned/warned in the past of using it in
> production systems due to the lombok non-support on different java
> platforms.

The resulting byte-code should work with every jvm, but lombok uses
internal compiler APIs to perform the code-injection so it will
certainly not work for all jdks.

Thomas


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [Math] Utilitzation of SLF4J?

[Thomas Neidhart]

On 09/26/2015 01:11 PM, Gilles wrote:
> On Sat, 26 Sep 2015 09:53:30 +0200, Thomas Neidhart wrote:
>> On 09/26/2015 02:33 AM, Gilles wrote:
>>> On Fri, 25 Sep 2015 16:52:26 -0700, Hasan Diwan wrote:
>>>> On 25 September 2015 at 16:47, Gilles 
>>>> wrote:
>>>>
>>>>> On Fri, 25 Sep 2015 17:30:33 +0200, Thomas Neidhart wrote:
>>>>>
>>>>>> On Fri, Sep 25, 2015 at 5:09 PM, Gilles
>>>>>> 
>>>>>> wrote:
>>>>>>
>>>>>> On Fri, 25 Sep 2015 07:28:48 -0700, Phil Steitz wrote:
>>>>>>>
>>>>>>> On 9/25/15 7:03 AM, Gilles wrote:
>>>>>>>>
>>>>>>>> On Fri, 25 Sep 2015 15:54:14 +0200, Thomas Neidhart wrote:
>>>>>>>>>
>>>>>>>>> Hi Ole,
>>>>>>>>>>
>>>>>>>>>> for a start, I think you are asking the wrong question.
>>>>>>>>>> First of all we need to agree that we want to add some kind of
>>>>>>>>>> logging
>>>>>>>>>> facility to CM.
>>>>>>>>>> If the outcome is positive, there are a handful of alternatives,
>>>>>>>>>> some of
>>>>>>>>>> them more viable than slf4j in the context of CM (e.g. JUL or
>>>>>>>>>> commons-logging).
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Could someone summarize why those alternatives were deemed "more
>>>>>>>>> viable"?
>>>>>>>>>
>>>>>>>>> btw. the same discussion has been done for other commons
>>>>>>>>>
>>>>>>>>>> components as
>>>>>>>>>> well, and the result usually was: do not add logging
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> What was the rationale?
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Look at the archives.  We have discussed this multiple times in the
>>>>>>>> past in [math] and each time came to the conclusion that Thomas
>>>>>>>> succinctly states above.  What has changed now?
>>>>>>>>
>>>>>>>>
>>>>>>> We also discussed several times to stick with Java 5.
>>>>>>> Fortunately, that has changed. [Although sticking with Java 7 is
>>>>>>> still
>>>>>>> a bad decision IMHO.]
>>>>>>>
>>>>>>> As for logging, IIRC, the sole argument was "no dependency" because
>>>>>>> (IIRC) of the potential "JAR hell".
>>>>>>>
>>>>>>>
>>>>>> that's not correct. The decision to not include any dependencies has
>>>>>> nothing to do with "JAR hell".
>>>>>>
>>>>>
>>>>> Although I can't find it now, I'm pretty sure that I more than once
>>>>> got such an answer.
>>>>>
>>>>> In order to prevent JAR hell, commons components strictly stick to the
>>>>>> "Versioning guidelines" [1]
>>>>>>
>>>>>
>>>>> I can't see how it relates.
>>>>> But if you mean that no JAR hell can emerge from using a logging
>>>>> framework,
>>>>> then that's good news.
>>>>>
>>>>> The no-dependency rule is more related to the proposal of the
>>>>> component,
>>>>>> see [2]
>>>>>>
>>>>>
>>>>> Thanks for the reminder; in that document, we read:
>>>>>
>>>>>   (1) Scope of the Package
>>>>>[...]
>>>>>5. Limited dependencies. No external dependencies beyond Commons
>>>>> components and the JDK
>>>>>
>>>>> So we are fine if use "Log4j 2" as kindly offered by Gary.
>>
>> log4j is not a commons component btw.
> 
> Too bad for me. :-/
> Case resolved, then, by the argument of authority?

I just pointed out that log4j is not a commons component and did not
imply anything else.

> "Commons" is OK but not another Apache project, by virt

Re: [Math] Utilitzation of SLF4J?

[Thomas Neidhart]

On 09/26/2015 02:33 AM, Gilles wrote:
> On Fri, 25 Sep 2015 16:52:26 -0700, Hasan Diwan wrote:
>> On 25 September 2015 at 16:47, Gilles 
>> wrote:
>>
>>> On Fri, 25 Sep 2015 17:30:33 +0200, Thomas Neidhart wrote:
>>>
>>>> On Fri, Sep 25, 2015 at 5:09 PM, Gilles 
>>>> wrote:
>>>>
>>>> On Fri, 25 Sep 2015 07:28:48 -0700, Phil Steitz wrote:
>>>>>
>>>>> On 9/25/15 7:03 AM, Gilles wrote:
>>>>>>
>>>>>> On Fri, 25 Sep 2015 15:54:14 +0200, Thomas Neidhart wrote:
>>>>>>>
>>>>>>> Hi Ole,
>>>>>>>>
>>>>>>>> for a start, I think you are asking the wrong question.
>>>>>>>> First of all we need to agree that we want to add some kind of
>>>>>>>> logging
>>>>>>>> facility to CM.
>>>>>>>> If the outcome is positive, there are a handful of alternatives,
>>>>>>>> some of
>>>>>>>> them more viable than slf4j in the context of CM (e.g. JUL or
>>>>>>>> commons-logging).
>>>>>>>>
>>>>>>>>
>>>>>>> Could someone summarize why those alternatives were deemed "more
>>>>>>> viable"?
>>>>>>>
>>>>>>> btw. the same discussion has been done for other commons
>>>>>>>
>>>>>>>> components as
>>>>>>>> well, and the result usually was: do not add logging
>>>>>>>>
>>>>>>>>
>>>>>>> What was the rationale?
>>>>>>>
>>>>>>>
>>>>>> Look at the archives.  We have discussed this multiple times in the
>>>>>> past in [math] and each time came to the conclusion that Thomas
>>>>>> succinctly states above.  What has changed now?
>>>>>>
>>>>>>
>>>>> We also discussed several times to stick with Java 5.
>>>>> Fortunately, that has changed. [Although sticking with Java 7 is still
>>>>> a bad decision IMHO.]
>>>>>
>>>>> As for logging, IIRC, the sole argument was "no dependency" because
>>>>> (IIRC) of the potential "JAR hell".
>>>>>
>>>>>
>>>> that's not correct. The decision to not include any dependencies has
>>>> nothing to do with "JAR hell".
>>>>
>>>
>>> Although I can't find it now, I'm pretty sure that I more than once
>>> got such an answer.
>>>
>>> In order to prevent JAR hell, commons components strictly stick to the
>>>> "Versioning guidelines" [1]
>>>>
>>>
>>> I can't see how it relates.
>>> But if you mean that no JAR hell can emerge from using a logging
>>> framework,
>>> then that's good news.
>>>
>>> The no-dependency rule is more related to the proposal of the component,
>>>> see [2]
>>>>
>>>
>>> Thanks for the reminder; in that document, we read:
>>>
>>>   (1) Scope of the Package
>>>[...]
>>>5. Limited dependencies. No external dependencies beyond Commons
>>> components and the JDK
>>>
>>> So we are fine if use "Log4j 2" as kindly offered by Gary.

log4j is not a commons component btw.

I have not seen a single example of a useful logging message that could
be added to commons-math, but we are already discussing which framework
to use.

The examples with println debugging are not valid imho, because how do
you know in advance what you will need to log in order to successfully
debug some piece of code and such low-level information should not be
captured in logs anyway.

>>> My long-standing mentioning of slf4j was only because of its
>>> "weightlessness" (thanks to the no-op implementation of its API).
>>> If "Log4j 2" has followed this path, good for everyone.
>>>
>>> No objection, then?
>>
>>
>> I'm still not clear what log4j 2 adds -- most Apache java projects
>> seem to
>> use log4j 1.2, seems to work well. -- H
>>
> 
> I can only answer about "slf4j" where the "f" stands for facade: it's
> "only"
> an API, with bridges to several logging frameworks (log4j, logback, etc.).
> 
> The separation of concerns (API vs one of several implementations to
> choose from)
> allows the top-level application to uniformly configure logging or to
> disable it
> completely (if choosing the "no-op" implementation).

That is virtually true for all logging frameworks, including log4j,
slf4j, commons-logging.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [Math] Utilization of Lombok

[Thomas Neidhart]

On 09/25/2015 05:04 PM, Ole Ersoy wrote:
> Hi Thomas,
> 
> On 09/25/2015 08:45 AM, Thomas Neidhart wrote:
>> Hi Ole,
>>
>> can you explain why you think that the addition of lombok brings any
>> benefit to our users?
> Sure - I'm looking at the LevenbergMarquardtOptimizer ATM, and it has
> the following set of parameters:
> 
> /* configuration parameters */
> /** Positive input variable used in determining the initial step
> bound. */
> private final double initialStepBoundFactor;
> /** Desired relative error in the sum of squares. */
> private final double costRelativeTolerance;
> /**  Desired relative error in the approximate solution parameters. */
> private final double parRelativeTolerance;
> /** Desired max cosine on the orthogonality between the function vector
>  * and the columns of the jacobian. */
> private final double orthoTolerance;
> /** Threshold for QR ranking. */
> private final double qrRankingThreshold;
> 
> And corresponding getters:
> /**
>  * Gets the value of a tuning parameter.
>  * @see #withInitialStepBoundFactor(double)
>  *
>  * @return the parameter's value.
>  */
> public double getInitialStepBoundFactor() {
> return initialStepBoundFactor;
> }
> 
> /**
>  * Gets the value of a tuning parameter.
>  * @see #withCostRelativeTolerance(double)
>  *
>  * @return the parameter's value.
>  */
> public double getCostRelativeTolerance() {
> return costRelativeTolerance;
> }
> 
> /**
>  * Gets the value of a tuning parameter.
>  * @see #withParameterRelativeTolerance(double)
>  *
>  * @return the parameter's value.
>  */
> public double getParameterRelativeTolerance() {
> return parRelativeTolerance;
> }
> 
> /**
>  * Gets the value of a tuning parameter.
>  * @see #withOrthoTolerance(double)
>  *
>  * @return the parameter's value.
>  */
> public double getOrthoTolerance() {
> return orthoTolerance;
> }
> 
> /**
>  * Gets the value of a tuning parameter.
>  * @see #withRankingThreshold(double)
>  *
>  * @return the parameter's value.
>  */
> public double getRankingThreshold() {
> return qrRankingThreshold;
> }
> 
> Lombok will generate all of these.  Eclipse can do the same thing, but
> if we delete one of the parameters, then the corresponding getter also
> has to be deleted.  Also Lombok cuts down on the source code noise,
> since it is a byte code generator.  The generated code does not appear
> in the source.
> 
> Lombok also has a @Builder annotation that can be used to generate a
> inner static builder class that provides a fluent construction API. So
> if we break off the LevenbergMarquardtOptimizer configuration into its
> own class, and generate all the getters and the fluid API, there should
> be substantial code reduction.
> 
> Gilles is also working on a snapshot capability for neural nets, and the
> @Synchronized annotation could come in handy here.  These are the items
> I have looked at so far.
> 
>>
>> >From my point of view, lombok can help developers by taking over some
>> tedious tasks, but this is quite irrelevant in the case of CM as the
>> majority of work goes into algorithm design and verification rather
>> than in
>> writing getters/setters (which btw has pretty good IDE support).
> 
> I agree that the majority of time goes into the design of the
> algorithm.  For me personally, when I'm looking at code, and it has a
> ton of boilerplate, it does slow my productivity...just because of all
> the noise.  I'm happy once I've gotten it as DRY as possible.
> 
> The more boilerplate, the more reluctant we are going to be to undertake
> refactoring, and we will make more mistakes (At least I will :) ).
> 
>>   So this
>> would just add additional complexity and the gain is very unclear.
> I think you will find it refreshing once you try it.  At this point
> though I just wanted to float the idea.  I'll complete the experiment
> and publish the result.  At that point we will have a good baseline to
> gage whether adding it will add enough value to offset the cost of
> adding it.

Well I know lombok.

Keep in mind that it is a bit more difficult to integrate it into our
build-chain. As you probably know, in order to generate proper javadoc,
you need to use delombok first to create source files which can be used
for the javadoc process.

In general I do not think that lombok is the right tool for a library
like CM, and I do not know any other library that uses lombok. It
certainly has strengths and uses for application development though.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [Math] Utilitzation of SLF4J?

[Thomas Neidhart]

On Fri, Sep 25, 2015 at 5:09 PM, Gilles 
wrote:

> On Fri, 25 Sep 2015 07:28:48 -0700, Phil Steitz wrote:
>
>> On 9/25/15 7:03 AM, Gilles wrote:
>>
>>> On Fri, 25 Sep 2015 15:54:14 +0200, Thomas Neidhart wrote:
>>>
>>>> Hi Ole,
>>>>
>>>> for a start, I think you are asking the wrong question.
>>>> First of all we need to agree that we want to add some kind of
>>>> logging
>>>> facility to CM.
>>>> If the outcome is positive, there are a handful of alternatives,
>>>> some of
>>>> them more viable than slf4j in the context of CM (e.g. JUL or
>>>> commons-logging).
>>>>
>>>
>>> Could someone summarize why those alternatives were deemed "more
>>> viable"?
>>>
>>> btw. the same discussion has been done for other commons
>>>> components as
>>>> well, and the result usually was: do not add logging
>>>>
>>>
>>> What was the rationale?
>>>
>>
>> Look at the archives.  We have discussed this multiple times in the
>> past in [math] and each time came to the conclusion that Thomas
>> succinctly states above.  What has changed now?
>>
>
> We also discussed several times to stick with Java 5.
> Fortunately, that has changed. [Although sticking with Java 7 is still
> a bad decision IMHO.]
>
> As for logging, IIRC, the sole argument was "no dependency" because
> (IIRC) of the potential "JAR hell".
>

that's not correct. The decision to not include any dependencies has
nothing to do with "JAR hell".

In order to prevent JAR hell, commons components strictly stick to the
"Versioning guidelines" [1]

The no-dependency rule is more related to the proposal of the component,
see [2]

[1] http://commons.apache.org/releases/versioning.html
[2] http://commons.apache.org/proper/commons-math/proposal.html

Thomas


> If there are now well-formed answers proving that the fear was
> unfounded, that is also a change.
>
> IMO, logging is quite important for any "non-obvious" code.[1]
> [I'm again stating that, in that respect, CM is not like the other
> "Commmons" components.]
>
> Several times, I've been obliged to create a modified version of CM
> to introduce "print" statements (poor man's logging!) in order to
> figure out why my code did not do what it was supposed to.
> It also makes a code easier to debug while developing or modifying it
> (without resorting to poor man's logging, then deleting the "print",
> then reinstating them, then deleting them again, ad nauseam).
>
> Gilles
>
> [1] No quality or complexity judgment implied.
>
>
> Phil
>>
>>>
>>>
>>> Gilles
>>>
>>> Thomas
>>>>
>>>>
>>>> On Fri, Sep 25, 2015 at 3:17 PM, Ole Ersoy 
>>>> wrote:
>>>>
>>>> Hello,
>>>>>
>>>>> We have been discussing various ways to view what's happening
>>>>> internally
>>>>> with algorithms, and the topic of including SLF4J has come up.
>>>>> I know that
>>>>> this was discussed earlier and it was decided that CM is a low
>>>>> level
>>>>> dependency, therefore it should minimize the transitive
>>>>> dependencies that
>>>>> it introduces.  The Java community has adopted many means of
>>>>> dealing with
>>>>> potential logging conflicts, so I'm requesting that we use SLF4J
>>>>> for
>>>>> logging.
>>>>>
>>>>> I know that JBoss introduced its own logging system, and this
>>>>> made me a
>>>>> bit nervous about this suggestion, so I looked up strategies for
>>>>> switching
>>>>> their logger out with SLF4J:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> http://stackoverflow.com/questions/14733369/force-jboss-logging-to-use-of-slf4j
>>>>>
>>>>>
>>>>> The general process I go through when working with many
>>>>> dependencies that
>>>>> might use commons-logging instead of SLF4J looks something like
>>>>> this:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> http://stackoverflow.com/questions/8921382/maven-slf4j-version-conflict-when-using-two-different-dependencies-that-requi
>>>>>
>>>>>
>>>>> With JDK9 individual modules can define their own isolated set of
>>>>> dependencies.  At this point the fix should be a permanent.  If
>>>>> someone has
>>>>> has a very intricate scenario that we have not yet seen, they
>>>>> could use
>>>>> (And probably should use) OSGi to isolate dependencies.
>>>>>
>>>>> WDYT?
>>>>>
>>>>> Cheers,
>>>>> - Ole
>>>>>
>>>>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [Math] Utilitzation of SLF4J?

[Thomas Neidhart]

Hi Ole,

for a start, I think you are asking the wrong question.
First of all we need to agree that we want to add some kind of logging
facility to CM.
If the outcome is positive, there are a handful of alternatives, some of
them more viable than slf4j in the context of CM (e.g. JUL or
commons-logging).

btw. the same discussion has been done for other commons components as
well, and the result usually was: do not add logging

Thomas


On Fri, Sep 25, 2015 at 3:17 PM, Ole Ersoy  wrote:

> Hello,
>
> We have been discussing various ways to view what's happening internally
> with algorithms, and the topic of including SLF4J has come up.  I know that
> this was discussed earlier and it was decided that CM is a low level
> dependency, therefore it should minimize the transitive dependencies that
> it introduces.  The Java community has adopted many means of dealing with
> potential logging conflicts, so I'm requesting that we use SLF4J for
> logging.
>
> I know that JBoss introduced its own logging system, and this made me a
> bit nervous about this suggestion, so I looked up strategies for switching
> their logger out with SLF4J:
>
> http://stackoverflow.com/questions/14733369/force-jboss-logging-to-use-of-slf4j
>
> The general process I go through when working with many dependencies that
> might use commons-logging instead of SLF4J looks something like this:
>
> http://stackoverflow.com/questions/8921382/maven-slf4j-version-conflict-when-using-two-different-dependencies-that-requi
>
> With JDK9 individual modules can define their own isolated set of
> dependencies.  At this point the fix should be a permanent.  If someone has
> has a very intricate scenario that we have not yet seen, they could use
> (And probably should use) OSGi to isolate dependencies.
>
> WDYT?
>
> Cheers,
> - Ole
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [Math] Utilization of Lombok

[Thomas Neidhart]

Hi Ole,

can you explain why you think that the addition of lombok brings any
benefit to our users?

>From my point of view, lombok can help developers by taking over some
tedious tasks, but this is quite irrelevant in the case of CM as the
majority of work goes into algorithm design and verification rather than in
writing getters/setters (which btw has pretty good IDE support). So this
would just add additional complexity and the gain is very unclear.

Thomas


On Fri, Sep 25, 2015 at 3:31 PM, Ole Ersoy  wrote:

> Hello,
>
> I'm going to utilize Lombok in a CM design experiment.  Once the
> experiment is done CM can decide if it likes Lombok.  I know that CM tries
> to stay dependency free, so I just want to make clear that Lombok is
> compile time only:
> http://stackoverflow.com/questions/6107197/how-does-lombok-work
>
> Lombok eliminates the need to code boilerplate plate code, like getters,
> setters, toString().  It can also generate a fluid builder for
> configuration objects, check for null arguments, etc.  It also has an
> @Synchronized annotation that is an improvement on the synchronized keyword.
>
> Lombok alters the byte code, keeping the source code clean and minimal.
> The additional generated code and be seen using an Eclipse plugin.  So for
> example when looking at the outline view, you can see the generated
> getters, etc.
>
>
> https://standardofnorms.wordpress.com/2013/05/10/reducing-java-boilerplate-code-with-lombok-with-eclipse-installation/
>
> Cheers,
> - Ole
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [Math] Compile only top-level package and below

[Thomas Neidhart]

On 09/04/2015 03:08 PM, Gilles wrote:
> Hello.
> 
> There are two branches for Commons Math.
> 
> For one, the top-level Java package is
>   org.apache.commons.math4
> For the other, it is
>   org.apache.commons.math3
> 
> Unless I'm mistaken, this should imply that maven tries to compile
> only files under either
>   src/main/java/org/apache/commons/math4
>   src/test/java/org/apache/commons/math4
> or
>   src/main/java/org/apache/commons/math3
>   src/test/java/org/apache/commons/math3
> 
> But it happens that I have currently files in "math3" not currently
> checked in into git: those are new files which git does not remove
> when switching branches.
> Then when starting a compilation in "master" (where the top-level
> is "math4"), lots of compilation errors occur.
> 
> The "source" top-level directories do not seem to be specified
> in the project's POM.
> Can the parent be changed in order to produce the desired behaviour?
> 
> Or is there a workaround?
> Is there a better way to handle the situation (short of manually
> moving the source files back and forth)?

you probably want to take a loot at the stash command from git.

This is very helpful (and needed) when switching between branches.
The source files are required to be tracked by git though (i.e. need to
be added).

afaik, you can also exclude files from the src folder, so that they are
not compiled.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] random boolean arrays

[Thomas Neidhart]

On 07/12/2015 04:58 PM, Phil Steitz wrote:
> On 7/12/15 2:50 AM, Thomas Neidhart wrote:
>> On 07/11/2015 09:43 PM, Phil Steitz wrote:
>>> On 7/11/15 12:29 PM, Thomas Neidhart wrote:
>>>> On 07/11/2015 09:08 PM, Phil Steitz wrote:
>>>>> The code implemented in MATH-1242 to improve performance of KS
>>>>> monteCarloP in-lines efficient generation of random boolean arrays.
>>>>>   Unfortunately, I think the implementation is not quite random (see
>>>>> comments on the ticket).  To verify it, we need to be able to test
>>>>> the random boolean array generation directly.  To do that, we have
>>>>> to either expose the method (at least as protected) in the KS class
>>>>> or add it somewhere else.  I propose the latter but am not sure
>>>>> where to put it.  For speed, we need to avoid array copies, so the
>>>>> API will have to be something like randomize(boolean[], nTrue).  It
>>>>> could go in the swelling MathArrays class, or RandomDataGenerator. 
>>>>> The latter probably makes more sense, but the API does not fit too
>>>>> well.  Any ideas?
>>>> If it is just for testing purposes, you can also access the method in
>>>> question via reflection, see an example here:
>>>> http://stackoverflow.com/questions/34571/how-to-test-a-class-that-has-private-methods-fields-or-inner-classes
>>> Do you think it *should* be a private method of the K-S class?
>> Right now, I do not see much uses outside the class, but if we decide to
>> make it public then I would prefer a special util class in the random
>> package to avoid cluttering the MathArrays class.
> 
> OK, for now we can make it private and use the reflection method
> above to test it.

ok, but I guess it is also fine to make it package private as sebb
suggested. We did something similar recently for some of the improved
sampling methods provided by Otmar.

>> Regarding the RandomDataGenerator: I think this class should be
>> deprecated and replaced by a Sampler interface as proposed by Gilles.
> 
> Please consider keeping this class.  Consider this a user request. 
> I have quite a few applications that use this class for two reasons:

ok, the reason why I thought the class should be deprecated is because
it was not kept up-to-date with all the new discrete and continuous
distributions that we added in the last 2-3 years. If you think it is
useful, then we can keep it of course.

> 1.  One object instance tied to one PRNG that generates data from
> multiple different distributions.   This is convenient.   Sure, I
> could refactor all of these apps to instantiate new objects for each
> type of generated data and hopefully still be able to peg them to
> one PRNG; but that is needless work that also complicates the code.
> 
> 2.  There are quite a few methods in this class that have nothing to
> do with sampling (nextPermutation, nextHexString, nextSecureXxx,
> etc) but which conveniently share the RandomGenerator.  I guess the
> utility methods get moved out somewhere else.  Again, I end up
> having to refactor all of my code that uses them and when I want
> simulations to be based on a single PRNG, I have to find a way to
> pass the RandomGenerator around to them.
> 
> I don't yet see the need to refactor the sampling support in the
> distributions package; but as my own apps are not impacted by this,
> if everyone else sees the user impact of the refactoring as
> outweighed by the benefit, I won't stand in the way.   Please lets
> just keep the RandomDataGenerator convenience class in the random
> package in any case.  I will take care of whatever adjustments are
> needed to adapt to whatever we settle on for sampling in the
> distributions package.

Well, it is not really necessary to do everything together and refactor
the distributions.

Probably it is better to start the other way round, and describe what I
want to add, and see how other things fit in:

 * I want a generic Sampler interface, i.e. something like this:
 ** nextSample()
 ** nextSamples(int size)
 ** nextSamples(double[] samples)

there could be a DiscreteSampler and ContinuousSampler interface to
handle the cases for int / double values.

The distributions could either be changed to return such a sampler as
Gilles proposed (with the advantage that no random instance is tied to
the distribution itself), or implement the interface directly (with the
advantage that we would not need to refactor too much).

>> One can then create a sampler for any distribution or from other
>> sources, e.g. when needing a fast and efficient sampler without
>> replaceme

Re: [math] random boolean arrays

[Thomas Neidhart]

On 07/11/2015 09:43 PM, Phil Steitz wrote:
> On 7/11/15 12:29 PM, Thomas Neidhart wrote:
>> On 07/11/2015 09:08 PM, Phil Steitz wrote:
>>> The code implemented in MATH-1242 to improve performance of KS
>>> monteCarloP in-lines efficient generation of random boolean arrays.
>>>   Unfortunately, I think the implementation is not quite random (see
>>> comments on the ticket).  To verify it, we need to be able to test
>>> the random boolean array generation directly.  To do that, we have
>>> to either expose the method (at least as protected) in the KS class
>>> or add it somewhere else.  I propose the latter but am not sure
>>> where to put it.  For speed, we need to avoid array copies, so the
>>> API will have to be something like randomize(boolean[], nTrue).  It
>>> could go in the swelling MathArrays class, or RandomDataGenerator. 
>>> The latter probably makes more sense, but the API does not fit too
>>> well.  Any ideas?
>> If it is just for testing purposes, you can also access the method in
>> question via reflection, see an example here:
>> http://stackoverflow.com/questions/34571/how-to-test-a-class-that-has-private-methods-fields-or-inner-classes
> 
> Do you think it *should* be a private method of the K-S class?

Right now, I do not see much uses outside the class, but if we decide to
make it public then I would prefer a special util class in the random
package to avoid cluttering the MathArrays class.

Regarding the RandomDataGenerator: I think this class should be
deprecated and replaced by a Sampler interface as proposed by Gilles.
One can then create a sampler for any distribution or from other
sources, e.g. when needing a fast and efficient sampler without
replacement (see MATH-1239).

I did not have time to complete a patch, but am working on it.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] random boolean arrays

[Thomas Neidhart]

On 07/11/2015 09:08 PM, Phil Steitz wrote:
> The code implemented in MATH-1242 to improve performance of KS
> monteCarloP in-lines efficient generation of random boolean arrays.
>   Unfortunately, I think the implementation is not quite random (see
> comments on the ticket).  To verify it, we need to be able to test
> the random boolean array generation directly.  To do that, we have
> to either expose the method (at least as protected) in the KS class
> or add it somewhere else.  I propose the latter but am not sure
> where to put it.  For speed, we need to avoid array copies, so the
> API will have to be something like randomize(boolean[], nTrue).  It
> could go in the swelling MathArrays class, or RandomDataGenerator. 
> The latter probably makes more sense, but the API does not fit too
> well.  Any ideas?

If it is just for testing purposes, you can also access the method in
question via reflection, see an example here:
http://stackoverflow.com/questions/34571/how-to-test-a-class-that-has-private-methods-fields-or-inner-classes

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: svn commit: r1688303 - /commons/proper/collections/trunk/src/changes/changes.xml

[Thomas Neidhart]

On 07/01/2015 01:08 AM, sebb wrote:
> On 29 June 2015 at 21:41,   wrote:
>> Author: tn
>> Date: Mon Jun 29 20:41:17 2015
>> New Revision: 1688303
>>
>> URL: http://svn.apache.org/r1688303
>> Log:
>> Add reference to parent ticket.
>>
>> Modified:
>> commons/proper/collections/trunk/src/changes/changes.xml
>>
>> Modified: commons/proper/collections/trunk/src/changes/changes.xml
>> URL: 
>> http://svn.apache.org/viewvc/commons/proper/collections/trunk/src/changes/changes.xml?rev=1688303&r1=1688302&r2=1688303&view=diff
>> ==
>> --- commons/proper/collections/trunk/src/changes/changes.xml (original)
>> +++ commons/proper/collections/trunk/src/changes/changes.xml Mon Jun 29 
>> 20:41:17 2015
>> @@ -44,7 +44,7 @@
>>  
>>Added decorators for "NavigableSet" interface.
>>  
>> -
>> +
> 
> Not sure that will work with the site report/release notes.
> 
> Please check.

I did the same for some changes in the 4.0 release, and the release
notes are manually post-processed anyway.

Thomas

> 
>>Added new class "FluentIterable" to support a fluent API for 
>> manipulating
>>Iterable instances. Additionally various supporting methods have been
>>added to "IterableUtils" and "IteratorUtils".
>>
>>
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [DISCUSS] Karma necessary for Release Manager roles

[Thomas Neidhart]

On 06/23/2015 05:59 PM, sebb wrote:
> The default karma needed for committing to the SVN dist/release/TLP
> areas is membership of the LDAP PMC group. This can be changed to be
> membership of the LDAP committer group if required.
> 
> I'm not sure about the current karma needed for Nexus upload, it may
> be either PMC or committer  membership. AFAIK this can be changed if
> necessary.
> 
> I think it would be useful to allow non-PMC members to act as release manager.
> This means relaxing some of the existing technical restrictions.
> 
> However I think it would too permissive to allow any ASF committer to act as 
> RM.
> Releases cannot be reverted in the same way as commits, and need a
> greater degree of experience than code/doc fixes.
> 
> So I suggest that Commons agrees that membership of the LDAP Commons
> committer group be regarded as sufficient karma for performing RM
> tasks.
> 
> [We no longer use the LDAP Commons committer group for controlling
> access to the code itself.
> I think the LDAP committer group is largely redundant. This would give
> it a new role.]
> 
> The PMC does not need to involve the board in changes to the committer group.
> Updates to the LDAP group can be made by a PMC chair.
> Apart from the actual Commons chair there are other chairs on the PMC
> as well as Infra folks, so that should not be an issue once it has
> been decided on the RM.
> 
> So it would be a relatively simple process to grant karma to a new RM
> once any necessary technical changes have been made.
> 
> Thoughts?

I think that every committer shall be able to do a release for a commons
component after it has been discussed on the mailinglist.

Otoh, I think it would be very beneficial if at least one PMC member
provides mentoring for somebody doing his / her first release here, as
the process can be quite complex and frustrating.

I did survive my first release only because I followed the wonderful
release guide for math, but only few people know about it.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [Math] Nearest neighbors search

[Thomas Neidhart]

On Fri, Jun 19, 2015 at 3:56 PM, Gilles 
wrote:

> Hi.
>
> Referring to the newly created MATH-1235.
>
> I was looking for a way to improve the performance of a code that also
> needs, for some point "p", to select the "n" closest neighbours (from
> a list of sample data points).
>
> Browsing through web pages, some refer to BSP as a generalization of
> kd-tree; hence the question whether the implementation in CM can be
> used to provide the functionality in a way that is independent of its
> projected use to resolve MATH-1235.
>
> What do you think?
>

I do not know if the BSPTree can be adapter for this purpose.

In the case of DBSCAN I need an n-dimensional KD tree, so the BSPTree
solution would not work for me at least.

btw. scipy also offers a Kdtree as part of its spatial package.

Thomas

Re: porting fixes to v 3.2.X

[Thomas Neidhart]

On 06/19/2015 05:41 PM, Marc-André Chartrand wrote:
> Can bug fixes from version 4.0 be ported to v 3.2.X ?   and also, when is
> the next 3.2.X version coming out ?
> 
> The fix I'm looking at in particular is :
> 
> https://issues.apache.org/jira/browse/COLLECTIONS-294
> 
> Which is only fixed in 4.0 apparently.
> 

Currently, there are no plans to release another 3.x version.

Imho, a 3.2.2 release would only make sense if we really fix a
considerable amount of known bugs present in the 3.2.1 release, but this
would mean a lot of work.

If a *big corp* would sponsor a few working days, I could imagine that a
new release is feasible, otherwise try switching to collections 4 which
has fixed all known bugs in the 3.x branch.

btw. collections 3 and 4 can be used at the same time as they use
different package names.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [LANG][COLLECTIONS] Require a static function to swap 2 elements in a array (primitive as well non-primitive)

[Thomas Neidhart]

Hi,

I think such a feature belongs to the ArrayUtils class in lang.

Thomas

On Thu, Jun 18, 2015 at 5:08 AM, venkatesha murthy <
venkateshamurth...@gmail.com> wrote:

> Hi All
>
> I was looking to ArrayUtils(version 3.4) to hopefully find a swap method .
> Is this some thing available else where or can i implement one. please
> suggest.
>
> Also i am not sure which commons lib this can be placed with. please let
> know about that as well.
>
> thanks
> venkat.
>

Re: [VOTE] Release Configuration 2.0-beta1 based on RC1

[Thomas Neidhart]

On 06/14/2015 10:10 PM, Oliver Heger wrote:
> Hi all,
> 
> this is a vote for the first beta version of [configuration] 2.0 based
> on the first release candidate. After two alpha versions, it is now time
> to move to a beta version. Since the last alpha version, there has only
> be a small change: an obsolete interface has been removed. Details are
> available in the release notes.
> 
> Switching to beta indicates that we expect the interfaces to become more
> stable. However, there is still not yet a guarantee that there will be
> no more changes on the API.
> 
> Because this is a beta and not yet an official release I did not deploy
> the artifacts to Nexus; only the distributions were created.
> 
> Configuration 2.0-beta1 RC1 is available for review here:
> https://dist.apache.org/repos/dist/dev/commons/configuration
> (revision 9374)
> 
> Details of changes since 1.10 and the alpha versions are in the
> release notes:
> 
> https://dist.apache.org/repos/dist/dev/commons/configuration/RELEASE-NOTES.txt
> 
> http://people.apache.org/~oheger/configuration-2.0-beta1-rc1/changes-report.html
> 
> Here is the tag:
> 
> http://svn.apache.org/repos/asf/commons/proper/configuration/tags/CONFIGURATION_2_0_beta1_RC1
> (revision 1685450)
> 
> Site:
>http://people.apache.org/~oheger/configuration-2.0-beta1-rc1/
> (note some links in the menu for version 1.10 are not yet working)
> 
> KEYS:
>   http://www.apache.org/dist/commons/KEYS
> 
> Please review the release candidate and vote.
> This vote will close no sooner than 72 hours from now, i.e. after 2000
> GMT 17-June 2015
> 
>   [x] +1 Release these artifacts

tested with

Apache Maven 3.1.1 (0728685237757ffbf44136acec0402957f723d9a; 2013-09-17
17:22:22+0200)
Maven home: /home/tn/bin/apache-maven-3.1.1
Java version: 1.7.0_79, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-openjdk-i386/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.13.0-49-generic", arch: "i386", family: "unix"

I got the same test error as Bruno, but not a blocker imho.

Thomas

>   [ ] +0 OK, but...
>   [ ] -0 OK, but really should fix...
>   [ ] -1 I oppose this release because...
> 
> Thanks!
> Oliver
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Finding a Place to Help

[Thomas Neidhart]

On 06/16/2015 07:21 PM, Landon Blake wrote:
> I'm interested in helping a bit with Jakarta Commons. Is there anyone
> currently working on the BeanUtils or Collections modules? What about the
> image IO modules?

We are currently in the progress of making a new release for collections
and could need help, especially wrt writing tests.

There are several new interfaces / implementations  (MultiSet,
MultiValuedMap, FluentIterable, IterableUtils) that have been added
since the last release, would be great if someone could further increase
our test coverage in this area.

Thomas

> I thought I might start by dressing up the documentation or writing some
> unit tests for one of those modules if they are currently without a
> maintainer. I hope I can help.
> 
> Landon
> 


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collections] New interface: MultiSet

[Thomas Neidhart]

On 06/11/2015 11:18 PM, Matt Benson wrote:
> On Thu, Jun 11, 2015 at 3:37 PM, Thomas Neidhart
>  wrote:
>> Hi,
>>
>> I have just committed the first draft of the new interface MultiSet.
>>
>> What is the idea behind this?
>>
>> The rationale of this interface (and corresponding implementations) is
>> to replace the existing Bag interface, which does violate the Collection
>> contract. Actually, I wanted to make the Bag interface compliant to the
>> general Collection contract prior to the 4.0 release, but was convinced
>> otherwise (which I now consider a big mistake).
>>
>> Thus, in an attempt to rid commons-collections of such violations, the
>> Bag interface shall be deprecated and replaced by a MultiSet, which is
>> basically the same, but the interface is cleaned up and a little bit
>> enhanced.
>>
>> Operations that take cardinality into account (as currently the Bag
>> does), can be provided as static utility methods in a MultiSetUtils
>> class, or optionally, as additional methods in the interface (I am yet
>> undecided about that).
>>
>> It would be nice if some people could review the current interface and
>> give feedback. There might still be some typos and the base
>> implementation needs to be improved, but it already works as intended.
> 
> I feel a bit weird about the name... why should any kind of "Set"
> include > 1 of given element? If you're already entertaining the idea
> of using externally defined static utility methods, couldn't all the
> "Bag"-style functionality be provided as such against e.g. a List?

@name: the collection javadoc explicitly states the following:

Bags or multisets (unordered collections that may contain duplicate
elements) should implement this interface directly.

see http://docs.oracle.com/javase/7/docs/api/java/util/Collection.html

Furthermore, multisets are a pretty common name, e.g. stl c++ does have
them, google guava has them, ...

Imho, any other name as Bag or MultiSet would be confusing, and we want
to replace our Bag interface, so we do not have much choice, I guess.

@Bag style list: yes, but it would be inefficient. In fact the set-like
operations in CollectionUtils use Lists, although Bags or MultiSets
would be more suitable (this is btw. another change I want to introduce
in 4.1)

Just to list the number of changes of the MultiSet compared to the Bag:

 * containsAll, removeAll and retainAll violated the Collection contract
as they respected cardinality, but these methods can easily be provided
as static utility methods

 * the add method in Bag was not consistent with the Collection contract
wrt to the returned boolean. To support the old behavior one can use
add(obj, 1) and analyze the returned old count like that: add(obj, 1) ==
0 means the obj was not present in the multiset before.

 * the remove method in Bag removed *all* occurrences from the Bag,
while the new MultiSet only removes 1 occurrence, like all Collections.
To support the old behavior, the setCount() method has been added, so
setCount(obj, 0) is equivalent to the old remove().

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[collections] New interface: MultiSet

[Thomas Neidhart]

Hi,

I have just committed the first draft of the new interface MultiSet.

What is the idea behind this?

The rationale of this interface (and corresponding implementations) is
to replace the existing Bag interface, which does violate the Collection
contract. Actually, I wanted to make the Bag interface compliant to the
general Collection contract prior to the 4.0 release, but was convinced
otherwise (which I now consider a big mistake).

Thus, in an attempt to rid commons-collections of such violations, the
Bag interface shall be deprecated and replaced by a MultiSet, which is
basically the same, but the interface is cleaned up and a little bit
enhanced.

Operations that take cardinality into account (as currently the Bag
does), can be provided as static utility methods in a MultiSetUtils
class, or optionally, as additional methods in the interface (I am yet
undecided about that).

It would be nice if some people could review the current interface and
give feedback. There might still be some typos and the base
implementation needs to be improved, but it already works as intended.

Thanks,

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Pool 2.4 jar SNAFU is possible for all components

[Thomas Neidhart]

On 05/30/2015 10:16 PM, Gary Gregory wrote:
> I am wondering if we need an updated build process here.
> 
> At work, in a large Ant based project, we run unit tests against the jars
> files built earlier by the build. We ship jars, not class files, so we test
> jars.
> 
> It seems like a basic design failing in Maven or the way we use Maven.
> 
> Is it possible to set up Maven to run tests against the built jar?

take a look at commons-logging, which uses the failsafe plugin to run
integration tests using the packaged jar.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collections] FluentIterable

[Thomas Neidhart]

On 05/27/2015 06:52 PM, Gary Gregory wrote:
> Let's not reinvent the wheel indeed.

This is not about reinventing the wheel or reimplementing Java 8
streams. The rationale is to glue together existing functionality in
collections by a fluent API, very much the same as already exists in
guava or other libraries (see Seq in https://github.com/jOOQ/jOOL).

Just to give you an example: we had a feature request to override some
method in CollectionUtils to perform filtering a collection and
transforming the result in one go.

This can also be done like that in collections:

Collection result =
  CollectionUtils.collect(
IteratorUtils.filteredIterator(coll.iterator(), predicate),
transformer
  );

which is a bit inconvenient to write, and one has to know the available
iterators to avoid intermediate results being created (which would be
the case when naively using
collect(select(coll, predicate), transformer).

The proposed FluentIterable just simplifies this and similar use-cases.
Furthermore, a FluentIterable operates on an Iterable, as the name
implies, which has the advantage over streams that it can be re-used, as
it really just provides a view of an Iterable.

Note: the API also provides a way to evaluate a fluent iterable and
create a new iterable based on the results. That way one can decouple
the view from the source.

And let's not forget that collections still targets Java 6.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[collections] FluentIterable

[Thomas Neidhart]

Hi,

today, I have committed a first version of a FluentIterable
(COLLECTIONS-464).

Example usage:

List result =
  FluentIterable
.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)
.filter(new Predicate() {
public boolean evaluate(Integer number) {
  return number % 2 == 0;
}
  )
.transform(TransformerUtils.stringValueTransformer())
.limit(3)
.toList();

will return [2, 4, 6]

I still need to add unit tests, but I'd love to get some feedback on the
current API.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[ANNOUNCE] Commons Email version 1.4 released

[Thomas Neidhart]

Hello.

The Apache Commons team is pleased to announce the release of
commons-email-1.4.

Commons-Email aims to provide an API for sending email. It is built on
top of the JavaMail API, which it aims to simplify.

Commons Email can be downloaded from the following page:
  http://commons.apache.org/email/download_email.cgi

For information on Commons Email, please visit our website:
  http://commons.apache.org/email/

Details of the changes and bug fixes in this release can be found in the
release notes:
  http://www.apache.org/dist/commons/email/RELEASE-NOTES.txt

The changes are also available at:
  http://commons.apache.org/email/changes-report.html

Commons Email requires Java 5 or later.

The Maven coordinates are:
org.apache.commons
commons-email
1.4

Best regards,

Thomas, on behalf of the Apache Commons community

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [collections] Preparing a 4.1 release

[Thomas Neidhart]

On 05/24/2015 05:52 PM, Gary Gregory wrote:
> Why not just move to Java 7?

The reason to move to Java 6 is to add support for NavigableMap, which
unfortunately is missing right now.

Moving to Java 7 is not really necessary, the next logical move would be
to Java 8 imho.

In fact it would be worthwhile to have a separate collections component
that targets Java 8 but for this we are clearly lacking the manpower /
interest from the community.

Thomas

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org