Re: [collection][security] InvokerTransformer missused in java object serialisation exploits
On 11/07/2015 04:25 AM, Bernd Eckenfels wrote: > Hello, > > I tried to raise that concern in the message already, but it is probably > worth repeating it explicitly: this is not a real bug > in the Commons-Collection class, and it might not be worse fixing it, as > there are possibly tons of other vectors. This was also addressed by the > original authors in the talk and even here on Twitter: > > https://twitter.com/gebl/status/662754611304996866 > > however, as the "foxglove" article shows, people still point at the > apache project, and after all it is good pratice to reduce footprints > and attack surfaces. it is clear that the InvokerTransformer by itself does not have a bug, but due to the way how java serialization is applied and considering the fact that at least collections-3.2.1 is used *a lot* it would make sense to provide a hardened version of collections to give people a chance to easily avoid this line of attack in their application. Instead of removing the class we could prevent de-serialization of it in the hardened jar. This would not break b/c and it is very unlikely that the InvokerTransformer is serialized in legit ways. Other ways to fix this issue seem to be quite complex, i.e. change the way serialization is performed (use whitelisting, ...), and will most likely not be executed in a timely manner. I wonder if some people already ship a collections version with this class being removed/manipulated, or use a runtime weaving technique to remove the serialization support from it? Setting up a reasonable security policy for this line of attack also seems to be futile as it would have to be so restrictive that most applications probably wont work anymore. > So it seems to be a good idea to discuss some hardening. Unfortunatelly > having a hardened distribution with only this one class removed might > be a bit overkill. Are there other candidates to be left out in such a > more specific distribution? Maybe everything proxy/reflection related? I checked the codebase, and there are other Invoker* type classes but they all use internally the InvokerTransformer. > Both sample payloads have "gadget chains" which do start (readObject()) > in JCL classes and then use pretty generic interfaces like Annotations > or Comparators, so there is really no link between the types and the > specific weakness. I did not see JCL (commons logging?) used in the gadget chains. btw. the technique using the TransformerImpl from xalan (which is part of the oracle/openjdk jre) is really scary as it allows an attacker to inject arbitrary bytecode. Thomas - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [collection][security] InvokerTransformer missused in java object serialisation exploits
On 07/11/2015 10:13, Thomas Neidhart wrote: > On 11/07/2015 04:25 AM, Bernd Eckenfels wrote: >> Hello, >> >> I tried to raise that concern in the message already, but it is probably >> worth repeating it explicitly: this is not a real bug >> in the Commons-Collection class, and it might not be worse fixing it, as >> there are possibly tons of other vectors. This was also addressed by the >> original authors in the talk and even here on Twitter: >> >> https://twitter.com/gebl/status/662754611304996866 >> >> however, as the "foxglove" article shows, people still point at the >> apache project, and after all it is good pratice to reduce footprints >> and attack surfaces. > > it is clear that the InvokerTransformer by itself does not have a bug, > but due to the way how java serialization is applied and considering the > fact that at least collections-3.2.1 is used *a lot* it would make sense > to provide a hardened version of collections to give people a chance to > easily avoid this line of attack in their application. > > Instead of removing the class we could prevent de-serialization of it in > the hardened jar. This would not break b/c and it is very unlikely that > the InvokerTransformer is serialized in legit ways. Rather than having hardened vs unhardened JARs, it would probably be better to use a system property to enable/disable the behaviour. I don't know the code or the vulnerability well enough to know exactly where to put this switch so it prevents the attack but has minimal impact on other uses. Mark - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Smaller Packages / Artifacts / Dependencies
On Fri, 6 Nov 2015 15:06:35 -0600, Ole Ersoy wrote: If math is broken up into smaller artifacts it will make it easier for users to upgrade, even if it it breaks compatibility, as well as speed up the release frequency. So for example: commons-math-optimization (Or even more granular commons-math-optimization-lp, commons-math-optimization-ga, commons-math-optimization-nlp, etc) commons-math-simulation commons-math-statistics commons-math-ai (Neural Networks, ...) etc. I also believe that modularity is a worthy goal. A first step would be to collect some statistics on inter-package dependencies. There will certainly be a "commons-math-core" containing packages like "o.a.c.m.util" and "o.a.c.m.exception". [At some point, releasing separate JARs could also provide us with indirect feedback on which parts of CM are actually used.] Best regards, Gilles - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Version mgt idea
A roughly equivalent alternative would be to release beta artifacts until the API stabilizes and use a different base package and different Maven coordinates for each iteration. For example, commons-math 4.0-beta1 is released with the org.apache.commons:commons-math4-beta1 coordinates and the classes living in the org.apache.commons.math4.beta1 package. Once we are happy with the state of the API we release org.apache.commons:commons-math4 with the org.apache.commons.math4 base package and we stop breaking the binary compatibility. In my opinion the "beta" qualifier better conveys the unstable nature of the API than an arbitrary convention like "the whole 4.x line is unstable". This is a concept people are already familiar with. Emmanuel Bourg - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Smaller Packages / Artifacts / Dependencies
Le 07/11/2015 11:00, Gilles a écrit : > [At some point, releasing separate JARs could also provide us with > indirect feedback on which parts of CM are actually used.] You might find this interesting: https://codesearch.debian.net/perpackage-results/import org.apache.commons.math Emmanuel - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Smaller Packages / Artifacts / Dependencies
I guess searching github is more representative: https://github.com/search?q=org.apache.commons.math3=Code=%E2%9C%93 Am 07.11.2015 um 18:43 schrieb Gilles: On Sat, 7 Nov 2015 16:56:45 +0100, Emmanuel Bourg wrote: Le 07/11/2015 11:00, Gilles a écrit : [At some point, releasing separate JARs could also provide us with indirect feedback on which parts of CM are actually used.] You might find this interesting: Thanks for the link. Interesting indeed. https://codesearch.debian.net/perpackage-results/import org.apache.commons.math I did the more useful search for "org.apache.commons.math3" Apart from CM itself, there is only one (1) other package that uses CM... But isn't looking for Java application usage through the lens of Debian packages totally misleading? I mean, beyond installing the JDK, isn't the Java applications "eco-system" largely independent of a distribution (since a Java program can be "run anywhere")? Gilles - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Version mgt idea
On 11/7/15 10:29 AM, Gilles wrote: > On Sat, 7 Nov 2015 16:52:21 +0100, Emmanuel Bourg wrote: >> A roughly equivalent alternative would be to release beta artifacts >> until the API stabilizes and use a different base package and >> different >> Maven coordinates for each iteration. >> >> For example, commons-math 4.0-beta1 is released with the >> org.apache.commons:commons-math4-beta1 coordinates and the classes >> living in the org.apache.commons.math4.beta1 package. Once we are >> happy >> with the state of the API we release >> org.apache.commons:commons-math4 >> with the org.apache.commons.math4 base package and we stop >> breaking the >> binary compatibility. > > With this scheme, binary compatibility is in effect never broken > since > the top-level package is different in each codebase: > org.apache.commons.math4.beta1 > org.apache.commons.math4.beta2 > etc. > >> In my opinion the "beta" qualifier better conveys the unstable >> nature >> of the API than an arbitrary convention like "the whole 4.x line is >> unstable". > > "math4.beta1" would be fine too (although "math4u0" was a little > shorter). +1 for this idea - gets around the compat breaks within normal-looking major version releases and conveys the right idea in the package name. Also sets clear enough expectation that you are not going to get bug fixes is 4.beta1 without going to 4.beta2 (so bug fixes may come along with compat breaks). Phil > > Regards, > Gilles > >> This is a concept people are already familiar with. > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Version mgt idea
On 11/7/15 12:58 PM, James Carman wrote: > As long as the maven coordinates follow suit, go for it. The community will > let us know if it is a pain in the ass. Also, no need to worry about > even/odd with this approach I think its better to use the even/odd approach with this package naming trick to stay legal within even-numbered lines so we can then keep cutting stable odd-numbered (e.g. 3.x) releases and we can agree that when and even line (e.g. 4) stabilizes we can move to the next odd number (5) and set expectation that releases in that line (5.x) will be stable. Then, e.g. 6. becomes a unstable API line, etc. Otherwise you end up with sequences like 4.beta1, 4.beta2, 4, 4.1, 5.beta1, etc, with no idea when the API has stabilized (most importantly what releases are going to get backward-compatible patch releases with bug-fixes). If we adopt the even/odd convention both for [math] developers and users, we can then set the clear expectation that it's OK to break compat all we like within even-numbered (effectively beta) lines but once we cut an odd-numbered release we are going to keep that API stable until the next odd release. Phil > > On Sat, Nov 7, 2015 at 12:29 PM Gilleswrote: > >> On Sat, 7 Nov 2015 16:52:21 +0100, Emmanuel Bourg wrote: >>> A roughly equivalent alternative would be to release beta artifacts >>> until the API stabilizes and use a different base package and >>> different >>> Maven coordinates for each iteration. >>> >>> For example, commons-math 4.0-beta1 is released with the >>> org.apache.commons:commons-math4-beta1 coordinates and the classes >>> living in the org.apache.commons.math4.beta1 package. Once we are >>> happy >>> with the state of the API we release org.apache.commons:commons-math4 >>> with the org.apache.commons.math4 base package and we stop breaking >>> the >>> binary compatibility. >> With this scheme, binary compatibility is in effect never broken since >> the top-level package is different in each codebase: >> org.apache.commons.math4.beta1 >> org.apache.commons.math4.beta2 >> etc. >> >>> In my opinion the "beta" qualifier better conveys the unstable nature >>> of the API than an arbitrary convention like "the whole 4.x line is >>> unstable". >> "math4.beta1" would be fine too (although "math4u0" was a little >> shorter). >> >> Regards, >> Gilles >> >>> This is a concept people are already familiar with. >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >> For additional commands, e-mail: dev-h...@commons.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Version mgt idea
As long as the maven coordinates follow suit, go for it. The community will let us know if it is a pain in the ass. Also, no need to worry about even/odd with this approach On Sat, Nov 7, 2015 at 12:29 PM Gilleswrote: > On Sat, 7 Nov 2015 16:52:21 +0100, Emmanuel Bourg wrote: > > A roughly equivalent alternative would be to release beta artifacts > > until the API stabilizes and use a different base package and > > different > > Maven coordinates for each iteration. > > > > For example, commons-math 4.0-beta1 is released with the > > org.apache.commons:commons-math4-beta1 coordinates and the classes > > living in the org.apache.commons.math4.beta1 package. Once we are > > happy > > with the state of the API we release org.apache.commons:commons-math4 > > with the org.apache.commons.math4 base package and we stop breaking > > the > > binary compatibility. > > With this scheme, binary compatibility is in effect never broken since > the top-level package is different in each codebase: > org.apache.commons.math4.beta1 > org.apache.commons.math4.beta2 > etc. > > > In my opinion the "beta" qualifier better conveys the unstable nature > > of the API than an arbitrary convention like "the whole 4.x line is > > unstable". > > "math4.beta1" would be fine too (although "math4u0" was a little > shorter). > > Regards, > Gilles > > > This is a concept people are already familiar with. > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > >
Re: [math] Version mgt idea
On 11/7/15 2:15 PM, Phil Steitz wrote: > On 11/7/15 12:58 PM, James Carman wrote: >> As long as the maven coordinates follow suit, go for it. The community will >> let us know if it is a pain in the ass. Also, no need to worry about >> even/odd with this approach > I think its better to use the even/odd approach with this package > naming trick to stay legal within even-numbered lines so we can then > keep cutting stable odd-numbered (e.g. 3.x) releases and we can > agree that when and even line (e.g. 4) stabilizes we can move to the > next odd number (5) and set expectation that releases in that line > (5.x) will be stable. Then, e.g. 6. becomes a unstable API line, > etc. Otherwise you end up with sequences like 4.beta1, 4.beta2, 4, > 4.1, 5.beta1, etc, with no idea when the API has stabilized > (most importantly what releases are going to get backward-compatible > patch releases with bug-fixes). If we adopt the even/odd convention > both for [math] developers and users, we can then set the clear > expectation that it's OK to break compat all we like within > even-numbered (effectively beta) lines but once we cut an > odd-numbered release we are going to keep that API stable until the > next odd release. Sorry, strike that. I think I kind of missed James' point above. As long as once you drop the .beta in a line you keep the API stable, I guess it would be OK to dispense with the even/odd. So basically, we are just saying cut betas until the API stabilizes and only commit to patch releases for releases without the .beta in the package / release name. I would be OK with that. Phil > > Phil >> On Sat, Nov 7, 2015 at 12:29 PM Gilleswrote: >> >>> On Sat, 7 Nov 2015 16:52:21 +0100, Emmanuel Bourg wrote: A roughly equivalent alternative would be to release beta artifacts until the API stabilizes and use a different base package and different Maven coordinates for each iteration. For example, commons-math 4.0-beta1 is released with the org.apache.commons:commons-math4-beta1 coordinates and the classes living in the org.apache.commons.math4.beta1 package. Once we are happy with the state of the API we release org.apache.commons:commons-math4 with the org.apache.commons.math4 base package and we stop breaking the binary compatibility. >>> With this scheme, binary compatibility is in effect never broken since >>> the top-level package is different in each codebase: >>> org.apache.commons.math4.beta1 >>> org.apache.commons.math4.beta2 >>> etc. >>> In my opinion the "beta" qualifier better conveys the unstable nature of the API than an arbitrary convention like "the whole 4.x line is unstable". >>> "math4.beta1" would be fine too (although "math4u0" was a little >>> shorter). >>> >>> Regards, >>> Gilles >>> This is a concept people are already familiar with. >>> - >>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org >>> For additional commands, e-mail: dev-h...@commons.apache.org >>> >>> > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Version mgt idea
On Sat, 7 Nov 2015 16:52:21 +0100, Emmanuel Bourg wrote: A roughly equivalent alternative would be to release beta artifacts until the API stabilizes and use a different base package and different Maven coordinates for each iteration. For example, commons-math 4.0-beta1 is released with the org.apache.commons:commons-math4-beta1 coordinates and the classes living in the org.apache.commons.math4.beta1 package. Once we are happy with the state of the API we release org.apache.commons:commons-math4 with the org.apache.commons.math4 base package and we stop breaking the binary compatibility. With this scheme, binary compatibility is in effect never broken since the top-level package is different in each codebase: org.apache.commons.math4.beta1 org.apache.commons.math4.beta2 etc. In my opinion the "beta" qualifier better conveys the unstable nature of the API than an arbitrary convention like "the whole 4.x line is unstable". "math4.beta1" would be fine too (although "math4u0" was a little shorter). Regards, Gilles This is a concept people are already familiar with. - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Smaller Packages / Artifacts / Dependencies
On Sat, 7 Nov 2015 16:56:45 +0100, Emmanuel Bourg wrote: Le 07/11/2015 11:00, Gilles a écrit : [At some point, releasing separate JARs could also provide us with indirect feedback on which parts of CM are actually used.] You might find this interesting: Thanks for the link. Interesting indeed. https://codesearch.debian.net/perpackage-results/import org.apache.commons.math I did the more useful search for "org.apache.commons.math3" Apart from CM itself, there is only one (1) other package that uses CM... But isn't looking for Java application usage through the lens of Debian packages totally misleading? I mean, beyond installing the JDK, isn't the Java applications "eco-system" largely independent of a distribution (since a Java program can be "run anywhere")? Gilles - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [math] Version mgt idea
On Fri, Nov 6, 2015 at 11:17 AM Phil Steitzwrote: > Here is an idea that might break our deadlock re backward > compatibility, versioning and RERO: > > Agree that odd numbered versions have stable APIs - basically adhere > to Commons rules - no breaks within 3.0, 3.1, ..., 3.x... or 5.0, > 5.1... but even-numbered lines can include breaks - so 4.0 and 4.1 > might not be compatible. We would always maintain both an odd and > even branch - ideally in such a way that when an even numbered line > stabilized it would add a last hurrah of breaks and move to odd. > People wanting stable APIs could just stick with the odd-numbered > lines and [math] developers wanting to experiment with things and > not worry about compatibility could do that in the even-numbered > lines. In effect, this is sort of what we are doing now in 3.x / 4.x. > > I know above violates Commons policy if we actually cut releases > from the even-numbered branches - we would have to get agreement > from the Commons PMC that this is OK or somehow label the releases > differently. Just an idea to get us out of our current bind... > > I'm -1 on this idea. I don't like the idea of causing compatibility breaks within a major revision, no matter what. We should follow semantic versioning guidelines wrt compatibility breaks. If not, you're going to run into trouble in places like OSGi, which assumes you're following those rules.
Re: [math] Version mgt idea
So is the idea to change both the Maven artifact ID and package name for the beta releases? i.e. the stable releases would use Maven AID: commons-math4 package: org.apache.commons.math4 and beta releases: Maven AID: commons-math4-beta{n} package: org.apache.commons.math4.beta{n} Have I got this right? If so, there should be no possibility of jar hell. == However, to test with the beta code will require the user to modify their source code to use the new package names. [And of course fix the source for any API breakages.] It would be nice if the package rename could be avoided. It might be worth checking how the Maven classifier affects the classpath. I suspect that Maven will not consider versions of artifacts whose classifer does not match, so there won't be any danger of math4-beta1 being used instead of math4 unless the user specifically asks for beta1. There could still be a classpath issue if the application has another jar that depends on Commons Math4 and that does not use the classifier. There would then be two incompatible copies of Math4 on the classpath. Given that the beta releases should only be used by the adventurous, this might be a risk worth taking to simply the testing. On 7 November 2015 at 21:25, Phil Steitzwrote: > On 11/7/15 2:15 PM, Phil Steitz wrote: >> On 11/7/15 12:58 PM, James Carman wrote: >>> As long as the maven coordinates follow suit, go for it. The community will >>> let us know if it is a pain in the ass. Also, no need to worry about >>> even/odd with this approach >> I think its better to use the even/odd approach with this package >> naming trick to stay legal within even-numbered lines so we can then >> keep cutting stable odd-numbered (e.g. 3.x) releases and we can >> agree that when and even line (e.g. 4) stabilizes we can move to the >> next odd number (5) and set expectation that releases in that line >> (5.x) will be stable. Then, e.g. 6. becomes a unstable API line, >> etc. Otherwise you end up with sequences like 4.beta1, 4.beta2, 4, >> 4.1, 5.beta1, etc, with no idea when the API has stabilized >> (most importantly what releases are going to get backward-compatible >> patch releases with bug-fixes). If we adopt the even/odd convention >> both for [math] developers and users, we can then set the clear >> expectation that it's OK to break compat all we like within >> even-numbered (effectively beta) lines but once we cut an >> odd-numbered release we are going to keep that API stable until the >> next odd release. > > Sorry, strike that. I think I kind of missed James' point above. > As long as once you drop the .beta in a line you keep the API > stable, I guess it would be OK to dispense with the even/odd. So > basically, we are just saying cut betas until the API stabilizes and > only commit to patch releases for releases without the .beta in the > package / release name. I would be OK with that. > > Phil >> >> Phil >>> On Sat, Nov 7, 2015 at 12:29 PM Gilles wrote: >>> On Sat, 7 Nov 2015 16:52:21 +0100, Emmanuel Bourg wrote: > A roughly equivalent alternative would be to release beta artifacts > until the API stabilizes and use a different base package and > different > Maven coordinates for each iteration. > > For example, commons-math 4.0-beta1 is released with the > org.apache.commons:commons-math4-beta1 coordinates and the classes > living in the org.apache.commons.math4.beta1 package. Once we are > happy > with the state of the API we release org.apache.commons:commons-math4 > with the org.apache.commons.math4 base package and we stop breaking > the > binary compatibility. With this scheme, binary compatibility is in effect never broken since the top-level package is different in each codebase: org.apache.commons.math4.beta1 org.apache.commons.math4.beta2 etc. > In my opinion the "beta" qualifier better conveys the unstable nature > of the API than an arbitrary convention like "the whole 4.x line is > unstable". "math4.beta1" would be fine too (although "math4u0" was a little shorter). Regards, Gilles > This is a concept people are already familiar with. - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org >> > > > - > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org > For additional commands, e-mail: dev-h...@commons.apache.org > - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org
Re: [collection][security] InvokerTransformer missused in java object
Howdy, Thought I'd dive in here. Sorry that things got pointed in your direction on this. That was out of our control. Chris and I had a bunch of conversations about if we thought this was worth reporting to you when we discovered it. Perhaps we made the wrong decision, hard to say. We don't think this is a problem with the functionality in your library, instead its with the core Serialization/Deserialization logic flows. Blaming you is like blaming a library used to build a ROP chain and suggesting we brake or remove the assembly that contributes to that ROP chain. Assuming you fix/change your code, then its just a matter of finding another similar gadget somewhere else Just thought i'd join in the discussion. I've joined the maillist. Thanks, Gabriel Lawrence @gebl
Re: [lang] Outdated Information on Jira Summary Page
Thanks. :) Am 06.11.2015 um 11:46 schrieb Benedikt Ritter: Hello Pascal, 2015-11-05 20:25 GMT+01:00 Pascal Schumacher: Hello everybody, https://issues.apache.org/jira/browse/LANG/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel shows: RELEASE PLAN Lang 2.x - There are no plans for major work on the 2.x line, but bug fix releases are made. Lang 3.4 - Next release from 3.x branch. There haven't been any 2.x release for a few years, so this is misleading. 3.4 was released on months ago. It would be nice if somebody could update this. I've corrected the next release version to 3.5. We keep the 2.x version, because we may release a new version if critical issues show up. But there is no active development anymore. Thank you! Benedikt Thanks, Pascal - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org - To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org For additional commands, e-mail: dev-h...@commons.apache.org