Re: [VOTE] Release Apache Commons JCS 2.0 based on RC1

[Stefan Bodewig]

On 2016-12-25, Thomas Vandahl wrote:

> I would like to release the [jcs] component.

+1

You should probably exclude src/scripts/zipcodes.txt from the RAT
report, but that doesn't affect my vote :-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[RESULT][LAZY] Release Commons Parent 42 Based on RC3

[Stefan Bodewig]

Given this is a lazy vote, the two +1 by Jörg Schaible and myself are
enough.

I'll finish the release process now.

Thanks

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[RESULT] Release Compress 1.13 based on RC1

[Stefan Bodewig]

Hello

with +1 by Oliver Heger, Jörg Schaible and my own implicit one the vote
has passed and I'll start the release process.

I'll give the mirrors time to catch up before sending the announcement
or publishing th site.

Thanks

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [RESULT] Release Compress 1.13 based on RC1

[Stefan Bodewig]

On 2016-12-29, Gary Gregory wrote:

> Did you push to Maven Central too? Just curious.

Yes, but I got confused by the Nexus UI and managed to "promote" Commons
Parent twice (didn't know this was possible). I realized this a few
hours later. I have seen Compress 1.13 in repo2.maven.org by now.

Will send the announcement tomorrow morning my time (UTC+1).

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[ANN] Apache Commons Compress 1.13 Released

[Stefan Bodewig]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

The Apache Commons Team is pleased to announce the release of Apache
Commons Compress 1.13.

Compress 1.13 adds write support for LZMA as raw streams as well as
inside 7z archives. Independently developed compressors and archivers
can now be added via the standard ServiceLocator mechanism and ZIP and
7z archives can now be read from a SeekableByteChannel rather than a
File.

Compress 1.13 is the first release that requires Java 7 at runtime.

The Apache Commons Compress Library defines a Java API for working with
ar, cpio, tar, zip, 7z, arj, dump, gzip, pack200, bzip2, lzma, snappy,
Z, xz and deflate files.

Source and binary distributions are available for download from the
Apache Commons download site:

http://commons.apache.org/proper/commons-compress/download_compress.cgi

When downloading, please verify signatures using the KEYS file available
at the above location when downloading the release.

Changes in this version include:

New features:
o SevenZFile, SevenZOutputFile, ZipFile and
  ZipArchiveOutputStream can now work on non-file resources if
  they can be accessed via SeekableByteChannel.
  Issue: COMPRESS-327.
o Allow compressor extensions through a standard JRE ServiceLoader.
  Issue: COMPRESS-368.
o Allow archive extensions through a standard JRE ServiceLoader.
  Issue: COMPRESS-369.
o Add write support for the legacy LZMA format, this requires XZ
  for Java 1.6.
  Issue: COMPRESS-373.
o Add write support for the legacy LZMA stream to 7z, this
  requires XZ for Java 1.6.
  Issue: COMPRESS-374.
o Allow the clients of ParallelScatterZipCreator to provide
  ZipArchiveEntryRequestSupplier.
  Issue: COMPRESS-375. Thanks to Plamen Totev.
o Add a version-independent link to the API docs of the latest
  release.
  Issue: COMPRESS-372.

Fixed Bugs:
o BitInputStream could return bad results when overflowing
  internally - if two consecutive reads tried to read more than
  64 bits.
  Issue: COMPRESS-363.
o ZipArchiveInputStream.closeEntry does not properly advance to
  next entry if there are junk bytes at end of data section.
  Issue: COMPRESS-364. Thanks to Mike Mole.
o ZipArchiveInputStream now throws an Exception if it encounters
  a broken ZIP archive rather than signaling end-of-archive.
  Issue: COMPRESS-367. Thanks to Mike Mole.
o ScatterZipOutputStream didn't close the StreamCompressor
  causing a potential resource leak.
  Issue: COMPRESS-377.

Changes:
o Update Java requirement from 6 to 7.
  Issue: COMPRESS-360.
o Clarified which TarArchiveEntry methods are useless for
  entries read from an archive.
  Issue: COMPRESS-366.

For complete information on Commons Compress, including instructions
on how to submit bug reports, patches, or suggestions for improvement,
see the Apache Commons Compress website:

http://commons.apache.org/compress/

Stefan Bodewig, on behalf of the Apache Commons community
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlhmBO0ACgkQohFa4V9ri3J7UgCeKXvTwNKeclhNeE52xQSlszPV
eFUAoIAGjm3yKwEKCDj940FCNe2DRhlY
=NPv1
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ALL] how to move a project out of the sandbox?

[Stefan Bodewig]

On 2017-01-03, Rob Tompkins wrote:

> As a relative novice to the Apache/Apache Commons sandboxing process,
> I figure that it might be prudent to ask what the process is for
> moving a component out of the sandbox, before I attempt a release on
> TEXT. So, I was wondering if someone could point me in the direction
> of the documentation or if someone has an answer here.

> I did poke around a bit in markmail to see if such a discussion has
> been had on the dev lis, and don’t see anything in the last several
> years that has anything like this context.

Looks as if we haven't promoted that may components lately. Basically
you need to ensure you are able to get three PMC members to vote on a
release, so the process is to start a vote about promoting the component
and this vote is subject to "Majority Approval"[1].

At least that's what I remember and [2] seems to agree with
it. Strangely I can't find this documented anywhere either.

Stefan

[1] http://www.apache.org/foundation/glossary.html#MajorityApproval

[2] 
http://markmail.org/message/rzrvfn6q2x472yni#query:+page:1+mid:skx4im664wt36qyp+state:results


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ALL] how to move a project out of the sandbox?

[Stefan Bodewig]

On 2017-01-03, Rob Tompkins wrote:

>> On Jan 3, 2017, at 8:57 AM, Stefan Bodewig  wrote:

>>> On 2017-01-03, Rob Tompkins wrote:

>>> As a relative novice to the Apache/Apache Commons sandboxing process,
>>> I figure that it might be prudent to ask what the process is for
>>> moving a component out of the sandbox, before I attempt a release on
>>> TEXT. So, I was wondering if someone could point me in the direction
>>> of the documentation or if someone has an answer here.

>>> I did poke around a bit in markmail to see if such a discussion has
>>> been had on the dev lis, and don’t see anything in the last several
>>> years that has anything like this context.

>> Looks as if we haven't promoted that may components lately. Basically
>> you need to ensure you are able to get three PMC members to vote on a
>> release, so the process is to start a vote about promoting the component
>> and this vote is subject to "Majority Approval"[1].

> Let me make sure I understand correctly here. We need a released
> artifact before approaching the PMC for this vote?

No, I'm sorry I've been too vague.  You start a vote to promote the
component and the same rules apply that would apply to a release vote.

Sorry 

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Promote TEXT to Proper

[Stefan Bodewig]

On 2017-01-03, Rob Tompkins wrote:

> I propose that we move [text] to Commons Proper.

+1

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [commons-complex] Pushing Apache git repo to Github mirror

[Stefan Bodewig]

On 2017-01-04, Eric Barnhill wrote:

> I have read the information regarding read-only git mirrors at
> http://apache.org/dev/git.html .

You are not using a read-only mirror for [complex], so
http://www.apache.org/dev/writable-git applies. Not that it'd say too
much or help in a particular way :-)

> However I have a git mirror for commons-complex at
> https://github.com/apache/commons-complex , it is empty while the
> apache git repo at
> https://git-wip-us.apache.org/repos/asf?p=commons-complex.git has a
> commit.

Has the github integration been set up for this repo at all? This
doesn't happen automatically IIRC, you may need to file a JIRA ticket
for the INFRA project.

It may also depend on which branch you've pushed to and which branch
github integration has been set up for.

> Can anyone tell me how to push this commit to the github mirror, or does it
> propagate over time.

You don't do that yourself, it happens automatically. Sometimes
something goes wrong, though. The mirror of Ant is out of date as the
github integration seems to have had problems the last weekend. Those
hiccups usually heal themselves on subsequent commits.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Best Home for Checksum Implementations?

[Stefan Bodewig]

Hi

inside of [compress] we've already got an implementation of CRC32C (used
by Snappy, "borrowed" from Hadoop) and will soonish have xxhash32 (used
by LZ4).

I was wondering whether these implementations would belong in one of the
other components (as well as - rather than instead of - compress) but
I'm not sure where it would fit.

[codec] and [io] could be candidates but neither holds any
java.util.zip.Checksum implementations as of now.

Neither CRC32C nor xxhash32 are cryptographic hashes so using codec's
digest package feels wrong.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Best Home for Checksum Implementations?

[Stefan Bodewig]

On 2017-01-24, Torsten Curdt wrote:


>>> Maybe create a "checksum" package in codec?

>> Alternatively, we could start a new component called Commons Checksum…

> Depends on how many implementations we can gather I guess.

Compress will have two by the end of this week. Cassandra likely has a
MurmurHash implementation we could port if we wanted to go on a hunt.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Best Home for Checksum Implementations?

[Stefan Bodewig]

On 2017-01-24, Stefan Bodewig wrote:

> On 2017-01-24, Torsten Curdt wrote:


>>>> Maybe create a "checksum" package in codec?

>>> Alternatively, we could start a new component called Commons Checksum…

>> Depends on how many implementations we can gather I guess.

> Compress will have two by the end of this week. Cassandra likely has a
> MurmurHash implementation we could port if we wanted to go on a hunt.

https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=tree;f=hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/hash;h=71bdf14e483f7faee57591784b1b3efa65206800;hb=refs/heads/trunk

Hadoop has murmur and jenkins.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Best Home for Checksum Implementations?

[Stefan Bodewig]

On 2017-01-24, Javen O'Neal wrote:

> [Crypto] is another possible home since hashing and crypto are commonly
> used together,

The checksums we have in [compress] are not cryptographically strong,
adding them to [crypto] would send the wrong signal IMHO.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Best Home for Checksum Implementations?

[Stefan Bodewig]

On 2017-01-24, Gary Gregory wrote:

> Commons Codec already has:

> org.apache.commons.codec.digest.PureJavaCrc32
> org.apache.commons.codec.digest.PureJavaCrc32C

> This is in SVN slated for 1.11.

Oh, didn't see that as I only looked at the released javadocs.

PureJavaCrc32C likely has the same roots as the in in [compress].

To me the digest package doesn't really fit but I've never been involved
with [codec] before.

XXHash32 is ready in [compress] but I'd like to do some benchmarking to
see whether my usage if ByteBuffer is good enough our whether I should
do the byte[] -> int conversions manually.

I can add it to codec once done.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Best Home for Checksum Implementations?

[Stefan Bodewig]

On 2017-01-24, Emmanuel Bourg wrote:

> Le 24/01/2017 à 10:08, Stefan Bodewig a écrit :

>> I was wondering whether these implementations would belong in one of the
>> other components (as well as - rather than instead of - compress) but
>> I'm not sure where it would fit.

> What about a dedicated component for checksum implementations?

This would start with three or four implementations of which people will
probably only use a single one. Components that small feel a bit like
the npm world :-)

> It could also take over the digest package of [codec] which looks a
> bit misplaced (codec is more about bijective coding/decoding than
> one-way hashing).

Interesting idea.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Fwd: commons-compress git commit: COMPRESS-271 xxhash32 checksum

[Stefan Bodewig]

On 2017-01-24, Gary Gregory wrote:

> Once [compress] has its next release, what about copying XXHash32 to
> [codec]? That seems to me like the proper home for such things.

That's why I started the other thread ;-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [DISCUSS] Scala at Commons

[Stefan Bodewig]

On 2017-02-12, Benedikt Ritter wrote:

> Commons mission has always been to provide a home for common _Java_
> libraries. Since my personal and professional interests are slowly
> shifting from Java towards Scala, I’m looking for ways to contribute
> to Open Source with Scala. For this reason I’d like to discuss whether
> Commons could be a home for common Scala libraries as well.

I don't see why not as long as you find three people who are willing to
to touch the code it if needs to get changed ;-)

> For example I’m thinking about wrapper libraries which add Scala sugar
> to the existing libraries.

As additional artifacts, I assume.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[compress] Some Compressor Benchmarks

[Stefan Bodewig]

Hi all

I've started to play with jmh[1] a bit, primarily as I wanted to tune
the LZ77 code a little before proposing the next release. Code is in my
github space[2]

While I was at it I thought it would be nice to compare the performance
of the compression formats we support:

  
https://github.com/bodewig/commons-compress-benchmarks/wiki/Comparison-of-Compression-Formats

As expected the performance of the zlib based native gzip/deflate
compressors/decompressors is superior to all others.  The performance of
the pure Java LZ77 I've implemented for snappy and lz4 doesn't look too
bad, but I'm surprised to see Snappy is so much faster than LZ4 (they
share most of the code). I'll write benchmarks for the Checksum algos
next, it might be the xxhash32 implementation being slower than the
crc32-c we've got.

By all means I'm not an expert in benchmarks, much less so in jmh, so if
I'm doing anything wrong, please tell me.

Stefan

[1] http://openjdk.java.net/projects/code-tools/jmh/
[2] https://github.com/bodewig/commons-compress-benchmarks/

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] Some Compressor Benchmarks

[Stefan Bodewig]

On 2017-02-25, Stefan Bodewig wrote:

> I'll write benchmarks for the Checksum algos next, it might be the
> xxhash32 implementation being slower than the crc32-c we've got.

It is, by a factor of two.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] Some Compressor Benchmarks

[Stefan Bodewig]

On 2017-02-25, Stefan Bodewig wrote:

> On 2017-02-25, Stefan Bodewig wrote:

>> I'll write benchmarks for the Checksum algos next, it might be the
>> xxhash32 implementation being slower than the crc32-c we've got.

> It is, by a factor of two.

It was, by now xxhash32 is a tiny bit faster :-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] Some Compressor Benchmarks

[Stefan Bodewig]

On 2017-02-25, Stefan Bodewig wrote:

> While I was at it I thought it would be nice to compare the performance
> of the compression formats we support:

>   
> https://github.com/bodewig/commons-compress-benchmarks/wiki/Comparison-of-Compression-Formats

> As expected the performance of the zlib based native gzip/deflate
> compressors/decompressors is superior to all others.  The performance of
> the pure Java LZ77 I've implemented for snappy and lz4 doesn't look too
> bad, but I'm surprised to see Snappy is so much faster than LZ4 (they
> share most of the code).

But not of the configuration space. I've started to run the benchmark
for a bigger file (Compress 1.13's uncompressed tar archive) and Snappy
degrades to the performance of bzip2 while lz4 gets even slower than
lzma.

What happens here is that the current code is more or less equivalent to
the "maximum compression" case of zlib (without lazy matches[1] which
would slow down things even more).

It searches backwards for matches until it finds one that is as long as
the maximum match (or data is exceeded). For Snappy maximum match is 64
bytes, for LZ4 it is 2^16-1 (for comparison, for DEFLATE it is 258). In
addition it stops searching backwards at a maximum distance from the
current position wich is 2^15 for Snappy (by default) and 2^16-1 for
lz4 (2^15-3 for DEFLATE).

So the current lz4 configuration tries a harder to find a lot longer
matches than Snappy does (or DEFLATE would do).

zlib has two parameters that tune the performance for the no-lazy match
case we've currently implemented.

* A "nice match length" that says stop searching once you've found a
  match of at least this length.

* a "maximum chain length" that puts a cap on the number of potential
  matches that are tried.

I plan to implement both of these but am a little sure about the
API. They'll go into lz77support.Parameter as optional properties. Of
course we will want people who know what they do to provide arbitrary
values but it would be nices to have something like

new Parameters(WINDOW_SIZE).tunedForBestCompression() or new
Parameters(WINDOW_SIZE).tunedForBestSpeed() but I'm unsure how many
level we need in between (zlib goes from 0 - no compression - to 9 -
maximum compression). And I'm also not sure there is a simple way of
picking the best possible parameters for arbitrary LZ77 variants.

Also, I'm not sure what the default should be for people using
CompressorStreamFactory.createCompressorOutputStream.

Ideas are more than welcome.

Stefan

[1] using lazy matches you find the best back-reference for the data
starting at the current byte and calculate the best back-reference if
you started with the next byte rather than the current byte and pick the
longer one. This is something I also want to add before the 1.14
release.

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[compress] Ramping up for 1.14

[Stefan Bodewig]

Hi all

I've finished the LZ77 tweaks I wanted to apply before the next release,
only need to add some more documentation for the options that allow you
to tweak LZ4 or Snappy. And we should probably put a damper on the
expectations for LZ4 (which labels itself as "fast compression format").

https://github.com/bodewig/commons-compress-benchmarks/wiki/Comparison-of-Compression-Formats

Our implementation of Snappy is quite a bit faster than LZ4, but LZ4's
compression ratio is better.

I'd be grateful for a few more eyes looking over the new code (Snappy
output, lz4 and lz77 packages).

If anybody wants to look over the reports of the current master branch,
I've created a SNAPSHOT site at

https://stefan.samaflost.de/staging/commons-compress-1.14/

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[compress] there seems to be a bug within LZ4 compression

[Stefan Bodewig]

Hi

after fixing my benchmark to actually measure decompression I'm facing a
"checksum mismatch" when compressing the source tarball of Commons
Compress 1.13. It seems to be sheer luck I'm using this file as a
roundtrip using many other files - including the binary tarball - just
works.

By now I've pinpointed the problem to be inside the compression code as
not only the checksum doesn't match but the content of the tar (after
running command line lz4 over the compressed output) is different from
the original. But still a valid tar, mind you.

I'll need to debug this further which is cumbersome. In either case this
will block all attempts of cutting a 1.14 release.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] there seems to be a bug within LZ4 compression

[Stefan Bodewig]

On 2017-04-05, Stefan Bodewig wrote:

> I'll need to debug this further which is cumbersome. In either case this
> will block all attempts of cutting a 1.14 release.

Fixed now, BTW.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [DRAFT] Board report

[Stefan Bodewig]

On 2017-04-11, Gary Gregory wrote:

> ## Committer base changes:

...

> - (Missing from LDAP people) was added as a committer on Thu Apr 06
> 2017

(Missing from LDAP people) ? Actually, I don't recall a committer vote
in April and neither does my mail archive.

Rest looks good, thanks

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: commons-compress git commit: reduce GC pressure by avoiding File(In|Out)putStreams

[Stefan Bodewig]

On 2017-04-11, Benedikt Ritter wrote:

>> Am 11.04.2017 um 14:47 schrieb bode...@apache.org:

>> Repository: commons-compress
>> Updated Branches:
>>  refs/heads/master 5ce60be9d -> 08113862c


>> reduce GC pressure by avoiding File(In|Out)putStreams

> So you also saw the blog post by CloudBees ;-)

Yes :-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ALL] The Commons Math issue

[Stefan Bodewig]

[sorry for the comments from the peanut gallery, I haven't been
following MATH and likely will never contribute anything substantial]

On 2017-04-12, Gilles wrote:

> On Wed, 12 Apr 2017 12:03:05 +0200, Emmanuel Bourg wrote:

>> What do you expect now that is blocked by the PMC?

> With "Commons RNG", I think that I showed that the "new, small,
> focused components" route is the right one.[1]

I agree.

> Last time I acted (to request a "git" repository from INFRA), you
> (IIRC, pardon me if I'm wrong) complained ;-) that it had not been
> agreed upon...

The details are escaping me, maybe you need to state your desire to
break a new piece out of MATH as a separate component first and give
people time to comment before requesting a repo for it? Alternatively
start inside the sandbox (I hope nobody has ever stopped anybody from
creating a sandbox component - but I may certainly have missed it).

> IMO, there is a contradiction in the PMC being both passive (not
> contributing to the overall health of CM[2]) and active (in preventing
> "do-ocracy" wrt the choice of a roadmap for CM[3]).

You can count myself into the camp of people who are willing to let a
component go dormant if it doesn't get maintained. But I'm unlikely to
go actively looking for unmaintained components. Looking at the commits
of commons-math it seems to get some love from time to time, not enough
that anybody would want to cut a release, though.

I'm not sure we need a roadmap. IMHO if you can identify a viable subset
of MATH you want to maintain as a separate component, then you should be
able to do just that. At the same time this shouldn't prevent anybody
else from working on MATH if they really want to.

> Moreover, the lack of interest shown by the PMC:
>   http://markmail.org/thread/pgrgnnwjnqrtzrw3

proves that nobody apart from Rob is willing to cut a MATH release right
now. The same would likely be true for a few other components as well
(the last DISCOVERY release is almost six years old) and it doesn't
worry me at all. It only says "MATH may be a candidate for going
dormant" to me.

This is a PMC member view and certainly not the view of a MATH user. But
you are asking for a PMC opinion.

> is a worrying indication that any further work can be doomed to not
> get the minimal support for an official release, even if there would
> be no "technical reason"[4] to prevent such release.

Not really. I regularly ask for feedback on COMPRESS without getting any
responses but still manage to collect enough votes when I want to carve
a new release. Fortunately there are a few "hero"s (many thanks to you!)
on this list who manage to review RCs and cast votes for almost every
candidate thrown their way, I'm not one of them.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ALL] The Commons Math issue

[Stefan Bodewig]

On 2017-04-12, Gilles wrote:

> [Reminder: a big part of "Commons RNG" was developed inside CM and
> most PMC people did not even know about it (although I was opening
> JIRA issues all along.  Hence creating a "git" repository is not
> futile if it can raise awareness.]

By now you've probably learned that people won't look at JIRA issues
raised for components they don't work on. At least I don't :-)

> On Wed, 12 Apr 2017 15:22:23 +0200, Stefan Bodewig wrote:

>> On 2017-04-12, Gilles wrote:

>>> IMO, there is a contradiction in the PMC being both passive (not
>>> contributing to the overall health of CM[2]) and active (in
>>> preventing "do-ocracy" wrt the choice of a roadmap for CM[3]).

>> You can count myself into the camp of people who are willing to let a
>> component go dormant if it doesn't get maintained. But I'm unlikely
>> to go actively looking for unmaintained components.

> a _lot_ of work has been done (since at least 4 years) on the branch
> that was bound to become CM 4.0.  I'm inclined to think that it
> deserves more than being thrown away.

I'm not suggesting to throw away anything. All I said was that I'm
prepared to move unmaintained components to dormant where anybody can
pick them up again later. You're saying MATH isn't unmaintained, that's
fine.

I'm still not sure where you see do-ocracy being prevented. If anybody
wanted to RM a MATH release, they can do so. And to me it looks as if
nobody was preventing you - or anybody else - from creating new
components seeded by code taken from MATH (as long as the number doesn't
get scary, I hear you, Oliver :-).

It seems creating a git repository as the first step may not be the
preferred approach, though.

>> I'm not sure we need a roadmap. IMHO if you can identify a viable
>> subset of MATH you want to maintain as a separate component, then you
>> should be able to do just that. At the same time this shouldn't
>> prevent anybody else from working on MATH if they really want to.

> Exactly (although the latter did not happen, and it's something for
> the PMC to take into account when alternative are proposed).

It is probably a lot easier to accept "let's create a new component that
focusses on X with code seeded from MATH" than "here is a big plan for
how we want to deal with breaking up MATH". I prefer the "small steps"
approach taken with RNG and NUMBERS.

> As you know, this CM issue has created a lot of grievance.

> I do complain that the PMC did not fulfill its role, by not even
> trying to safe-guard the "Commons" project's integrity.

> I expect the "Commons" PMC to _support_ the people who work here
> (cf. "git log").

I read you expect the PMC to do something, but unfortunately I don't
understand what it is that you want the PMC to do. Maybe we are are
interpreting the role of the PMC differently.

In what way has the integrity of the Commons project been endangered?
I've seen people fork the code of MATH - which is fine by our license -
and move to work in a different environment - which is their choice and
I'm not willing to judge them. But the code is still here and anybody is
free to keep working on it. No danger for the Commons project IMHO,
maybe a danger for the MATH component going dormant which is something
that may happen to any other component as well when people stop working
on them.

I've seen you sticking around to work on MATH and keeping the parts
alive that you care deeply about and finding new contributors that share
those goal - which is great.

The PMC has not been standing in the way of RNG or NUMBERS, maybe
discussions have been taking longer than you'd have wanted. But that's
what you get inside a chatty community (I'm deliberatly rate-limiting my
responses :-). The new contributors have been made committers by the
PMC.

I'm confident the PMC won't stand in the way of creating new
self-contained components in the future, some members of the PMC may
quibble over the details, though, and you'll need even more discussions.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ALL] The Commons Math issue

[Stefan Bodewig]

On 2017-04-13, Jörg Schaible wrote:

> Oliver Heger wrote:

>> I am fine with Commons RNG and Commons Numbers.

>> I would feel uneasy with a significant number of mathematical components
>> extracted from [math] that are added to Commons, even if they are small
>> and focused. It would seem strange if you opened the Commons Web site
>> and about half of the components were math-related. If this is the goal,
>> I would prefer to start again the top-level-project discussion.

> Then let's continue with it unless we *have* a significant number of
> components. If those attract in completion enough contributors/committers,
> we can again try to form a TLP and donate all of them. IMHO the creation of
> RNG and Numbers was healthy to our ecosystem, therefore I don't see a reason
> to stop with the separation of more component out of Math now.

+1

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [COMPRESS] zip-bomb prevention for Z?

[Stefan Bodewig]

On 2017-04-13, Allison, Timothy B. wrote:

> On TIKA-1631 [1], users have observed that a corrupt Z file can cause
> an OOM at Internal_.InternalLZWStream.initializeTable.

Ouch.

> Should we try to protect against this at the Tika level, or should we
> open an issue on commons-compress's JIRA?

If there is anything COMPRESS can do to detect and avoid the situation,
then please open an issue over here.

> A second question, we're creating a stream with the
> CompressorStreamFactory when all we want to do is detect.  Is there a
> recommended way to detect the type of compressor without creating a
> stream?

This has never been a goal of the *StreamFactory's but it would be
pretty easy to add "guess" or "detect" methods to them. At least for the
formats built-in into Compress.

Since 1.13 we support extensions of the factories via ServiceLoader and
LZO for Java[2] uses it for example.  Looking into the code again, we
don't support autodetection for formats added via ServiceProvider
anyway, so this is no restriction for the Tika case.

If we wanted to add such a method, what would the return value be? One
of the String constants contained inside the *Factory classes,
likely. Tika would have to be prepared for new strings popping up when
using a newer version of Compress (1.14 will add "lz4-framed" for
example).

Stefan

> [1] https://issues.apache.org/jira/browse/TIKA-1631

[2] https://github.com/shevek/lzo-java

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ALL] The Commons Math issue

[Stefan Bodewig]

On 2017-04-13, Gilles wrote:

> On Thu, 13 Apr 2017 11:53:27 +0200, Stefan Bodewig wrote:
>> On 2017-04-12, Gilles wrote:

>>> [Reminder: a big part of "Commons RNG" was developed inside CM and
>>> most PMC people did not even know about it (although I was opening
>>> JIRA issues all along.  Hence creating a "git" repository is not
>>> futile if it can raise awareness.]

>> By now you've probably learned that people won't look at JIRA issues
>> raised for components they don't work on. At least I don't :-)

> A priori, I don't have any problem with an individual taking that
> stance. [I do it too, because a day is only 24 hours long.]

> But then, one is not entitled to claim a say about the issues which
> he let pass...

I don't recall ever claiming anything like this.

Not sure what you are trying to say here. It could be that you are
trying to attack me but I hope you are not. Email is a difficult beast,
in particular emails written in a foreign language (German is my native
language and I don't think English is yours, either, there is lot of
room for misunderstandings).

>> I prefer the "small steps" approach taken with RNG and NUMBERS.

> That's what I've been advocating all along.

Fine, then we all seem to be on the same page. Let's move on.

>> I read you expect the PMC to do something, but unfortunately I don't
>> understand what it is that you want the PMC to do. Maybe we are are
>> interpreting the role of the PMC differently.

>> In what way has the integrity of the Commons project been endangered?
>> I've seen people fork the code of MATH - which is fine by our license
>> - and move to work in a different environment - which is their choice
>> and I'm not willing to judge them.

> And I think that the PMC has been wrong in passively accepting the
> "surprise" fork.

Oops, I thought you were talking about the PMC harming MATH right now,
after all you started the thread based on the report for the past
quarter, not years ago. I'm sorry I misunderstood you.

> Because it came from _inside_ the community, the PMC would have
> been right to demand that a reasonable attempt be made at exposing
> the reasons, and at trying to fix issues while preserving the
> community.

> I was hurt by the fork, and the way it happened.  And I was hurt that
> the PMC did not see anything wrong in "community fellows" keeping me
> in the dark for five months, to work alone on a doomed project, while
> they were sneakily setting up an alternative environment.

I understand the action hurt you. Absolutely.

On the road that led to people starting their fork somewhere else there
have been lots of heated arguments. It looked like bad flame-wars that
happen in other communities as well and yes, the PMC should probably
have tried to stop them and remind people to treat each other with
respect. We didn't and I think this has been acknowledged in the past. I
don't have the links ready but I know several PMC members have said so
already.  We try to learn from it.

We don't need to tell the board that we are still trying to do better
with each report, though. :-)

To be frank my recollection of said arguments is not one where one side
was the reasonable voice and victim of attacks while the other side was
wielding flame-throwers. We should have called out *all* of you.

As to the action of forking itself, I still don't see anything the PMC
should have done about it. We should have interfered before it
happened. That doesn't mean I'm convinced that we could have been
successfull back then.

>> I've seen you sticking around to work on MATH and keeping the parts
>> alive that you care deeply about and finding new contributors that
>> share those goal - which is great.

> Or stupid...

No more stupid than most of us working on any other component in their
spare time.

>> The PMC has not been standing in the way of RNG or NUMBERS, maybe
>> discussions have been taking longer than you'd have wanted.

> Yes that's one of the things that prevent "do-ocracy": someone
> willing to do the work is stalled by (non-technical) arguments
> from someone not intending to back them with actual work (same
> reference):

That's the price of consensus. You don't get to choose who needs to
agree with you, you have to convince all people who show up. This takes
time and drains energy. Yes, a dictator style development approach can
move a lot faster. This is a drawback of consensus based development
that we have accepted, or else we'd all by playing with our github repos
on our own.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [COMPRESS] zip-bomb prevention for Z?

[Stefan Bodewig]

On 2017-04-14, Allison, Timothy B. wrote:

>>> If there is anything COMPRESS can do to detect and avoid the
>>> situation, then please open an issue over here.

> Done: COMPRESS-385, PR submitted

Thanks.

>>> If we wanted to add such a method, what would the return value be?
>>> One of the String constants contained inside the *Factory classes,
>>> likely. Tika would have to be prepared for new strings popping up
>>> when using a newer version of Compress (1.14 will add "lz4-framed"
>>> for example).

> Y, I'm ok with a String...perhaps longer term or for 2.0, move to an
> enum?  Thank you for the heads-up!

enum wouldn't work for formats added via ServiceLoader. LZO supports a
couple of names of its own and you couldn't inject them into the enum.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [COMPRESS] zip-bomb prevention for Z?

[Stefan Bodewig]

On 2017-04-14, Allison, Timothy B. wrote:

>> enum wouldn't work for formats added via ServiceLoader. LZO supports a 
>> couple of names of its own and you couldn't inject them into the enum.

> Doh!  Got it.  New code base...Sorry.

No problem :-)

I've taken a prolonged offline Easter-weekend, will look into the your
PRs and comments in JIRA the coming days.

Many thanks

 Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ALL] The Commons Math issue

[Stefan Bodewig]

[sorry, been offline for a few days and don't want to re-start a thread
that seems to have come to a conclusion, just need to clarify one thing]

On 2017-04-14, Gilles wrote:

> On Fri, 14 Apr 2017 16:34:22 +0200, Stefan Bodewig wrote:
>> On 2017-04-13, Gilles wrote:

>>> On Thu, 13 Apr 2017 11:53:27 +0200, Stefan Bodewig wrote:

>>>> By now you've probably learned that people won't look at JIRA
>>>> issues raised for components they don't work on. At least I don't
>>>> :-)

>>> A priori, I don't have any problem with an individual taking that
>>> stance. [I do it too, because a day is only 24 hours long.]

>>> But then, one is not entitled to claim a say about the issues which
>>> he let pass...

>> Not sure what you are trying to say here. It could be that you are
>> trying to attack me but I hope you are not.

> English is not my native language, but I don't think that the
> sentence you refer to contains anything offensive: "one" is not
> "you".

Thank you.

The way to construct the attack is that I talk about me and you talk
about "one" and I had no reason to assume the "one" could be anybody
other than me. I'm still not sure why you said the above at all as
Emmanuel wasn't participating in the conversation at all.

Thanks for clarifying it.

>>> Yes that's one of the things that prevent "do-ocracy": someone
>>> willing to do the work is stalled by (non-technical) arguments from
>>> someone not intending to back them with actual work (same
>>> reference):

>> That's the price of consensus. You don't get to choose who needs to
>> agree with you, you have to convince all people who show up. This
>> takes time and drains energy. Yes, a dictator style development
>> approach can move a lot faster. This is a drawback of consensus based
>> development that we have accepted, or else we'd all by playing with
>> our github repos on our own.

> It's also my opinion that we are more strongly contributing to the
> open source model by being a team.  But I often have the feeling that
> the PMC operates as an aggregate of individualities rather than as a
> community.

Well, that's because we are all individuals. :-)

For Commons it is more difficult to form a community than for most
"normal" ASF projects as the least common denominator is much
smaller. In "normal" project you've got a product vision and a shared
code base to align folks. All we've got is the goal to produce something
useful - where many of us have different ideas of what will be useful -
and the idea of doing so via collaboration.

IMHO we need to accept that we are a pretty diverse bunch of
people. We've got different reasons for being here and we are certainly
different in our approaches of reaching our goals. Mutual respect is
what can bind us - and I believe this is what was lost in the MATH case.

> There are low-level requirements (naming of releases, vote counts,
> etc.), but hardly any policies.

Well, some of us may enjoy working here because we don't have that many
rules and policies. I think I am one of those.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: commons-compress git commit: Unnecessary casts.

[Stefan Bodewig]

On 2017-05-02,  wrote:

> Unnecessary casts.

> -n = (long) len - bytes;
> +n = len - bytes;

Sonar complains without them.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[all][osgi] how to deal with non-bundle dependencies?

[Stefan Bodewig]

Hi all

I must admit I've never been a fan of OSGi and likely don't know enough
about it.

Compress like many of the components around here is an OSGi bundle and I
recall I submitted a trivial patch to XZ for Java - our only dependency
as of now - to turn it into a bundle as well.

How does one deal with adding dependencies on jars that are not bundles?
Is it possible at all?

A quick search seems to imply we'd either have to bundle the dependency
jar with ours or turn it into a bundle ourselves (by wrapping it).

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: commons-compress git commit: [COMPRESS-392] Add Brotli decoder based on the Google Brotli library.

[Stefan Bodewig]

On 2017-05-02,  wrote:

> @@ -245,6 +251,7 @@ jar, tar, zip, dump, 7z, arj.
>  
>
>  
> org.tukaani.xz;resolution:=optional
> +
> org.brotli.dec;resolution:=optional
>
>  
>

org.brotli:dec is not an OSGi bundle as far as I can tell. Is this going
to work or will it break Compress in an OSGi environment?

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: commons-compress git commit: [COMPRESS-392] Add Brotli decoder based on the Google Brotli library.

[Stefan Bodewig]

On 2017-05-03, Gary Gregory wrote:

> On May 2, 2017 9:19 PM, "Stefan Bodewig"  wrote:

>> On 2017-05-02,  wrote:

>>> @@ -245,6 +251,7 @@ jar, tar, zip, dump, 7z, arj.
>>>  
>>>
>>>  
>>> org.tukaani.xz;resolution:=optional
>>> +
>>> org.brotli.dec;resolution:=optional
>>>
>>>  
>>>

>> org.brotli:dec is not an OSGi bundle as far as I can tell. Is this going
>> to work or will it break Compress in an OSGi environment?

Looking at what we did back with
https://issues.apache.org/jira/browse/COMPRESS-199 (Jukka's proposed
patch) this may actually be enough.

> Good question... I do not see use of the bundle plugin in
> https://github.com/google/brotli/blob/master/java/org/brotli/dec/pom.xml

> Assuming I am looking in right place...

> Maybe a PR would help ;-)

Yeah, I did that for XZ for Java at that time.

dec probably doesn't really use Maven to build even though there is a
POM. I suspect it is using Bazel (see the BUILD file inside the source
directory), I'd need to figure out how to add a manifest to a Bazel
built jar first. Might move "take a closer look at Bazel" further up on
my TODO list. :-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [all][osgi] how to deal with non-bundle dependencies?

[Stefan Bodewig]

On 2017-05-02, Matt Sicker wrote:

> Apache ServiceMix tends to repackage a lot of 3rd party libraries as
> bundles, so that's one method. End users can use things like bnd to
> generate a manifest for 3rd party jars (e.g., installing jars using the
> "wrap:" URI in Apache Karaf), though that's kind of similar to automodules
> in JPMS which isn't the best thing.

In our case with org.brotli:dec at Compress the "automodules" approach
would be good enough. The whole library has a single public class and no
dependencies, the bundle definition would be trivial.

We've now marked the dependency with "resolution:=optional" and hope
this allows Compress to remain a valid bundle. Do you know whether there
is an easy way to test whether a library would work as a bundle without
actually trying to deploy an application using it into a container?

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: commons-compress git commit: [COMPRESS-392] Add Brotli decoder based on the Google Brotli library.

[Stefan Bodewig]

On 2017-05-04, Stefan Bodewig wrote:

> On 2017-05-03, Gary Gregory wrote:

>> Maybe a PR would help ;-)

> dec probably doesn't really use Maven to build even though there is a
> POM.

Turns out a patch to the POM has been good enough :-)

https://github.com/google/brotli/pull/545

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [all][osgi] how to deal with non-bundle dependencies?

[Stefan Bodewig]

On 2017-05-04, Matt Sicker wrote:

> Without a container, the "easy" way would be to embed Felix and start up
> the OSGi framework itself. The easiest way to do this in unit tests would
> be using something like Pax Exam <
> https://ops4j1.jira.com/wiki/display/PAXEXAM4/Pax+Exam>.

Ah, many thanks to you and Bertrand

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] thoughts on releasing 1.14?

[Stefan Bodewig]

On 2017-05-07, Gary Gregory wrote:

> What are your thoughts on releasing compress 1.14?

I was ready to cut the release around easter when a few valuable patches
came in and needed to get integrated.

> What's left on the must do list?

COMPRESS-391 (waiting for an updated PR) and waiting for a new release
of the Brotli lib that includes the patch which turns it into an OSGi
bundle.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] thoughts on releasing 1.14?

[Stefan Bodewig]

On 2017-05-07, Stefan Bodewig wrote:

> On 2017-05-07, Gary Gregory wrote:

>> What's left on the must do list?

> COMPRESS-391 (waiting for an updated PR) and waiting for a new release
> of the Brotli lib that includes the patch which turns it into an OSGi
> bundle.

Scratch the second part, release has been cut less than an hour ago. :-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [COMPRESS] Anyone implemented "pigz"?

[Stefan Bodewig]

On 2017-05-10, Gary Gregory wrote:

> I think the question is can/should [Compress] use any of the stock code
> in java.util.zip in a multi-threaded fashion for performance gains.

We rely on java.util.zip.Deflater for DEFLATE which isn't thread safe by
itself.

But we could implement the same strategy pigz uses, which is to break up
the stream into chunks and work on the chunks in parallel. Combining the
output of several streams may become tricky using the Java API.

If my first read of the comments in
https://github.com/madler/pigz/blob/master/pigz.c is correct then we'd
need to manipulate the output of Deflater in order to strip headers and
trailers and insert empty stored blocks as well as create headers and
trailers of our own for the combined output.

In theory we could implement something like pigz on top of the LZ77
support I've added for Snappy and LZ4 (and some additional Hufmann code
yet to be written) but it would be slower than zlib - probably a lot -
and likely eat up the speed gain provided by parallel processing.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[VOTE] Release Compress 1.14 Based on RC1

[Stefan Bodewig]

Hi all

we've added a whole lot of new features in git.

Compress 1.14 RC1 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/compress/ (svn revision 
19628)

The tag is 1.14-RC1 (92708a964) on commit dd7c770:

https://git-wip-us.apache.org/repos/asf?p=commons-compress.git;a=commit;h=dd7c7702bf51886aa8bd88b24d98619f310fbeda

Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1246/org/apache/commons/commons-compress/1.14/

These are the Maven artifacts and their hashes

commons-compress-1.14.jar
SHA1: 7b18320d668ab080758bf5383d6d8fcf750babce
MD5: 6dbbb8b86e98bde1f240ea475cf829fb

commons-compress-1.14-javadoc.jar
SHA1: 93dd5215d4a359cb4f48402456f25cc3e078f3cd
MD5: 8995c93dd0d58fe12500bb9ecc855048

commons-compress-1.14-sources.jar
SHA1: 43a23e357fa7b187c261520914c0b001ea1a62a2
MD5: 6623f55fd998191a15809405af231393

commons-compress-1.14-tests.jar
SHA1: df238b015789a3be595fbc7e1127b6e2f906be99
MD5: 14d48f071c5ee7944e6ac1ddd7a2d4d1

commons-compress-1.14-test-sources.jar
SHA1: a439baff85f1c5d13fcc396e416a8d21af213ec8
MD5: e48038cd5bfb4ed9772bd1e2c97ffbd1

commons-compress-1.14.pom
SHA1: 1360415ca6825c7195366620256c0797463d5807
MD5: 8df9d49405e3c7afc8ccf55cd207bbd5

I have tested this with JDK 7 and 8 ... using Maven 3.3.9.

Details of changes since 1.13 are in the release notes:
https://dist.apache.org/repos/dist/dev/commons/compress/RELEASE-NOTES.txt

https://stefan.samaflost.de/staging/commons-compress-1.14/changes-report.html

Site:
https://stefan.samaflost.de/staging/commons-compress-1.14/
  (as usual I'll re-create the site once the vote has passed and will
add the 1.14 javadocs then)

Japicmp Report (compared to 1.13):
https://stefan.samaflost.de/staging/commons-compress-1.14/japicmp.html

RAT Report:
https://stefan.samaflost.de/staging/commons-compress-1.14/rat-report.html

KEYS:
  https://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.
This vote will close no sooner that 72 hours from now,
i.e. sometime after 19:30 UTC 14-May 2017

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Thanks!

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[compress] ZIP Integration Tests (was Re: [VOTE] Release Compress 1.14 Based on RC1)

[Stefan Bodewig]

On 2017-05-12, Bruno P. Kinoshita wrote:

> Build with `mvn clean test` passing OK in my local environment

> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 
> 2015-11-11T05:41:47+13:00)
> Maven home: /opt/maven
> Java version: 1.8.0_131, vendor: Oracle Corporation
> Java home: /usr/lib/jvm/java-8-oracle/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "4.4.0-77-generic", arch: "amd64", family: "unix"

> Site can also be built with `mvn package site`. No outstanding issues in the 
> reports.

> Integration tests with `mvn test -Prun-tarit` are run quickly and report no 
> issues.

> Integration tests with `mvn test -Prun-zipit` take between 5-10 minutes, and 
> fail (tried twice). Sample error logs: 
> https://gist.github.com/kinow/b7a773721ae94b55b299d421d4655fa3

> Does anyone know if the integration tests are maintained, or whether they are 
> failing for not being updated with the other changes?

Many thanks for running the tests, Bruno!

I must admit I haven't run them for a long time. The ZIP tests require >
5GB of disk space that's why we don't run them on Jenkins. Your gist
doesn't look like a disk space issue, though.

The tests are expected to run very long but not expected to fail. I
don't recall any change that could have broken them but they are
sensitive to the way the file meta data is written.

I'll run the tests myself to see what happens here.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] ZIP Integration Tests (was Re: [VOTE] Release Compress 1.14 Based on RC1)

[Stefan Bodewig]

On 2017-05-12, Stefan Bodewig wrote:

> I'll run the tests myself to see what happens here.

$ mvn test -Dtest=Zip64SupportIT#writeSmallStoredEntryKnownSizeToFileModeAlways 
-Prun-zipit

...

Failed tests: 
  
Zip64SupportIT.writeSmallStoredEntryKnownSizeToFileModeAlways:1618->withTemporaryArchive:2323
 arrays first differed at element [4]; expected:<64> but was:<-1>

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

I'll try to understand what's going on.

Thanks again

   Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] ZIP Integration Tests (was Re: [VOTE] Release Compress 1.14 Based on RC1)

[Stefan Bodewig]

On 2017-05-12, Stefan Bodewig wrote:

> On 2017-05-12, Stefan Bodewig wrote:

>> I'll run the tests myself to see what happens here.

> $ mvn test 
> -Dtest=Zip64SupportIT#writeSmallStoredEntryKnownSizeToFileModeAlways 
> -Prun-zipit

> ...

> Failed tests:
>   
> Zip64SupportIT.writeSmallStoredEntryKnownSizeToFileModeAlways:1618->withTemporaryArchive:2323
>  arrays first differed at element [4]; expected:<64> but was:<-1>

> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

> I'll try to understand what's going on.

First data point, it has been failing since 1.11, it passes with 1.10.

I'll wade through the changes we've made to ZipArchiveOutputStream, at
first glance the test verifies what I'd expect the archive to contain.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] ZIP Integration Tests (was Re: [VOTE] Release Compress 1.14 Based on RC1)

[Stefan Bodewig]

On 2017-05-12, Stefan Bodewig wrote:

> On 2017-05-12, Stefan Bodewig wrote:

>> On 2017-05-12, Stefan Bodewig wrote:

>>> I'll run the tests myself to see what happens here.

>> $ mvn test 
>> -Dtest=Zip64SupportIT#writeSmallStoredEntryKnownSizeToFileModeAlways 
>> -Prun-zipit

>> ...

>> Failed tests:
>>   
>> Zip64SupportIT.writeSmallStoredEntryKnownSizeToFileModeAlways:1618->withTemporaryArchive:2323
>>  arrays first differed at element [4]; expected:<64> but was:<-1>

>> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

>> I'll try to understand what's going on.

> First data point, it has been failing since 1.11, it passes with 1.10.

> I'll wade through the changes we've made to ZipArchiveOutputStream, at
> first glance the test verifies what I'd expect the archive to contain.

https://github.com/apache/commons-compress/pull/10

we forgot to adapt the test, will do so now.

Given the test was wrong, not the implementation I think I don't need to
cancel the vote.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [COMPRESS] Anyone implemented "pigz"?

[Stefan Bodewig]

On 2017-05-10, Matt Sicker wrote:

> Would the scattering and gathering byte channel APIs in java.nio be helpful
> in splitting up a stream into chunks for parallel processing?

Possibly. pigz breaks up the stream into chunks of 128k and using the
scattering part we should be able to do the same. I'm not so sure about
gathering as we'd need to massage the individual outputs and create new
overall headers and trailers.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] ZIP Integration Tests (was Re: [VOTE] Release Compress 1.14 Based on RC1)

[Stefan Bodewig]

On 2017-05-12, Gary Gregory wrote:

> Can you please add a BUILDING.txt file to the root of the project?

Done.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[RESULT] Release Compress 1.14 Based on RC1

[Stefan Bodewig]

Hi all

with +1s by Gary Gregory and Bruno P. Kinoshita, my implicit one and no
other votes, the vote has passed.

I'll publish the artifacts now and will announce the relase after the
mirrors have had time to catch up.

Thanks to those who took the time to review the release

   Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[ANN] Apache Commons Compress 1.14 Released

[Stefan Bodewig]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

The Apache Commons Team is pleased to announce the release of Apache
Commons Compress 1.14.

Compress 1.14 adds support for new compression formats:

* write support for Snappy (read-only support has been added with
  Compress 1.7)
* read-only support for Brotli by wrapping the otherwise optional
  org.brotli:dec library
* Support for LZ4.

In addition new methods inside the factories separate format detection
from providing the corresponding streams and the amount of memory being
used can be limited for selected compression formats. Finally we've
fixed bugs and added new features for the zip, Snappy, CPIO and BZIP2
formats.

The Apache Commons Compress Library defines an API for working with
compression and archive formats.  These include: bzip2, gzip, pack200,
lzma, xz, Snappy, traditional Unix Compress, DEFLATE, LZ4, Brotli and
ar, cpio, jar, tar, zip, dump, 7z, arj.

Source and binary distributions are available for download from the
Apache Commons download site:

http://commons.apache.org/proper/commons-compress/download_compress.cgi

When downloading, please verify signatures using the KEYS file available
at the above location when downloading the release.

Changes in this version include:

New features:
New features:
o Added write support for Snappy.
  Issue: COMPRESS-246.
o Added support for LZ4 (block and frame format).
  Issue: COMPRESS-271.
o Add static detect(InputStream in) to CompressorStreamFactory
  and ArchiveStreamFactory
  Issue: COMPRESS-385.
o Added a way to limit amount of memory ZCompressorStream may
  use.
  Issue: COMPRESS-382. Thanks to Tim Allison.
o Added a way to limit amount of memory ZCompressorStream may
  use.
  Issue: COMPRESS-386. Thanks to Tim Allison.
o Added a way to limit amount of memory LZMACompressorStream and
  XZCompressorInputStream may use.
  Issue: COMPRESS-382. Thanks to Tim Allison.
o Add Brotli decoder based on the Google Brotli library.
  Issue: COMPRESS-392. Thanks to Philippe Mouawad.
o ZipEntry now exposes its data offset.
  Issue: COMPRESS-390. Thanks to Zbynek Vyskovsky.
o Using ZipArchiveEntry's setAlignment it is now possible to
  ensure the data offset of an entry starts at a file position
  that at word or page boundaries.
  A new extra field has been added for this purpose.
  Issue: COMPRESS-391. Thanks to Zbynek Vyskovsky.

Fixed Bugs:
o SnappyCompressorInputStream slides the window too early
  leading to ArrayIndexOutOfBoundsExceptions for some streams.
  Issue: COMPRESS-378.
o ZipArchiveEntry#isUnixSymlink now only returns true if the
  corresponding link flag is the only file-type flag set.
  Issue: COMPRESS-379. Thanks to Guillaume Boué.
o Fixed an integer overflow in CPIO's CRC calculation.
  Pull Request #17. Thanks to Daniel Collin.
o Make unit tests work on Windows paths with spaces in their names.
  Issue: COMPRESS-387.
o Internal location pointer in ZipFile could get incremented
  even if nothing had been read.
  Issue: COMPRESS-389.
o LZMACompressorOutputStream#flush would throw an exception
  rather than be the NOP it promised to be.
  Issue: COMPRESS-393.

Changes:
o The blocksize for FramedSnappyCompressorInputStream can now be
  configured as some IWA files seem to be using blocks larger
  than the default 32k.
  Issue: COMPRESS-358.
o BZip2CompressorInputstream now uses BitInputStream internally.
  Pull Request #13. Thanks to Thomas Meyer.
o Improved performance for concurrent reads from ZipFile when
  reading from a file.
  Issue: COMPRESS-388. Thanks to Zbynek Vyskovsky.

For complete information on Commons Compress, including instructions
on how to submit bug reports, patches, or suggestions for improvement,
see the Apache Commons Compress website:

http://commons.apache.org/compress/

Stefan Bodewig, on behalf of the Apache Commons community
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlkZKjsACgkQohFa4V9ri3L9fwCfQ8RHuCQ3XCKKaVcR+0QLRGvk
cHMAnjlABgyeyw3VG+qQFsx7ARJ4dX84
=tqqW
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [math] porting other code bases into ours

[Stefan Bodewig]

On 2017-05-16, Rob Tompkins wrote:

> I’m currently working on an issue in the [math] codebase, and the suggested 
> fix is to port some python code over to Java from scipy 
> (https://github.com/scipy/scipy , 
> https://scipy.org , license: BSD-3-clause). Clearly I 
> can’t copy and paste their exact code in to our codebase as it’s python, but 
> clearly I’m lifting their ideas.

> So, what is and isn’t allowed in this case? Do I simply go through the 
> exercise of porting the code over and make a reference or is there something 
> else that I should do?

I've been doing similar things in [compress] coming from C rather than
Python. My code is usually not a verbatim port but rather a Java rewrite
of the ideas of the original algorithm.

AFAIU a straight port would be fine as well as the license is
compatible. You may want to keep pointers to the original code base and
license with your code.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [LANG] Add Automatic-Module-Name MANIFEST entry

[Stefan Bodewig]

On 2017-06-07, Benedikt Ritter wrote:

> here [1] is my proposal on how to add the Automatic-Module-Name entry
> to MANIFEST. This just duplicates the maven-jar-plugin configuration
> from parent pom. I don’t want to wait much longer to release
> 3.6. After we have implemented a more general solution in parent pom,
> we can revert this fix.

I've done something similar to Compress already. As Compress has been
overriding the jar-plugin configuration already (in order to add a
main-class) there's been no other option anyway.

I'm afraid Compress is not the only component that overrides the
parent's jar config and thus will require copying the change manually
even if you happen to find a solution for parent.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [LANG] Add Automatic-Module-Name MANIFEST entry

[Stefan Bodewig]

On 2017-06-07, Jörg Schaible wrote:

> Stefan Bodewig wrote:

>> On 2017-06-07, Benedikt Ritter wrote:

>>> here [1] is my proposal on how to add the Automatic-Module-Name entry
>>> to MANIFEST. This just duplicates the maven-jar-plugin configuration
>>> from parent pom. I don’t want to wait much longer to release
>>> 3.6. After we have implemented a more general solution in parent pom,
>>> we can revert this fix.

>> I've done something similar to Compress already. As Compress has been
>> overriding the jar-plugin configuration already (in order to add a
>> main-class) there's been no other option anyway.

>> I'm afraid Compress is not the only component that overrides the
>> parent's jar config and thus will require copying the change manually
>> even if you happen to find a solution for parent.

> You don't have to overwrite the jar's parent config. Simply append/overwrite
> the existing entries:

>   
> 
>   
> org.apache.commons.compress.archivers.Lister
>   
> 
>   

> That's it ;-)

I'm pretty sure we've tried that before. I'll give it another shot,
thanks.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: OSGi Version at Package Level

[Stefan Bodewig]

On 2017-06-07, Bertrand Delacretaz wrote:

> On Wed, Jun 7, 2017 at 9:10 AM, Emmanuel Bourg  wrote:
>> ...Do I understand well that we would have to micro manage versions at the
>> package level different from the version at the component level, and
>> sometimes significantly different versions (like foo 1.2.3 and
>> org.apache.commons.foo.bar 2.3.4)?...

> That's how it's done in OSGi-based systems.

>> ... That sounds like a nightmare...

> With the right tools as mentioned in my previous message it's quite easy.

>> ...What existing problem would that solve exactly?...

> Implementing semantic versioning at the java package level as opposed
> to bundle level.

> In heavily componentized systems that's a big help, assuming the Java
> packages make sense - to paraphrase Grady Booch, one package should do
> one thing and one thing well.

> On the other hand, if you don't care about package-level versioning
> and consider your bundle as one big thing on which other things
> depend, that doesn't help much.

Compress, which is targeted by Simon's pull request, is probably one of
the few components where package level versioning might make sense. For
most of our releases the packages are independent of each other and many
of our implementation packages don't change at all with new releases.

I'm not a citizen of OSGi land and find it hard to fully understand
whether the suggested changes make things better in any way.

Right now we bump all package level versions together with the
artifact. We intend to use semantic versioning, if we've failed to do so
in the past, then this is the problem more than whether the version
applies to the package or the library as a whole.

So I've got a bunch of practical questions:

Say we started versioning packages with 1.15 and all untouched APIs stay
at 1.14. We update package versions whenever there is an API change. We
go ahead and fix a problem with an implementation detail in - say - "ar"
in 1.19 but there is no API change. Would we modify the package version?

What if we change the implementation in "util", not in "ar", what is
going to happen to the version in "ar"? What will the bundle plugin tell
us?

Would this mean we'd need to start using bugfix version numbers for
packages where the implementaion changes but the API doesn't? I.e. "ar"
in Compress 1.19 would have version 1.14 (no API changes) or 1.14.5
(five implementation changes since 1.14)?

How would we tell our users to upgrade to version 1.19 in order to use a
bug fix that has been applied if this is the version of the library as a
whole and not the version of a package?

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: OSGi Version at Package Level

[Stefan Bodewig]

On 2017-06-12, Bertrand Delacretaz wrote:

> I'll do my best to answer from my heavy OSGi user's point of view -
> but as mentioned I know little about the specifics of Compress.

Thanks, Bertrand. You don't really need to know specifics about
Compress. It contains some packages that provide the generic API for
compression and archiving and they don't change that often. Then there
is a utils package that changes more frequently. And we've got a bunch
of format specific packages, many of which don't change at all between
versions, some of them only change the implementation (fixing bugs) and
some may add new methods or classes to add format specific features.

We use the default configuration of the bundle plugin, only marking some
imports as optional. So we are exporting all our packages.

To be honest I still feel I must be missing something.

So far all packages have the same version as the library itself,
i.e. they change the minor version with each release, even if there is
no change at all.

I do understand why signaling a minor change when there really is a
major change is a problem. But this is unrelated to the OSGi issue, so
I'd like to leave this out.

If the API of a package changes with a new release we are on the safe
side with the current approach.

What is the expectation if only the implementation of a package changes?
If we need to update the version of the package as well, them I'm afraid
no tools is going to catch this.

What is the expectation for the format specific packages if the version
of the utils package they internally depend on changes? Do the format
specific packages need a version update as well? Does the bundle plugin
detect this?

What bad happens if the published version of a package changes even if
there is no change at all - this is what happens for some packages if we
keep on doing what we have done all the time.

> On Mon, Jun 12, 2017 at 4:39 PM, Stefan Bodewig  wrote:

>> ...What if we change the implementation in "util", not in "ar", what
>> is going to happen to the version in "ar"? What will the bundle
>> plugin tell us?..

> if util is exported and changes require a version number increase then
> you do that.

> And if ar has not changed its version number stays the same.

This is what I've rephrased above. "ar" depends on "util" internally,
but that's (currently?) not visible looking at the list of exported
packages.

>> ...How would we tell our users to upgrade to version 1.19 in order to use a
>> bug fix that has been applied if this is the version of the library as a
>> whole and not the version of a package?...

> If an non-exported implementation package includes changes then you
> can tell your users to upgrade that bundle to include those fixes but
> at the OSGi level those changes are invisible as the OSGi framework
> only sees exported packages.

> That's an implementation upgrade then, but if exported packages have
> not changed you can guarantee that the system will startup without
> requiring any other upgrades, that's the beauty of OSGi.

> And if the version of an exported package changes, it will cascade
> into the whole system and force you to upgrade bundles that depend on
> it (based on their importing version range), which guarantees that if
> your system starts it has all the right versions.

What I really wanted to say is if the bugfix means the exported package
version now is "1.17.1" and the bundle version is "1.19" won't we
confuse the hell out of our users if we tell them they need version
1.17.1 of the package that's contained inside of Compress 1.19 - as
opposed to telling them "upgrade to Compress 1.19"?

I'm also afraid we'll have to grep through git logs to figure out which
version of Compress a bug is reported against if the reporter talks
about the version of an exported package rather than the version of the
bundle.

Maybe I'm making a bigger problem of this than it really is.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[all] dev@ traffic

[Stefan Bodewig]

Hi all

apart from dev@ and user@ we've got three lists that have been created
for specific notifications that people may be interested in or not:
issues, commits and notifications. My MUA is configured to merge those
three with dev so I didn't really notice we are getting more messages
(and for most people more noise) to dev than I would have expected.

JIRA seems to be set up to send to issues@ for all components, and the
git and svn commit messages of components go to commits@ while svn
commit messages for the website go to notifications@.

Things are different between components for at least

* github integration (some components like Compress send mails to dev,
  others like Numbers send them to notifications)

* CI messages (at least Compress uses dev for Jenkins mails)

IMHO neither should go to dev@ and using notifications@ seems more
logical to me. I'm going to change this for Compress and would likt to
ask other components to check their settings as well so we can raise the
signal ratio for dev@ again.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [all] dev@ traffic

[Stefan Bodewig]

On 2017-06-19, Emmanuel Bourg wrote:

> Le 19/06/2017 à 12:32, Stefan Bodewig a écrit :

>> IMHO neither should go to dev@ and using notifications@ seems more
>> logical to me. I'm going to change this for Compress and would likt to
>> ask other components to check their settings as well so we can raise the
>> signal ratio for dev@ again.

> +1 for redirecting Jenkins to notifications@

> GitHub PRs and JIRA should target the same list.

the same "notifications" list or those two should go to the same list
even if its "issues"?

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [all] dev@ traffic

[Stefan Bodewig]

On 2017-06-19, Simon Spero wrote:

> (this was prompted by me interrupting a quasi-appropriate JIRA
> meta-discussion about JIRA workflow & permissions (on an issue I had opened
> because I encountered it whilst working another issue, and had fixed but
> not PRed, which Stefan then also fixed)  to further apologize to Stefan for
> not replying to an email from him (on dev)  that I'd missed because of
> coveralls) .

with all parenteses properly nested, wow ;-)

No need to apologize, really.

Thanks

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress][all]Fwd: Build failed in Jenkins: Commons-Compress PullRequest Builder #56

[Stefan Bodewig]

On 2017-06-19, Gary Gregory wrote:

> Is it just me or have there been Jenkins failures left and right for the
> last week or so?

That's the Jenkins job I've created for github pull requests and it
fails whenever there are merge conflicts, for example.

Right now it looks as if the workspace was caught in an unhealthy state,
I'll wipe it and reconfigure the job to perform clean checkouts.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Code formatting for COMMONS (was Re: [GitHub] commons-text issue #52: Test: Improved testcase coverage for StrBuilder)

[Stefan Bodewig]

On 2017-06-19, Simon Spero wrote:

> Is there a set of commons code formatting conventions? More particularly,
> is there a machine readable file of same? I use intellij, so native,
> checkstyle, or Eclipse is fine.

This probably would be something that was decided at the component level
and may be different between components. Some components may not have
any conventions at all - Compress is one such component for example:
http://commons.apache.org/proper/commons-compress/conventions.html

I don't see any conventions documented for Text either.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: OSGi Version at Package Level

[Stefan Bodewig]

On 2017-06-20, Bertrand Delacretaz wrote:

> On Thu, Jun 15, 2017 at 6:53 PM, Stefan Bodewig  wrote:
>> ...We use the default configuration of the bundle plugin, only marking some
>> imports as optional. So we are exporting all our packages...

> Ok. In the OSGi world it's only APIs and utility classes that should
> be exported, implementations should be hidden.

> But this requires separating these things (API, public utilities,
> implementation) in their own packages - if you don't do that, using
> OSGi semantic versioning might not help much and the whole thing might
> be moot.

It may be more difficult than that. Some format specific packages are
actually mixtures of API and implementation as we provide extensions
only supported for a specific format. Things like which compression
level to use for gzip or the dialect to use when tar encounters file
names longer thann 100 characters ...

> And reorganizing your packages might cause too much pain compared to
> the benefits, although it's good in the long term. So maybe something
> for a major release.

If we ever get enough momentum for this, yes :-)

> The format specific packages should be only implementation, correct?
> With only their APIs being exported.

Not in our current world, no.

>> ... What bad happens if the published version of a package changes even if
>> there is no change at all - this is what happens for some packages if we
>> keep on doing what we have done all the time

> As above - this will require some OSGi users to upgrade or recompile
> other bundles that depends on yours, although technically that
> wouldn't be needed.

Understood, thanks.

>> ...What I really wanted to say is if the bugfix means the exported package
>> version now is "1.17.1" and the bundle version is "1.19" won't we
>> confuse the hell out of our users if we tell them they need version
>> 1.17.1 of the package that's contained inside of Compress 1.19 - as
>> opposed to telling them "upgrade to Compress 1.19"?...

> If the exported package versions haven't changed (because the bugfix
> code is implementation only so not exported) , in OSGi you can use
> either the old or the new version of the bundle, without any other
> changes to your system as OSGi won't see the change. Users will have
> to be informed to upgrade to 1.19 to get the bugfix, and based on no
> exported package versions having changed they will know they don't
> need any other changes in their system.

Interesting. This means OSGi only provides a way for consumers to say
which version of an API they want, but not which implementation. This
means as a consumer you can't say "I want version 1.19 of Compress
because I know it contains an important fix". This feels more limited in
expressiveness than I had expected.

>> ...I'm also afraid we'll have to grep through git logs to figure out which
>> version of Compress a bug is reported against if the reporter talks
>> about the version of an exported package rather than the version of the
>> bundle

> They shouldn't, tracking should always happen against the version of
> the artifact that you release.

Phew, this is good. :-)

> I think the key is cleanly separating the code so that Java packages
> do not mix APIs, public utilities and bundle-private code.

> If that's done, the rest follows naturally.

> If that's not done, having independent version numbers for each
> package might cause more pain that benefits.

Thank you very much.

> HTH, and nice "talking" to you Stefan after all this time!

Indeed!

Cheers

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Some ideas for improving Travis build performance improvements:

[Stefan Bodewig]

Hi Simon et al

I'm not sure what we are using Travis CI for in the first place. Maybe
we can minimize build time by reducing the build to the utter minimum
required to achieve that - whatever that is?

Honestly I don't see anything we couldn't do with our Jenkins
installation as well. Maybe having Coveralls for pull requests, but
that's it (and Coveralls itself seems to produce random ups and downs
occasionally).

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [DISCUSS] Do we need a vote for moving components to git?

[Stefan Bodewig]

On 2017-06-28, Benedikt Ritter wrote:

> do we still need a vote for moving components to git? We have done
> this so often that I think we could change the process to a simply
> mail announcing intents to move a component.

Hmm, what if one of the active developers of the component then says
she/he would prefer to stick with svn? We've had such cases in the past.

If you then reconsider or at least start to discuss I don't really see
the difference between a lazy vote and the announcement.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [DISCUSS] Do we need a vote for moving components to git?

[Stefan Bodewig]

On 2017-06-28, Amey Jadiye wrote:

> I think just announcing with intention is enough as git is always better
> than svn and at some point everyone will wish to move on git.

There certainly are people who disagree with your (I'm not one of them) ;-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: Fwd: [jira] [Created] (DISCOVERY-23)

[Stefan Bodewig]

On 2017-07-05, Amey Jadiye wrote:

> Looks very suspicious to me, like created with script / bot ?

Maybe you shouldn't cite it completely, now the spam is part of all
dev@commons archives as well :-)

I've deleted the two issues and asked INFRA to deal with it as the same
user has been hitting other projects as well.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] FW: Tika content detection and crawled "remote" content

[Stefan Bodewig]

This looks great, well done Tika!

Thank you for sharing, Tim

Cheers

  Stefan

On 2017-07-05, Allison, Timothy B. wrote:

> Fellow file-philes on [compress],

> Sebastian Nagel has added file type id via Apache Tika to Common Crawl.  
> While Tika is not 100% accurate, this means that we have far better clarity 
> on mime type than relying on the http header+file suffix.  So, for testing 
> purposes, you (or we over on Tika) can much more easily gather a small test 
> corpus of files by mime type.

> Many, many thanks to Sebastian and Common Crawl!

>   Cheers,

>   Tim

> -Original Message-
> From: Sebastian Nagel [mailto:wastl.na...@googlemail.com]
> Sent: Tuesday, July 4, 2017 6:18 AM
> To: u...@tika.apache.org
> Subject: Tika content detection and crawled "remote" content

> Hi,

> recently I've plugged in Tika's content detection into Common Crawl's crawler 
> (modified Nutch) with the target to get clean and correct MIME type - the 
> HTTP Content-Type may contain garbage and isn't always correct [1].

> For the June 2017 crawl I've prepared a comparison of content types sent by 
> the server in the HTTP header and as detected by Tika 1.15 [2].  It shows 
> that content types by Tika are definitely clean
> (1,400 different content types vs. more than 6,000 content type "strings" 
> from HTTP headers).

> A look on the "confusions" where Content-Type and Tika differ, shows a mixed 
> picture: some pairs are plausible, e.g., if Tika changes the type to a more 
> precise subtype or detects the MIME at all:

> Tika-1.15HTTP-Content-Type
> 1001968023  application/xhtml+xmltext/html
>2298146  application/rss+xml  text/xml
> 617435  application/rss+xml  application/xml
> 613525  text/htmlunk
> 361525  application/xhtml+xmlunk
> 297707  application/rdf+xml  application/xml


> However, there are a few dubious decisions, esp. the group of web server-side 
> scripting languages (ASP, JSP, PHP, ColdFusion, etc.):

>  Tika-1.15 HTTP-Content-Type
> 2047739  text/x-phptext/html
>  681629  text/asp  text/html
>  193095  text/x-coldfusion text/html
>  172318  text/aspdotnettext/html
>  139033  text/x-jsptext/html
>   38415  text/x-cgitext/html
>   32092  text/x-phptext/xml
>   18021  text/x-perl   text/html

> Of course, due to misconfigurations some servers may deliver the script files 
> unmodified but in general I wouldn't expect that this happens for millions of 
> pages.  I've checked some of the affected URLs:

> - HTML fragment (no declaration of  or  opening tag)

> https://www.projectmanagement.com/profile/profile_contributions.cfm?profileID=46773580&popup=&c_b=0&c_mb=0&c_q=0&c_a=2&c_r=1&c_bc=1&c_wc=0&c_we=0&c_ar=0&c_ack=0&c_v=0&c_d=0&c_ra=2&c_p=0
> http://www.privi.com/product-details.asp?cno=C10910011
> http://mental-ray.de/Root_alt/Default.asp
> http://ekyrs.org/support/index.php?action=profile
> http://cwmorse.eu5.org/lineal/mostrar.php?contador=200

> - (overlong) comment block at start of HTML which "masks" the HTML declaration
> http://www.mannheim-virtuell.de/index.php?branchenID=2&rubrikID=24

> http://www.exoduschurch.org/bbs/view.php?id=sunday_school&page=1&sn1=&divpage=1&sn=off&ss=on&sc=on&select_arrange=headnum&desc=asc&no=6
> 
> https://www.preventiongenetics.com/About/Resources/disease/MarfansSyndrome.php
> https://de.e-stories.org/categories.php?&lan=nl&art=p

> - HTML with some scripting fragments ("") present:
> http://www.eco-ani-yao.org/shien/

> - others are clearly HTML (looks more like a bug, at least, there is no 
> simple explanation)
> http://www.proedinc.com/customer/content.aspx?redid=9
> 
> http://cball.dyndns.org/wbb2/board.php?boardid=8&sid=bf3b7971faa23413fa1164be0c068f79
> http://eusoma.org/Engx/Info/ContactUs.aspx?cont=contact
> http://cball.dyndns.org/wbb2/map.php?sid=bf3b7971faa23413fa1164be0c068f79


> Obviously certain file suffixes (.php, .aspx) should get less weight compared 
> to Content-Type sent from the responding server.
> Now my question: where's the best place to fix this: in the crawler [3] or in 
> Tika?

> If anyone is interested in using the detected MIME types or anything else 
> from Common Crawl - I'm happy to help!  The URL index [4] contains now a new 
> field "mime-detected" which makes it easy to search or grep for confusion 
> pairs.


> Thanks and best,
> Sebastian


> [1] https://github.com/commoncrawl/nutch/issues/3
> [2] 
> s3://commoncrawl-dev/tika-content-type-detection/content-type-diff-tika-1.15-cc-main-2017-26.txt.xz

> https://commoncrawl-dev.s3.amazonaws.com/tika-content-type-detection/content-type-diff-tika-1.15-cc-main-2017-26.txt.xz
> [3] 
> https://github.com/apache/nutch/blob/master/src/java/org/apache/nutch/util/MimeUtil.java#L152
> [4] http://commoncrawl.org/2015/04/announcing-the-common-crawl-ind

Re: [compress] HasCharset is a bad name

[Stefan Bodewig]

On 2017-07-05, Gary Gregory wrote:

> The new interface name HasCharset is pretty bad IMO.

fair enough.

> CharsetProvider is the obvious (to me) better name even though there
> already is a class called java.nio.charset.spi.CharsetProvider.

I don't think so, this really is a marker interface for classes also
implementing ZipEncoding (Simon didn't want to extend the existing
interface).

Hmm, maybe we could make it extend ZipEncoding and call it something
like ZipEncodingWithCharset?

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [all] Removal of old pgp key from https://www.apache.org/dist/commons/KEYS

[Stefan Bodewig]

On 2017-07-18, Amey Jadiye wrote:

> I observed we have lot of keys in https://www.apache.org/dist/commons/KEYS,
> even keys of developers who might have resigned from commons, can we just
> review and  remove keys of developers who resigned or no more active ?

We shouldn't remove any key that has been used to sign a release in the
past. No matter how long in the past :-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ANNOUNCE] Apache Commons Collections moved to git

[Stefan Bodewig]

On 2017-07-19, Rob Tompkins wrote:

> The Apache Commons Collections component has been moved to git.

Many thanks for doing that, Rob!

Are you following some kind of playbook with the git migration? If so,
we should add one point:

* edit the svn:externals property of
  https://svn.apache.org/repos/asf/commons/trunks-proper and remove the
  entry for the component just migrated.

Cheers

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ANNOUNCE] Apache Commons Collections moved to git

[Stefan Bodewig]

On 2017-07-19, Rob Tompkins wrote:

>> On Jul 19, 2017, at 8:13 AM, Stefan Bodewig  wrote:

>> On 2017-07-19, Rob Tompkins wrote:

>>> The Apache Commons Collections component has been moved to git.

>> Many thanks for doing that, Rob!

>> Are you following some kind of playbook with the git migration? If so,
>> we should add one point:

>> * edit the svn:externals property of
>>  https://svn.apache.org/repos/asf/commons/trunks-proper and remove the
>>  entry for the component just migrated.

> Good point, and no playbook here. I suppose we should either:
> 1. create such a playbook, or 2. do the remainder of the repos so that
> such a playbook is no longer needed. :-)

True.

I'm all for option 1 and may find some time over the weekend to start a
page inside the wiki, based on what I recall doing back when I migrated
Compress. Given that a few things have changed since then (the
self-service stuff didn't exist) and you've done it a few times now, I
wouldn't feel bad if you beat me to it and started the page ;-)

Cheers

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [ANNOUNCE] Apache Commons Collections moved to git

[Stefan Bodewig]

On 2017-07-21, Stefan Bodewig wrote:

> On 2017-07-19, Rob Tompkins wrote:

>>> On Jul 19, 2017, at 8:13 AM, Stefan Bodewig  wrote:

>>> On 2017-07-19, Rob Tompkins wrote:

>>>> The Apache Commons Collections component has been moved to git.

>>> Many thanks for doing that, Rob!

>>> Are you following some kind of playbook with the git migration? If so,
>>> we should add one point:

>>> * edit the svn:externals property of
>>>  https://svn.apache.org/repos/asf/commons/trunks-proper and remove the
>>>  entry for the component just migrated.

>> Good point, and no playbook here. I suppose we should either:
>> 1. create such a playbook, or 2. do the remainder of the repos so that
>> such a playbook is no longer needed. :-)

> True.

> I'm all for option 1 and may find some time over the weekend to start a
> page inside the wiki, based on what I recall doing back when I migrated
> Compress.

Not really weekend anymore. :-)

When I wanted to begin a new page I found this one

https://wiki.apache.org/commons/MovingToGit

Not sure whether everything on it is still accurate, but at least it
contains all the data points that I remembered myself.

Cheers

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

git migration guide (was Re: [ANNOUNCE] Apache Commons Collections moved to git)

[Stefan Bodewig]

On 2017-07-25, Rob Tompkins wrote:

>> When I wanted to begin a new page I found this one

>> https://wiki.apache.org/commons/MovingToGit

> If I recall correctly Benedikt was the last to update this page when
> he migrated [cli] over (even as late as last month on 6/28). So I
> think that it’s fairly up to date, aside from the bit that you did
> with “trunks-proper” in svn.

That's been present already as well.

> I unfortunately don’t have permissions to edit out pages on the wiki
> (I think that I need some one with the karma to add me).

I should be able to do so. What is your account name?

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [wiki] access

[Stefan Bodewig]

On 2017-07-25, Rob Tompkins wrote:

> Would someone with sufficient wiki karma add “RobTompkins” to the list
> of folks that can edit the commons wiki?

Done

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[VOTE] Release Commons Email 1.5 Based on RC1

[Stefan Bodewig]

We have fixed quite a few bugs and added some significant enhancements
since Email 1.4 was released, so I would like to release Email 1.5.

Email 1.5 RC1 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/email/
(svn revision 20667)

The tag is here:
http://svn.apache.org/repos/asf/commons/proper/email/tags/EMAIL_1_5_RC1/
(svn revision 1803366)
N.B. the SVN revision is required because SVN tags are not immutable.

Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1255/org/apache/commons/commons-email/1.5/

These are the Maven artifacts and their SHA1 hashes

e8e677c6362eba14ff3c476ba63ccb83132dbd52  commons-email-1.5.jar
6ccc8b44cb666b849b71ecc6fa32549a6461c099  commons-email-1.5-javadoc.jar
09d31911480b5148275d8a26c60e355bbcbe9ae3  commons-email-1.5.pom
dbe951d1b89eb9472b4b2c8f618b8bc9783b6623  commons-email-1.5-sources.jar
15afde52264e316438802b5bd05d34bc424bf659  commons-email-1.5-tests.jar
e157492dfd776839387a6ce4af0d191e0a92a534  commons-email-1.5-test-sources.jar

I have tested this with JDK 8 using Maven 3.0.5.

As usual when I cut a release the site is not the one I'll publish
once the vote has succeeded. I'll recreate the site once the release
date is known.

I've already copied the javadocs for 1.4 so
http://commons.apache.org/proper/commons-email/javadocs/api-1.4/index.html
already is available. The javadoc links and download page certainly
don't work in the staging site.

Details of changes since 1.4 are in the release notes:
https://dist.apache.org/repos/dist/dev/commons/email/RELEASE-NOTES.txt
https://stefan.samaflost.de/staging/commons-email-1.5/changes-report.html

Site:
https://stefan.samaflost.de/staging/commons-email-1.5/

Clirr Report (compared to 1.4):
https://stefan.samaflost.de/staging/commons-email-1.5/clirr-report.html

RAT Report:
https://stefan.samaflost.de/staging/commons-email-1.5/rat-report.html

KEYS:
  https://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.
This vote will close no sooner that 72 hours from now,
i.e. sometime after 14:30 UTC 1-Aug 2017

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Thanks!

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [OT] Job offer

[Stefan Bodewig]

On 2017-07-29, Oliver Heger wrote:

> my apologies in advance if this is considered spam, but as some of my
> colleagues know that I am involved in an open source project, they asked
> my to post the following Job offer here.

Oliver, are you aware of the jobs@apache mailing list? You may find a
broader audience there.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Email 1.5 Based on RC1

[Stefan Bodewig]

On 2017-07-30, Gary Gregory wrote:

> Branding: The RELEASE-NOTES.txt file should start with "Apache Commons
> Email Package" instead of "Commons Email Package".

I was under the impression it had been generated by the commons-build
plugin. Anyway, will fix it when I publish the release (no reason to
vote on that).

> RAT check fails with:

> Unapproved licenses:

>   src/test/resources/eml/attachment-only.eml
>   src/test/resources/eml/html-attachment-content-disposition.eml
>   src/test/resources/eml/html-attachment-encoded-filename.eml
>   src/test/resources/eml/html-attachment.eml
>   src/test/resources/eml/multipart-report.eml
>   src/test/resources/eml/multipart-text-attachment-only.eml
>   src/test/resources/eml/multipart-text-attachment.eml
>   src/test/resources/eml/outofmemory-no-header-seperation.eml
>   src/test/resources/eml/simple-reply.eml
>   src/test/resources/eml/simple.eml

The pom contains


org.apache.rat
apache-rat-plugin


src/test/resources/eml/*.eml




and RAT is prefectly happy inside the site build. How do you execute the
rat check? Comparing the POM with the one of Compress I see the plugin
is configured inside the  section for Email only, while it is
inside  for Compress. This may explain the difference when
running rat checks directly.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [email] failing with mvn clean cobertura:cobertura coveralls:report

[Stefan Bodewig]

On 2017-07-30, Amey Jadiye wrote:

> while fixing rat issue with commons email I also found that the travis is
> failing with the below configurations, thought build is working fine.
> seems like we dont have plugin configured in pom.xml


> after_success:
>   - mvn clean cobertura:cobertura coveralls:report

coveralls seems to be configured inside the parent POM and has been
added there with parent 42. Email still uses parent version 40.

I'll update the parent in trunk after the release has been published.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Commons Email 1.5 Based on RC1

[Stefan Bodewig]

On 2017-07-30, Amey Jadiye wrote:

> I have fixed this, and yes reason was though those .eml files was added in
> exclusion but in reports and not in build.
> I have raised PR and tested it on my local.

> https://github.com/apache/commons-email/pull/2

LGTM, but please allow us to get the release out of the door before
changung trunk :-)

Thanks

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[RESULT] Release Commons Email 1.5 Based on RC1

[Stefan Bodewig]

Hi all

with the binding +1s by Oliver Heger, Gary Gregory and myself (cast
implicitly) as well as the +1 by Amey Jadiye this vote has passed.

I'll publish the artifacts and give the mirrors a little time to catch
up before announcing the release.

Many thanks to those who reviewed the release

 Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[ANN] Apache Commons Email 1.5 Released

[Stefan Bodewig]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

The Apache Commons Team is pleased to announce the release of Apache
Commons Email 1.5.

This is a major and security bugfix release which adds some new
features and fixes several bugs present in the 1.4 release. All
current users are encouraged to upgrade.

For the security bugfix see
https://commons.apache.org/proper/commons-email/security-reports.html#Fixed_in_Apache_Commons_Email_1.5

The Apache Commons Email Library aims to provide a API for sending
email. It builds on the JavaMail API with the aim of presenting a
simplified API which is easy for developers who are not mail experts
to use. It is a compact component with a small number of classes.

Source and binary distributions are available for download from the
Apache Commons download site:

http://commons.apache.org/proper/commons-email/download_email.cgi

When downloading, please verify signatures using the KEYS file available
at the above location when downloading the release.

Changes in this version include:

== Compatibility ==

* Java 6 or later is required.

* JavaMail dependency has been upgraded to version 1.5.6,
  as a consequence, the maven dependency has changed to:

  
com.sun.mail
javax.mail
1.5.6
  

== New Features ==

* Add Support for International Domain Names. This change requires JDK 1.6+.
Issue: EMAIL-160

* Add Email#getHeader(String) and Email#getHeaders() methods.
Issue: EMAIL-154. Thanks to Ken Geis, Balachandran Sivakumar

== Updates ==

* Update Oracle JavaMail dependency from 1.5.2 to 1.5.6.
  Issue: EMAIL-165.

* Remove "javax.activation" dependency since it is included in JDK 1.6
  Issue: EMAIL-161.

== Fixed Bugs ==

* DataSourceClassPathResolver doesn't close InputStream when resolving resources
Issue: EMAIL-167. Thanks to Lucian Burja.

* CVE-2017-9801 - stripped all line-breaks from subjects in order to
  prevent SMTP header injection.

For complete information on Commons Email, including instructions
on how to submit bug reports, patches, or suggestions for improvement,
see the Apache Commons Email website:

http://commons.apache.org/email/

Stefan Bodewig, on behalf of the Apache Commons community
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlmAyFMACgkQohFa4V9ri3J+kACcDuO7+0echoLLZPDglWkot2FD
FlIAoJ5Lu12NRpmnnl6tVAP3qS8MK513
=t6js
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

CVE-2017-9801: Apache Commons Email SMTP header injection vulnerabilty

[Stefan Bodewig]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

CVE-2017-9801: Apache Commons Email SMTP header injection vulnerabilty

Severity: low

Vendor:
The Apache Software Foundation

Versions Affected:
Apache Commons Email 1.0 to 1.4.

Description:
When a call-site passes a subject for an email that contains
line-breaks, the caller can add arbitrary SMTP headers.

Mitigation:
Users should upgrade to Commons Email 1.5.
You can mitigate this vulnerability for older versions of Commons
Email by stripping line-breaks from the subject before passing it to
the setSubject(String) method.

Credit:
This issue was discovered by Adam Williams.

References:
http://commons.apache.org/proper/commons-email/security-reports.html
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlmAyP8ACgkQohFa4V9ri3K7XQCgj69yH9nkBGRVJBG9+0DS1jc8
GJUAnRZrLznaNRzokj08JGBMy5wwHNTt
=oSDx
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [compress] Tar stream signature detection regression as of 1.11

[Stefan Bodewig]

On 2017-08-11, Philipp Grasboeck wrote:

> I think I may have found a regression in
> ArchiveFormatFactory.createArchiveInputStream.
> Please run the JUnit testcase at the end of this mail.
> It demonstrates the archive detection working with 1.10, but failing from
> 1.11 through 1.14

As already pointed out by James Ring, please see
https://issues.apache.org/jira/browse/COMPRESS-331

The code in 1.10 would accept streams as tar archives that were not
valid. We made the check as strict as the one of GNU tar and only accept
streams with a correct header checksum as tar archives.

> The file "COMPRESS-117.tar" is taken from commons-compress-1.9-src.zip.
> If it's not a bug, please let me know.

The file COMPRESS-117.tar gets rejected by GNU tar as well. Please see
COMPRESS-331 for a bit of history.

If you've got a tar archive which gets accepted by GNU tar but rejected
by Commons Compress, then this is a bug - and I'd ask you to open a new
issue and attach the archive.

Thanks

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [COLLECTIONS] Time for 4.2

[Stefan Bodewig]

On 2017-09-20, Gilles wrote:

> On Wed, 20 Sep 2017 07:33:55 -0600, Gary Gregory wrote:

>> How about dealing with the Automatic-Module-Name MANIFEST header in
>> the parent POM and releasing that POM first?

> Is there a solution to this issue:
>   https://issues.apache.org/jira/browse/RNG-31

I'm afraid Gary and you are talking about completely different things
(and either of you as well as myself should have changed the subject
line ;-)

Unfortunately I cannot answer the question about signatures for modular
builds, I'm not really a maven person. What I can say is that I never
invoke any "assembly" goal when I create a release, the signatures are
created as a byproduct of running "deploy".

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [All][RNG] Generate signatures and checksums (Was: [COLLECTIONS] Time for 4.2)

[Stefan Bodewig]

On 2017-09-20, Gilles wrote:

> On Wed, 20 Sep 2017 17:49:38 +0200, Stefan Bodewig wrote:
>> On 2017-09-20, Gilles wrote:

>>> On Wed, 20 Sep 2017 07:33:55 -0600, Gary Gregory wrote:

>>>> How about dealing with the Automatic-Module-Name MANIFEST header in
>>>> the parent POM and releasing that POM first?

>>> Is there a solution to this issue:
>>>   https://issues.apache.org/jira/browse/RNG-31

>> I'm afraid Gary and you are talking about completely different things
>> (and either of you as well as myself should have changed the subject
>> line ;-)

> I did some 20 minutes ago. :-)


>> Unfortunately I cannot answer the question about signatures for
>> modular
>> builds, I'm not really a maven person. What I can say is that I never
>> invoke any "assembly" goal when I create a release, the signatures
>> are
>> created as a byproduct of running "deploy".

> "deploy" will upload to the nexus server, and there is no
> problem (signatures are generated automagically).

mvn deploy -Ptest-deploy -Prelease

creates all signatures but doesn't actually perform the upload. This is
what I use.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [All][RNG] Generate signatures and checksums

[Stefan Bodewig]

On 2017-09-21, Gilles wrote:

> On Thu, 21 Sep 2017 07:47:07 +0200, Stefan Bodewig wrote:
>> On 2017-09-20, Gilles wrote:

>>> [...]

>>>> Unfortunately I cannot answer the question about signatures for
>>>> modular
>>>> builds, I'm not really a maven person. What I can say is that I
>>>> never
>>>> invoke any "assembly" goal when I create a release, the signatures
>>>> are
>>>> created as a byproduct of running "deploy".

>>> "deploy" will upload to the nexus server, and there is no
>>> problem (signatures are generated automagically).

>> mvn deploy -Ptest-deploy -Prelease

>> creates all signatures but doesn't actually perform the upload. This
>> is
>> what I use.

> What this command did not do is create the checksum and
> signature for the _full_ distribution (the only _official_
> one IIUC) made available here:
>   http://commons.apache.org/proper/commons-rng/download_rng.cgi

It does for me when using the command on compress, I've just re-checked
(which caused the delay, sorry). And it does for rng as well, for the
tar.gzs at the module level when run from the root dir of rng.

You seem to have chosen a different way with the dist-archive
directory. This one is not part of the reactor in master and if I try to
run maven from inside the dist-archive folder it fails because mvn is
looking for a checkstyle.xml that isn't there. I'm afraid I'm out of my
depth here.

If you manage to create the dist artifacts that you want with your
custom mvn project then you could certainly always fallback to command
line tools in order to create the hashes and signatures.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [LOGGING] Release with Java 9 Module support

[Stefan Bodewig]

On 2017-10-01, Benedikt Ritter wrote:

> So what options are we left with?
> - Do nothing -> logging can’t be used as a Java 9 module
> - Drop api and adapters -> users have to change their dependencies and get 
> more code than they want

- only add the automatic module name to commons-logging and release api
and adapter as they are.

I think that's been suggested before. That way people who want to
consume logging as a Java9 module can do so (and maybe need to adapt
their dependency) while all others can keep doing what they've done
before.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[compress] Cutting 1.15

[Stefan Bodewig]

Hi all

there's not been the single big feature or bugfix since 1.14, but we've
accumulated a few changes. Also the last release hasn't included the
Automatic-Module name while master does.

I plan to cut a new release, likely by the next weekend. If you feel
there is an important issue that should be resolved first, please yell.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[VOTE] Release Compress 1.15 based on RC1

[Stefan Bodewig]

Hi all

this is mostly a bugfix release but it was about time for a new release.

Compress 1.15 RC1 is available for review here:
https://dist.apache.org/repos/dist/dev/commons/compress/ (svn revision 
22476)

The tag is here:

https://git-wip-us.apache.org/repos/asf?p=commons-compress.git;a=tag;h=ae82f93454d4dcd57ac1ed7d53114bda2f82c8f3
on commit

https://git-wip-us.apache.org/repos/asf?p=commons-compress.git;a=commit;h=01b06d5ef5c5ac3bd651bedcfec7433231cea371

Maven artifacts are here:

https://repository.apache.org/content/repositories/orgapachecommons-1277/org/apache/commons/commons-compress/1.15/

These are the Maven artifacts and their sha1 hashes

b686cd04abaef1ea7bc5e143c080563668eec17e  commons-compress-1.15.jar
3f5c9429da7cc2ca4b73a415928fac5f9949c798  commons-compress-1.15.jar.asc
9a165c3cc83158820bfa0691dcd15379fa3c8ab9  commons-compress-1.15-javadoc.jar
05b6759fa95e7981c7ca0982c37880bd784e6642  commons-compress-1.15-javadoc.jar.asc
670c082e6fc3a99950ee385edc4d9352fb0a99d7  commons-compress-1.15.pom
d68decca16fa849cf432fd8e29510f1a21100518  commons-compress-1.15.pom.asc
a82a61e0bbce4e06281c675af7422553bf20ca8f  commons-compress-1.15-sources.jar
73da7e4ccac26460b301edd7ece1ac3546e2842e  commons-compress-1.15-sources.jar.asc
0e23a1617451c69488e57141c1ca99929d25ed5e  commons-compress-1.15-tests.jar
8d7535a2d7efa4c870f93946289669efc74a7f78  commons-compress-1.15-tests.jar.asc
9149c762ebf2248c0f20b62b59d827535f6604a4  commons-compress-1.15-test-sources.jar
e545455b5fd7ec12b8ddfa8cec92af2df056aa9a  
commons-compress-1.15-test-sources.jar.asc

I have tested this with JDK 8 using Maven 3.3.9.

Details of changes since 1.14 are in the release notes:
https://dist.apache.org/repos/dist/dev/commons/compress/RELEASE-NOTES.txt

https://stefan.samaflost.de/staging/commons-compress-1.15/changes-report.html

Site:
https://stefan.samaflost.de/staging/commons-compress-1.15/
  (as usual I'll re-create the site once the release date is known,
download links and the links for thr 1.15 javadocs don't work)

japicmp Report (compared to 1.14):
https://stefan.samaflost.de/staging/commons-compress-1.15/japicmp.html

RAT Report:
https://stefan.samaflost.de/staging/commons-compress-1.15/rat-report.html

KEYS:
  https://www.apache.org/dist/commons/KEYS

Please review the release candidate and vote.
This vote will close no sooner that 72 hours from now,
i.e. sometime after 14:00 UTC 17-October 2017

  [ ] +1 Release these artifacts
  [ ] +0 OK, but...
  [ ] -0 OK, but really should fix...
  [ ] -1 I oppose this release because...

Thanks!

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Compress 1.15 based on RC1

[Stefan Bodewig]

On 2017-10-15, Bruno P. Kinoshita wrote:

> I may be doing something wrong, but the site generation with `mvn
> clean site` appears to be failing

[snip]

> With:

> (...)
> [INFO] Loading execution data file 
> /home/kinow/Development/java/apache/commons-compress/target/jacoco.exec
> [INFO] Generating "japicmp" report  --- 
> japicmp-maven-plugin:0.9.3:cmp-report
> [debug] No packaging support defined, no filtering
> [debug] Searching for versions in versionRange: (,1.15)
> [debug] Parameter  not configured, i.e. no version 
> filtered.
> [warn] No new version specified and file 
> '/home/kinow/Development/java/apache/commons-compress/target/classes' of 
> artifact could not be opened as jar archive: 
> /home/kinow/Development/java/apache/commons-compress/target/classes (Is a 
> directory)

https://github.com/apache/commons-compress/blob/master/BUILDING.md#building-the-site

:-)

japicmp only works properly if it can find a jar, so you need to run the
package goal together with the site goal. So the magic incantation is

mvn package site -Pjacoco

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [VOTE] Release Compress 1.15 based on RC1

[Stefan Bodewig]

On 2017-10-15, Bruno P. Kinoshita wrote:

> Oh, don't know how I didn't see the building file. Will make a mental
> note to look for it the next time :-) thank you.

No, thank you.

I've made a mental note to include site building instructions with the
vote thread next time :-)

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[RESULT] Release Compress 1.15 based on RC1

[Stefan Bodewig]

Hi all

with +1s by Bruno P. Kinoshita, Gary Gregory, Oliver Heger and my own
implicit one the vote has passed.

I'll publish the artifacts, give the mirrors some time to catch up and
announce the release later today.

Thanks to all wh looked into the RC.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

[ANN] Apache Commons Compress 1.15 Released

[Stefan Bodewig]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

The Apache Commons Team is pleased to announce the release of Apache
Commons Compress 1.15.

The Apache Commons Compress Library defines an API for working with
compression and archive formats.  These include: bzip2, gzip, pack200,
lzma, xz, Snappy, traditional Unix Compress, DEFLATE, LZ4, Brotli and
ar, cpio, jar, tar, zip, dump, 7z, arj.

Compress 1.15 contains big fixes and small improvements for the tar and
zip formats. In addition the jar's manifest file now contains an
Automatic-Module-Name entry that ensures the name will be
org.apache.commons.compress when the jar is used as an automatic module
in Java9.

TarArchiveOutputStream now ensures record size is 512 and block size is
a multiple of 512 as any other value would create invalid tar
archives. This may break compatibility for code that deliberately
wanted to create such files.

Source and binary distributions are available for download from the
Apache Commons download site:

http://commons.apache.org/proper/commons-compress/download_compress.cgi

When downloading, please verify signatures using the KEYS file available
at the above location when downloading the release.

Changes in this version include:

New features:

o Added magic MANIFEST entry Automatic-Module-Name so the module
  name will be org.apache.commons.compress when the jar is used
  as an automatic module in Java9.
  Issue: COMPRESS-397.

o Added a new utility class FixedLengthBlockOutputStream that
  can be used to ensure writing always happens in blocks of a
  given size.
  Issue: COMPRESS-405. Thanks to Simon Spero.

o It is now possible to specify/read custom PAX headers when
  writing/reading tar archives.
  Issue: COMPRESS-400. Thanks to Simon Spero.

Fixed Bugs:

o Make sure "version needed to extract" in local file header and
  central directory of a ZIP archive agree with each other.
  Also ensure the version is set to 2.0 if DEFLATE is used.
  Issue: COMPRESS-394.

o Don't use a data descriptor in ZIP archives when copying a raw
  entry that already knows its size and CRC information.
  Issue: COMPRESS-395.

o Travis build redundantly repeats compilation and tests redundantly
  GitHub Pull Request #43. Thanks to Simon Spero.
  Issue: COMPRESS-413

o The MANIFEST of 1.14 lacks an OSGi Import-Package for XZ for
  Java.
  Issue: COMPRESS-396.

o BUILDING.md now passes the RAT check.
  Issue: COMPRESS-406. Thanks to Simon Spero.

o Made sure ChecksumCalculatingInputStream receives valid
  checksum and input stream instances via the constructor.
  Issue: COMPRESS-412. Thanks to Michael Hausegger.

o TarArchiveOutputStream now verifies the block and record sizes
  specified at construction time are compatible with the tar
  specification. In particular 512 is the only record size
  accepted and the block size must be a multiple of 512.
  Issue: COMPRESS-407. Thanks to Simon Spero.

o Fixed class names of CpioArchiveEntry and
  CpioArchiveInputStream in various Javadocs.
  Issue: COMPRESS-415.

o The code of the extended timestamp zip extra field incorrectly
  assumed the time was stored as unsigned 32-bit int and thus
  created incorrect results for years after 2037.
  Issue: COMPRESS-416. Thanks to Simon Spero.

o Removed ZipEncoding code that became obsolete when we started
  to require Java 5 as baseline long ago.
  Issue: COMPRESS-410. Thanks to Simon Spero.

o The tar package will no longer try to parse the major and
  minor device numbers unless the entry represents a character
  or block special file.
  Issue: COMPRESS-417.

o When reading tar headers with name fields containing embedded
  NULs, the name will now be terminated at the first NUL byte.
  Issue: COMPRESS-421. Thanks to Roel Spilker.

o Simplified TarArchiveOutputStream by replacing the internal
  buffering with new class FixedLengthBlockOutputStream.
  Issue: COMPRESS-409.

For complete information on Commons Compress, including instructions
on how to submit bug reports, patches, or suggestions for improvement,
see the Apache Commons Compress website:

http://commons.apache.org/compress/

Stefan Bodewig, on behalf of the Apache Commons community
-BEGIN PGP SIGNATURE-
Version: GnuPG v1

iEYEARECAAYFAlnmRAkACgkQohFa4V9ri3L2cwCeLqFLHNuxK2YUuqzbJjXl8mZR
JF8AoNO7DXUdIbL3nMs74+kkuVb46T4C
=iZYS
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [1/2] commons-compress git commit: COMPRESS-423 - Add ZStandard decompression support using Zstd-JNI

[Stefan Bodewig]

On 2017-10-17, Gary Gregory wrote:

> Should it be "Z_STANDARD" instead of "ZSTANDARD" since "standard" is a word?

Zstandard seems to be the official captitalization: http://www.zstd.net/

But I should check whether it's been used consistently over the last few
commits, I doubt it.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [GUMP@vmgump]: Project commons-configuration (in module apache-commons) failed

[Stefan Bodewig]

On 2011-12-26, Konstantin Kolinko wrote:

> 2011/12/26 Oliver Heger :
>> Which version of JUnit is used by GUMP? According to the Javadocs for
>> version 4.10, there should be a method TemporaryFolder.newFile() with an
>> empty argument list.

> http://vmgump.apache.org/gump/public/apache-commons/commons-configuration/details.html
> says "junit-26122011.jar"
> and that that jar comes from junit project at Gump,
> http://vmgump.apache.org/gump/public/junit/junit/index.html

In theory it is the tip of JUnit's master branch.

> The git update log for junit project
> http://vmgump.apache.org/gump/public/junit/gump_work/update_junit.html
> says at the end
> [[[
> Pull is not possible because you have unmerged files.
> Please, fix them up in the work tree, and then use 'git add/rm '
> as appropriate to mark resolution, or use 'git commit -a'.
> ]]]

> Is that of any concern?

Probably.  I'm not sure how long this has been the case.  We may be
trailing behing JUnit's tip.

For now I have purged the working copy of JUnit and the next run will
check out a fresh one - which I hope will resolve the issue for
configuration.  Later (I intend to have a slow week) I'll try to figure
out why we've gotten into this situation.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: svn commit: r1294460 - in /commons/proper/compress/trunk/src: changes/ main/java/org/apache/commons/compress/archivers/zip/ test/java/org/apache/commons/compress/archivers/zip/ test/resources/

[Stefan Bodewig]

On 2012-02-28, sebb wrote:

> On 28 February 2012 05:00,   wrote:
>>     protected void setName(String name) {
>>+         if (name != null && getPlatform() == PLATFORM_FAT
>>+             && name.indexOf("/") == -1) {
>>+             name = name.replace('\\', '/');
>>+         }
>>         this.name = name;
>>     }

> The original problem is with unicode extra fields only.

The InfoZIP workaround replicated here is not conditional, it seems
there are other broken ZIPs that use backslashes in plain names as well.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [lang] test failures

[Stefan Bodewig]

On 2012-03-05, sebb wrote:

> The tests also fail in Gump.

vmgump.ao, gump.zones.ao and adam.ao run JDK 6 (vmgump uses OpenJDK, the
FreeBSD zone the FreeBSD port and adam the MacOS X version by Apple).
So it likely is not a Java7 issue.

Gump uses mvn2 to run the tests if that causes any difference.  Also I
note the number of failing tests is different for each machine.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [GUMP@vmgump]: Project commons-graph (in module commons-sandbox) failed

[Stefan Bodewig]

This is Gump running on commons-graph for the first time it has
re-entered the Sandbox.

I've set it up to run mvn2's install goal on trunk of commons-graph.
vmgump runs openjdk6.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org