[GitHub] [commons-compress] PeterAlfreadLee commented on a change in pull request #91: Make bz2 decompressor robust to selector overflow

[GitBox]

PeterAlfreadLee commented on a change in pull request #91: Make bz2 
decompressor robust to selector overflow
URL: https://github.com/apache/commons-compress/pull/91#discussion_r368829387
 
 

 ##
 File path: 
src/test/java/org/apache/commons/compress/compressors/bzip2/BZip2NSelectorsOverflowTest.java
 ##
 @@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.commons.compress.compressors.bzip2;
+
+import org.apache.commons.compress.AbstractTestCase;
+import org.apache.commons.compress.compressors.CompressorInputStream;
+import org.apache.commons.compress.compressors.CompressorStreamFactory;
+import org.apache.commons.compress.utils.IOUtils;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.InputStream;
+
+public class BZip2NSelectorsOverflowTest extends AbstractTestCase {
+
+/**
+ * See https://sourceware.org/ml/bzip2-devel/2019-q3/msg7.html
+ */
+@Test()
+public void shouldDecompressBlockWithNSelectorOverflow() throws Exception {
+File decompressed = 
getFile("rec02841dewiki-20191201-pages-meta-history5.xml-p8161946p8319641.bz2");
+final File toDecompress = 
getFile("rec02841dewiki-20191201-pages-meta-history5.xml-p8161946p8319641.bz2");
 
 Review comment:
   Why the decompressed file and toDecompress file are the same file?
   Maybe we should compare the decompressed file with 
src/test/resources/rec02841dewiki-20191201-pages-meta-history5.xml-p8161946p8319641?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [commons-compress] bodewig commented on issue #91: Make bz2 decompressor robust to selector overflow

[GitBox]

bodewig commented on issue #91: Make bz2 decompressor robust to selector 
overflow
URL: https://github.com/apache/commons-compress/pull/91#issuecomment-576541516
 
 
   Many thanks @jobar 
   
   The test fails on my Linux box and seems to fail on Travis as well, Could 
this be a line-separator OS-difference issue on the XML file? If so, it might 
be better to pick a different testcase - if possible - anyway as [it's 
licesne(https://dumps.wikimedia.org/legal.html) may force us to jump through a 
few extra hoops, see https://apache.org/legal/resolved.html#cc-sa
   
   One more thing, to me it looks as if the `currentWorkingBlock` variable you 
introduced was never read and can safely be removed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (CSV-248) CSVRecord is not Serializable

[Gary D. Gregory (Jira)]

[ 
https://issues.apache.org/jira/browse/CSV-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019774#comment-17019774
 ] 

Gary D. Gregory edited comment on CSV-248 at 1/20/20 11:40 PM:
---

[~aherbert]

The record's parser field is now transient. 

 


was (Author: garydgregory):
[~aherbert]

The record's parser field is now transient. Please verify and close.

 

> CSVRecord is not Serializable
> -
>
> Key: CSV-248
> URL: https://issues.apache.org/jira/browse/CSV-248
> Project: Commons CSV
>  Issue Type: Bug
>Affects Versions: 1.7
>Reporter: Alex Herbert
>Priority: Major
> Fix For: 1.8
>
>
> CSVRecord is no longer Serializable as it stores the CSVParser and that is 
> not serializable.
> The parser contains a list of all the CSVRecords. So to serialize this would 
> serialize a lot of extra information. The cascade of serialization eventually 
> includes the original BufferedReader used to read the data.
> The parser is required for the header map functionality and the getParser() 
> method. The easy fix is to not support any functionality related to the 
> parser after deserialization. If the header map functionality is to be 
> supported the class can store the header map (easy), or overload the 
> serialization methods to record and load the header map (more effort), or 
> something else.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (CSV-248) CSVRecord is not Serializable

[Gary D. Gregory (Jira)]

 [ 
https://issues.apache.org/jira/browse/CSV-248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gary D. Gregory updated CSV-248:

Fix Version/s: 1.8

> CSVRecord is not Serializable
> -
>
> Key: CSV-248
> URL: https://issues.apache.org/jira/browse/CSV-248
> Project: Commons CSV
>  Issue Type: Bug
>Affects Versions: 1.7
>Reporter: Alex Herbert
>Priority: Major
> Fix For: 1.8
>
>
> CSVRecord is no longer Serializable as it stores the CSVParser and that is 
> not serializable.
> The parser contains a list of all the CSVRecords. So to serialize this would 
> serialize a lot of extra information. The cascade of serialization eventually 
> includes the original BufferedReader used to read the data.
> The parser is required for the header map functionality and the getParser() 
> method. The easy fix is to not support any functionality related to the 
> parser after deserialization. If the header map functionality is to be 
> supported the class can store the header map (easy), or overload the 
> serialization methods to record and load the header map (more effort), or 
> something else.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CSV-248) CSVRecord is not Serializable

[Gary D. Gregory (Jira)]

[ 
https://issues.apache.org/jira/browse/CSV-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019774#comment-17019774
 ] 

Gary D. Gregory commented on CSV-248:
-

[~aherbert]

The record's parser field is now transient. Please verify and close.

 

> CSVRecord is not Serializable
> -
>
> Key: CSV-248
> URL: https://issues.apache.org/jira/browse/CSV-248
> Project: Commons CSV
>  Issue Type: Bug
>Affects Versions: 1.7
>Reporter: Alex Herbert
>Priority: Major
>
> CSVRecord is no longer Serializable as it stores the CSVParser and that is 
> not serializable.
> The parser contains a list of all the CSVRecords. So to serialize this would 
> serialize a lot of extra information. The cascade of serialization eventually 
> includes the original BufferedReader used to read the data.
> The parser is required for the header map functionality and the getParser() 
> method. The easy fix is to not support any functionality related to the 
> parser after deserialization. If the header map functionality is to be 
> supported the class can store the header map (easy), or overload the 
> serialization methods to record and load the header map (more effort), or 
> something else.
>  
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NUMBERS-142) Improve LinearCombination accuracy during summation of the round-off errors

[Alex Herbert (Jira)]

 [ 
https://issues.apache.org/jira/browse/NUMBERS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Herbert updated NUMBERS-142:
-
Attachment: cond_no.jpg

> Improve LinearCombination accuracy during summation of the round-off errors
> ---
>
> Key: NUMBERS-142
> URL: https://issues.apache.org/jira/browse/NUMBERS-142
> Project: Commons Numbers
>  Issue Type: Improvement
>  Components: arrays
>Affects Versions: 1.0
>Reporter: Alex Herbert
>Assignee: Alex Herbert
>Priority: Minor
> Attachments: array_performance.jpg, cond_no.jpg, 
> error_vs_condition_no.jpg, inline_perfomance.jpg
>
>
> The algorithm in LinearCombination is an implementation of dot2 from [Ogita 
> el al|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.2.1547] 
> (algorithm 5.3). There is a subtle difference in that the original dot2 
> algorithm sums the round-off from the products and the round-off from the 
> product summation together. The method in LinearCombination sums them 
> separately (using an extra variable) and adds them at the end. This actually 
> improves the accuracy under conditions where the round-off is of greater sum 
> than the products, as described below.
> The dot2 algorithm suffers when the total sum is close to 0 but the 
> intermediate sum is far enough from zero that there is a large difference 
> between the exponents of summed terms and the final result. In this case the 
> sum of the round-off is more important than the sum of the products which due 
> to massive cancellation is zero. The case is important for Complex numbers 
> which require a computation of log1p(x^2+y^2-1) when x^2+y^2 is close to 1 
> such that log(1) would be ~zero yet the true logarithm is representable to 
> very high precision.
> This can be protected against by using the dotK algorithm of Ogita et al with 
> K>2. This saves all the round-off parts from each product and the running 
> sum. These are subject to an error free transform that repeatedly adds 
> adjacent pairs to generate a new split pair with a closer upper and lower 
> part. Over time this will order the parts from low to high and these can be 
> summed low first for an error free dot product.
> Using this algorithm with a single pass (K=3 for dot3) removes the 
> cancellation error observed in the mentioned use case. Adding a single pass 
> over the parts changes the algorithm from 25n floating point operations 
> (flops) to 31n flops for the sum of n products.
> A second change for the algorithm is to switch to using 
> [Dekker's|https://doi.org/10.1007/BF01397083] algorithm (Dekker, 1971) to 
> split the number. This extracts two 26-bit mantissas from a 53-bit mantis (in 
> IEEE 754 the leading bit in front of the of the 52-bit mantissa is assumed to 
> be 1). This is done by multiplication by 2^s+1 with s = ceil(53/2) = 27:
> big = (2^s+1) * a
> a_hi = (big - (big - a))
> The extra bit of precision is carried into the sign of the low part of the 
> split number.
> This is in contrast to the method in LinearCombination that uses a simple 
> mask on the long representation to obtain the a_hi part in 26-bits and the 
> lower part will be 27 bits.
> The advantage of Dekker's method is it produces 2 parts with 26 bits in the 
> mantissa that can be multiplied exactly. The disadvantage is the potential 
> for overflow requiring a branch condition to check for extreme values.
> It also appropriately handles very small sub-normal numbers that would be 
> masked to create a 0 high part with all the non-zero bits left in the low 
> part using the current method. This will have knock on effects on split 
> multiplication which requires the high part to be larger.
> A simple change to the current implementation to use Dekker's split improves 
> the precision on a wide range of test data (not shown).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (NUMBERS-142) Improve LinearCombination accuracy during summation of the round-off errors

[Alex Herbert (Jira)]

[ 
https://issues.apache.org/jira/browse/NUMBERS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019773#comment-17019773
 ] 

Alex Herbert commented on NUMBERS-142:
--

The previous data suggested that the exact method of Shewchuk performs well 
relative to the dotK implementations. However that data did not use a poorly 
conditioned dot product in the speed test. The following data show that the 
Shewchuk method slows as the condition number increases whereas the DotK method 
is constant. This can be attributed to the extended precision required for the 
more difficult summation. Interestingly the BigDecimal method gets "faster" on 
a more difficult dot product.

This is for vectors of length 100. The same condition number for length 1000 
and 1 have similar relative performance.
||Condition No||Method||Score||Error||Median||Relative Median||
|1.00E+05|scalarDot3|1128677.76|37024.09|1134183.29|1.00|
|1.00E+05|scalarDot6|2284273.20|274981.79|2255035.39|1.99|
|1.00E+05|scalarDotExact|3837295.64|751706.22|3752483.44|3.31|
|1.00E+05|scalarBigDecimal|95740770.75|2267832.75|95783316.55|84.45|
|1.00E+10|scalarDot3|1125253.32|27139.50|1128633.77|1.00|
|1.00E+10|scalarDot6|2271400.29|256149.52|2233608.10|1.98|
|1.00E+10|scalarDotExact|4059975.57|22578.46|4061035.84|3.60|
|1.00E+10|scalarBigDecimal|95318128.67|4998706.87|95102055.45|84.26|
|1.00E+20|scalarDot3|1124323.07|13228.34|1123770.11|1.00|
|1.00E+20|scalarDot6|2282273.29|239657.26|2243262.44|2.00|
|1.00E+20|scalarDotExact|4710625.98|68596.77|4711449.46|4.19|
|1.00E+20|scalarBigDecimal|91728631.53|1450206.61|91689054.55|81.59|
|1.00E+40|scalarDot3|1129251.96|62729.61|1122781.51|1.00|
|1.00E+40|scalarDot6|2300410.32|336933.83|2253918.25|2.01|
|1.00E+40|scalarDotExact|5947991.96|215165.31|5926754.76|5.28|
|1.00E+40|scalarBigDecimal|78727492.06|7081432.86|79064450.08|70.42|
|1.00E+80|scalarDot3|1130034.38|43218.04|1129013.73|1.00|
|1.00E+80|scalarDot6|2293919.11|335906.85|2255447.04|2.00|
|1.00E+80|scalarDotExact|7689500.97|87407.59|7698025.89|6.82|
|1.00E+80|scalarBigDecimal|67884835.73|5311378.14|67382666.67|59.68|
|1.00E+160|scalarDot3|1119655.61|37682.30|1119518.49|1.00|
|1.00E+160|scalarDot6|2291444.63|227219.19|2284083.80|2.04|
|1.00E+160|scalarDotExact|9619403.93|129190.28|9603358.40|8.58|
|1.00E+160|scalarBigDecimal|75378899.45|21965377.80|72823951.43|65.05|

!cond_no.jpg!

> Improve LinearCombination accuracy during summation of the round-off errors
> ---
>
> Key: NUMBERS-142
> URL: https://issues.apache.org/jira/browse/NUMBERS-142
> Project: Commons Numbers
>  Issue Type: Improvement
>  Components: arrays
>Affects Versions: 1.0
>Reporter: Alex Herbert
>Assignee: Alex Herbert
>Priority: Minor
> Attachments: array_performance.jpg, cond_no.jpg, 
> error_vs_condition_no.jpg, inline_perfomance.jpg
>
>
> The algorithm in LinearCombination is an implementation of dot2 from [Ogita 
> el al|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.2.1547] 
> (algorithm 5.3). There is a subtle difference in that the original dot2 
> algorithm sums the round-off from the products and the round-off from the 
> product summation together. The method in LinearCombination sums them 
> separately (using an extra variable) and adds them at the end. This actually 
> improves the accuracy under conditions where the round-off is of greater sum 
> than the products, as described below.
> The dot2 algorithm suffers when the total sum is close to 0 but the 
> intermediate sum is far enough from zero that there is a large difference 
> between the exponents of summed terms and the final result. In this case the 
> sum of the round-off is more important than the sum of the products which due 
> to massive cancellation is zero. The case is important for Complex numbers 
> which require a computation of log1p(x^2+y^2-1) when x^2+y^2 is close to 1 
> such that log(1) would be ~zero yet the true logarithm is representable to 
> very high precision.
> This can be protected against by using the dotK algorithm of Ogita et al with 
> K>2. This saves all the round-off parts from each product and the running 
> sum. These are subject to an error free transform that repeatedly adds 
> adjacent pairs to generate a new split pair with a closer upper and lower 
> part. Over time this will order the parts from low to high and these can be 
> summed low first for an error free dot product.
> Using this algorithm with a single pass (K=3 for dot3) removes the 
> cancellation error observed in the mentioned use case. Adding a single pass 
> over the parts changes the algorithm from 25n floating point operations 
> (flops) to 31n flops for the sum of n products.
> A second change for the algorithm is to switch to using 
> 

[GitHub] [commons-csv] garydgregory merged pull request #55: Fix typo performance test

[GitBox]

garydgregory merged pull request #55: Fix typo performance test
URL: https://github.com/apache/commons-csv/pull/55
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (NUMBERS-142) Improve LinearCombination accuracy during summation of the round-off errors

[Alex Herbert (Jira)]

[ 
https://issues.apache.org/jira/browse/NUMBERS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019757#comment-17019757
 ] 

Alex Herbert commented on NUMBERS-142:
--

I have implemented various algorithms for extended precision and tested speed 
and accuracy. It is no surprise there is a trade-off with faster methods being 
less accurate.
h2. Algorithms tested
 # Dekker's dot product as described in Dekker, 1971. 
 # Ogita's fast variant of dot2 (dot2s)
 # Ogita's fast variant of dot2 using a binary split of the double into high 
and low parts (dot2sMask)
 # Ogita's dotK for K in 3 to 7
 # Shewchuk's method for arbitrary precision float numbers
 # BigDecimal

h2. Testing using a poorly conditioned dot-product

I implemented the method from Ogita et al to create a poorly conditioned dot 
product. This creates random numbers with a carefully chosen upper bounds for 
half of the arrays x and y. The remaining half is then created to gradually 
bring the sum back towards zero. The final dot product of x and y is in the 
range [-1, 1] and components may be up to 2^b/2 in magnitude where b is the 
logarithm of the condition number.

The method uses a target condition number to set bounds when creating the data. 
The final condition number (used for analysis) is defined as:
{noformat}
d = exact(x, y)
c = 2 * |x|^T * |y| / |d|
{noformat}
This require the exact dot product. I compute this using BigDecimal. The 
numerator is the dot product in standard precision of the absolute vectors. 
Higher means the magnitudes of the product components are further away from the 
magnitude of the dot sum and thus there is more cancellation during the sum.

Performance is assessed using relative error:
{noformat}
relative error = |observed - expected| / |expected|
{noformat}
This is clipped to a maximum of 2. Above that is meaningless, i.e the method is 
just wrong.
h2. Accuracy

To test the implementations I recreated Figure 6.2 to 6.4 from Ogita et al 
(2005). This plots the relative error against the condition number. I used 2000 
samples of length 100 vectors with a condition number of 1e10 to 1e120. At a 
certain condition number various methods can be seen to fail as shown below:

!error_vs_condition_no.jpg!
 * BigDecimal is not plotted as it is the reference
 * Shewchuk's method cannot be seen as it is exact (DotExact). This holds with 
a condition number up to 1e300.
 * Dekker's method can be seen marginally outperforming the other dot2 methods. 
It is nowhere close to dot3.
 * The variant of dot2s using a different split (Dot2sMask) cannot be 
distinguished in this test.

h2. Performance

Tested using 1000 random vectors of different lengths. The median of 5 JMH runs 
is shown:

!array_performance.jpg!
 * BigDecimal is slow
 * Shewchuk's method is about the same speed as dot6 or dot7. It is about 3.5 
slower than the speed of dot2s (the current implementation in 
LinearCombination).
 * The single precision scalar product (baseline) is about 8x faster than dot2s

h2. Inline Performance

I wrote a few different versions using the inlining of the dot product for 
small vectors. These were tested to ensure they return the same values as the 
generic array versions:

!inline_perfomance.jpg!
 * BigDecimal is not plotted as it is so slow
 * Shewchuk's method is close in speed to dot3 for length 2. The gaps grows for 
length 3 and 4.
 * The single precision scalar product is about 4-8x faster than dot2sMask
 * The simple masking approach (Mask suffix) which requires no branch 
conditions is definitely faster than using Dekker's split

The same data in a table. Note the slow speed of BigDecimal.
||Length||Baseline||BigDecimal||Dekker||Dot2s||Dot3||DotExact||Dot2sMask||Dot3Mask||
|2|2836.9|4786548.5|10386.0|10161.6|14985.9|17526.6|8045.9|12310.1|
|3|2896.1|5705362.5|20570.7|14778.2|23543.6|41584.1|12100.9|19618.6|
|4|3282.5|6609295.1|35116.4|24626.8|32823.7|64874.2|18174.7|25627.6|
h2. Discussion
 * BigDecimal is slow
 * The accuracy of BigDecimal can be achieved using Shewchuk's extended 
precision method
 * All double length precision methods are approximately equal.
 * Improving precision of dotK requires iterations of the summation (the 
parameter K)
 * Any implementation of dotK will have a failure threshold unless K is very 
large. At this point Shewchuk's method is faster and more accurate

The implementation of Shewchuk's method requires that an input of length n may 
be expanded to an arbitrary precision float represented as up to 2n double 
values in increasing order of magnitude. These values are non-overlapping. Each 
addition of a new product to the expanded number of size m requires 2m split 
sum operations and increases the length by 2. Thus the algorithm should have 
order(n^2) runtime complexity. However some additions occur with no round-off 
and thus the round-off does not have to be stored. This occurs in part as the 
numbers are 

[jira] [Updated] (NUMBERS-142) Improve LinearCombination accuracy during summation of the round-off errors

[Alex Herbert (Jira)]

 [ 
https://issues.apache.org/jira/browse/NUMBERS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Herbert updated NUMBERS-142:
-
Attachment: inline_perfomance.jpg

> Improve LinearCombination accuracy during summation of the round-off errors
> ---
>
> Key: NUMBERS-142
> URL: https://issues.apache.org/jira/browse/NUMBERS-142
> Project: Commons Numbers
>  Issue Type: Improvement
>  Components: arrays
>Affects Versions: 1.0
>Reporter: Alex Herbert
>Assignee: Alex Herbert
>Priority: Minor
> Attachments: array_performance.jpg, error_vs_condition_no.jpg, 
> inline_perfomance.jpg
>
>
> The algorithm in LinearCombination is an implementation of dot2 from [Ogita 
> el al|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.2.1547] 
> (algorithm 5.3). There is a subtle difference in that the original dot2 
> algorithm sums the round-off from the products and the round-off from the 
> product summation together. The method in LinearCombination sums them 
> separately (using an extra variable) and adds them at the end. This actually 
> improves the accuracy under conditions where the round-off is of greater sum 
> than the products, as described below.
> The dot2 algorithm suffers when the total sum is close to 0 but the 
> intermediate sum is far enough from zero that there is a large difference 
> between the exponents of summed terms and the final result. In this case the 
> sum of the round-off is more important than the sum of the products which due 
> to massive cancellation is zero. The case is important for Complex numbers 
> which require a computation of log1p(x^2+y^2-1) when x^2+y^2 is close to 1 
> such that log(1) would be ~zero yet the true logarithm is representable to 
> very high precision.
> This can be protected against by using the dotK algorithm of Ogita et al with 
> K>2. This saves all the round-off parts from each product and the running 
> sum. These are subject to an error free transform that repeatedly adds 
> adjacent pairs to generate a new split pair with a closer upper and lower 
> part. Over time this will order the parts from low to high and these can be 
> summed low first for an error free dot product.
> Using this algorithm with a single pass (K=3 for dot3) removes the 
> cancellation error observed in the mentioned use case. Adding a single pass 
> over the parts changes the algorithm from 25n floating point operations 
> (flops) to 31n flops for the sum of n products.
> A second change for the algorithm is to switch to using 
> [Dekker's|https://doi.org/10.1007/BF01397083] algorithm (Dekker, 1971) to 
> split the number. This extracts two 26-bit mantissas from a 53-bit mantis (in 
> IEEE 754 the leading bit in front of the of the 52-bit mantissa is assumed to 
> be 1). This is done by multiplication by 2^s+1 with s = ceil(53/2) = 27:
> big = (2^s+1) * a
> a_hi = (big - (big - a))
> The extra bit of precision is carried into the sign of the low part of the 
> split number.
> This is in contrast to the method in LinearCombination that uses a simple 
> mask on the long representation to obtain the a_hi part in 26-bits and the 
> lower part will be 27 bits.
> The advantage of Dekker's method is it produces 2 parts with 26 bits in the 
> mantissa that can be multiplied exactly. The disadvantage is the potential 
> for overflow requiring a branch condition to check for extreme values.
> It also appropriately handles very small sub-normal numbers that would be 
> masked to create a 0 high part with all the non-zero bits left in the low 
> part using the current method. This will have knock on effects on split 
> multiplication which requires the high part to be larger.
> A simple change to the current implementation to use Dekker's split improves 
> the precision on a wide range of test data (not shown).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NUMBERS-142) Improve LinearCombination accuracy during summation of the round-off errors

[Alex Herbert (Jira)]

 [ 
https://issues.apache.org/jira/browse/NUMBERS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Herbert updated NUMBERS-142:
-
Attachment: error_vs_condition_no.jpg

> Improve LinearCombination accuracy during summation of the round-off errors
> ---
>
> Key: NUMBERS-142
> URL: https://issues.apache.org/jira/browse/NUMBERS-142
> Project: Commons Numbers
>  Issue Type: Improvement
>  Components: arrays
>Affects Versions: 1.0
>Reporter: Alex Herbert
>Assignee: Alex Herbert
>Priority: Minor
> Attachments: array_performance.jpg, error_vs_condition_no.jpg, 
> inline_perfomance.jpg
>
>
> The algorithm in LinearCombination is an implementation of dot2 from [Ogita 
> el al|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.2.1547] 
> (algorithm 5.3). There is a subtle difference in that the original dot2 
> algorithm sums the round-off from the products and the round-off from the 
> product summation together. The method in LinearCombination sums them 
> separately (using an extra variable) and adds them at the end. This actually 
> improves the accuracy under conditions where the round-off is of greater sum 
> than the products, as described below.
> The dot2 algorithm suffers when the total sum is close to 0 but the 
> intermediate sum is far enough from zero that there is a large difference 
> between the exponents of summed terms and the final result. In this case the 
> sum of the round-off is more important than the sum of the products which due 
> to massive cancellation is zero. The case is important for Complex numbers 
> which require a computation of log1p(x^2+y^2-1) when x^2+y^2 is close to 1 
> such that log(1) would be ~zero yet the true logarithm is representable to 
> very high precision.
> This can be protected against by using the dotK algorithm of Ogita et al with 
> K>2. This saves all the round-off parts from each product and the running 
> sum. These are subject to an error free transform that repeatedly adds 
> adjacent pairs to generate a new split pair with a closer upper and lower 
> part. Over time this will order the parts from low to high and these can be 
> summed low first for an error free dot product.
> Using this algorithm with a single pass (K=3 for dot3) removes the 
> cancellation error observed in the mentioned use case. Adding a single pass 
> over the parts changes the algorithm from 25n floating point operations 
> (flops) to 31n flops for the sum of n products.
> A second change for the algorithm is to switch to using 
> [Dekker's|https://doi.org/10.1007/BF01397083] algorithm (Dekker, 1971) to 
> split the number. This extracts two 26-bit mantissas from a 53-bit mantis (in 
> IEEE 754 the leading bit in front of the of the 52-bit mantissa is assumed to 
> be 1). This is done by multiplication by 2^s+1 with s = ceil(53/2) = 27:
> big = (2^s+1) * a
> a_hi = (big - (big - a))
> The extra bit of precision is carried into the sign of the low part of the 
> split number.
> This is in contrast to the method in LinearCombination that uses a simple 
> mask on the long representation to obtain the a_hi part in 26-bits and the 
> lower part will be 27 bits.
> The advantage of Dekker's method is it produces 2 parts with 26 bits in the 
> mantissa that can be multiplied exactly. The disadvantage is the potential 
> for overflow requiring a branch condition to check for extreme values.
> It also appropriately handles very small sub-normal numbers that would be 
> masked to create a 0 high part with all the non-zero bits left in the low 
> part using the current method. This will have knock on effects on split 
> multiplication which requires the high part to be larger.
> A simple change to the current implementation to use Dekker's split improves 
> the precision on a wide range of test data (not shown).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (NUMBERS-142) Improve LinearCombination accuracy during summation of the round-off errors

[Alex Herbert (Jira)]

 [ 
https://issues.apache.org/jira/browse/NUMBERS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alex Herbert updated NUMBERS-142:
-
Attachment: array_performance.jpg

> Improve LinearCombination accuracy during summation of the round-off errors
> ---
>
> Key: NUMBERS-142
> URL: https://issues.apache.org/jira/browse/NUMBERS-142
> Project: Commons Numbers
>  Issue Type: Improvement
>  Components: arrays
>Affects Versions: 1.0
>Reporter: Alex Herbert
>Assignee: Alex Herbert
>Priority: Minor
> Attachments: array_performance.jpg, error_vs_condition_no.jpg, 
> inline_perfomance.jpg
>
>
> The algorithm in LinearCombination is an implementation of dot2 from [Ogita 
> el al|http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.2.1547] 
> (algorithm 5.3). There is a subtle difference in that the original dot2 
> algorithm sums the round-off from the products and the round-off from the 
> product summation together. The method in LinearCombination sums them 
> separately (using an extra variable) and adds them at the end. This actually 
> improves the accuracy under conditions where the round-off is of greater sum 
> than the products, as described below.
> The dot2 algorithm suffers when the total sum is close to 0 but the 
> intermediate sum is far enough from zero that there is a large difference 
> between the exponents of summed terms and the final result. In this case the 
> sum of the round-off is more important than the sum of the products which due 
> to massive cancellation is zero. The case is important for Complex numbers 
> which require a computation of log1p(x^2+y^2-1) when x^2+y^2 is close to 1 
> such that log(1) would be ~zero yet the true logarithm is representable to 
> very high precision.
> This can be protected against by using the dotK algorithm of Ogita et al with 
> K>2. This saves all the round-off parts from each product and the running 
> sum. These are subject to an error free transform that repeatedly adds 
> adjacent pairs to generate a new split pair with a closer upper and lower 
> part. Over time this will order the parts from low to high and these can be 
> summed low first for an error free dot product.
> Using this algorithm with a single pass (K=3 for dot3) removes the 
> cancellation error observed in the mentioned use case. Adding a single pass 
> over the parts changes the algorithm from 25n floating point operations 
> (flops) to 31n flops for the sum of n products.
> A second change for the algorithm is to switch to using 
> [Dekker's|https://doi.org/10.1007/BF01397083] algorithm (Dekker, 1971) to 
> split the number. This extracts two 26-bit mantissas from a 53-bit mantis (in 
> IEEE 754 the leading bit in front of the of the 52-bit mantissa is assumed to 
> be 1). This is done by multiplication by 2^s+1 with s = ceil(53/2) = 27:
> big = (2^s+1) * a
> a_hi = (big - (big - a))
> The extra bit of precision is carried into the sign of the low part of the 
> split number.
> This is in contrast to the method in LinearCombination that uses a simple 
> mask on the long representation to obtain the a_hi part in 26-bits and the 
> lower part will be 27 bits.
> The advantage of Dekker's method is it produces 2 parts with 26 bits in the 
> mantissa that can be multiplied exactly. The disadvantage is the potential 
> for overflow requiring a branch condition to check for extreme values.
> It also appropriately handles very small sub-normal numbers that would be 
> masked to create a 0 high part with all the non-zero bits left in the low 
> part using the current method. This will have knock on effects on split 
> multiplication which requires the high part to be larger.
> A simple change to the current implementation to use Dekker's split improves 
> the precision on a wide range of test data (not shown).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [commons-compress] jobar opened a new pull request #91: Make bz2 decompressor robust to selector overflow

[GitBox]

jobar opened a new pull request #91: Make bz2 decompressor robust to selector 
overflow
URL: https://github.com/apache/commons-compress/pull/91
 
 
   As explained in 
https://gnu.wildebeest.org/blog/mjw/2019/08/02/bzip2-and-the-cve-that-wasnt/
   bz2 has historically been able to decompress wrongly formatted compressed
   files, more precisely files stating more selectors that the algorithm
   actually uses.
   This patch replicates 
https://sourceware.org/ml/bzip2-devel/2019-q3/msg7.html
   and provides a test file extracted from 20191201 German Xikipedia XML dump:
   
https://dumps.wikimedia.org/dewiki/20191201/dewiki-20191201-pages-meta-history5.xml-p8161946p8319641.bz2
   (the link will disappear soon, as new dumps are generated).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Comment Edited] (CONFIGURATION-753) Handling of interpolation is inconsistent

[Gary D. Gregory (Jira)]

[ 
https://issues.apache.org/jira/browse/CONFIGURATION-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019728#comment-17019728
 ] 

Gary D. Gregory edited comment on CONFIGURATION-753 at 1/20/20 8:37 PM:


[~tpoliaw]

Looking at 
[https://commons.apache.org/proper/commons-configuration/userguide/howto_properties.html#Properties_files]
 it would seem that the way to do list values in properties file is, for 
example:

foo = value1, value2, value3

There is also 
[https://commons.apache.org/proper/commons-configuration/userguide/howto_basicfeatures.html#List_handling]

What is worrisome is that there is some code that must build a list based on 
repeated keys and I am not sure if that is documented or intentional. This is 
obviously an area where a unit test is needed if it is missing.

Maybe someone else can chime in. I find it odd that repeating keys would build 
a list since that is far from what a java.util.Properties object can do.

Thoughts?

 


was (Author: garydgregory):
[~tpoliaw]

Looking at 
[https://commons.apache.org/proper/commons-configuration/userguide/howto_properties.html#Properties_files]
 it would seem that the way to do list values in properties file is, for 
example:

foo = value1, value2, value3

There is also 
[https://commons.apache.org/proper/commons-configuration/userguide/howto_basicfeatures.html#List_handling]

What is worrisome is that there is some code that must build a list based on 
repeated keys and I am not sure if that is documented or intentional.

Maybe someone else can chime in. I find it odd that repeating keys would build 
a list since that is far from what a java.util.Properties object can do.

Thoughts?

 

> Handling of interpolation is inconsistent
> -
>
> Key: CONFIGURATION-753
> URL: https://issues.apache.org/jira/browse/CONFIGURATION-753
> Project: Commons Configuration
>  Issue Type: Bug
>  Components: Interpolation
>Affects Versions: 2.5
> Environment: Java 8, Configurations2 2.5
>Reporter: Peter
>Priority: Major
> Attachments: test.properties
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If a key is repeated in a configuration and then used in an interpolation 
> elsewhere, the behaviour is inconsistent. There are other tickets/discussions 
> about whether it should just pick the first value or not, but I don't think 
> it should do both.
> {code:java|title=/tmp/test.properties}
> abc = hello
> abc = world
> foo.one = ${abc}
> foo.two = prefix ${abc} suffix
> {code}
> {code:java|title=Demo.java (main)}
> Parameters params = new Parameters();
> FileBasedConfigurationBuilder builder = new 
> FileBasedConfigurationBuilder(PropertiesConfiguration.class)
> .configure(params.fileBased()
> .setFileName("/tmp/test.properties")
>   );
> try {
> FileBasedConfiguration config = builder.getConfiguration();
> System.out.println(config.getString("foo.one"));
> System.out.println(config.getString("foo.two"));
> } catch (ConfigurationException cex) {
> // pass
> }
> {code}
> The output from the above is
> {noformat}
> hello 
> prefix [hello, world] suffix
> {noformat}
> In the first case, only the first value is being matched, in the second both 
> values (and [, ]) are used.
> I'd expect the output to either be
> {noformat:title=First value only}
> hello
> prefix hello suffix
> {noformat}
> or
> {noformat:title=Both values used}
> [hello, world]
> prefix [hello, world] suffix
> {noformat}
> I can work around whichever style is chosen but think it'd be much more 
> intuitive if both cases were handled the same.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CONFIGURATION-753) Handling of interpolation is inconsistent

[Gary D. Gregory (Jira)]

[ 
https://issues.apache.org/jira/browse/CONFIGURATION-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019728#comment-17019728
 ] 

Gary D. Gregory commented on CONFIGURATION-753:
---

[~tpoliaw]

Looking at 
[https://commons.apache.org/proper/commons-configuration/userguide/howto_properties.html#Properties_files]
 it would seem that the way to do list values in properties file is, for 
example:

foo = value1, value2, value3

There is also 
[https://commons.apache.org/proper/commons-configuration/userguide/howto_basicfeatures.html#List_handling]

What is worrisome is that there is some code that must build a list based on 
repeated keys and I am not sure if that is documented or intentional.

Maybe someone else can chime in. I find it odd that repeating keys would build 
a list since that is far from what a java.util.Properties object can do.

Thoughts?

 

> Handling of interpolation is inconsistent
> -
>
> Key: CONFIGURATION-753
> URL: https://issues.apache.org/jira/browse/CONFIGURATION-753
> Project: Commons Configuration
>  Issue Type: Bug
>  Components: Interpolation
>Affects Versions: 2.5
> Environment: Java 8, Configurations2 2.5
>Reporter: Peter
>Priority: Major
> Attachments: test.properties
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If a key is repeated in a configuration and then used in an interpolation 
> elsewhere, the behaviour is inconsistent. There are other tickets/discussions 
> about whether it should just pick the first value or not, but I don't think 
> it should do both.
> {code:java|title=/tmp/test.properties}
> abc = hello
> abc = world
> foo.one = ${abc}
> foo.two = prefix ${abc} suffix
> {code}
> {code:java|title=Demo.java (main)}
> Parameters params = new Parameters();
> FileBasedConfigurationBuilder builder = new 
> FileBasedConfigurationBuilder(PropertiesConfiguration.class)
> .configure(params.fileBased()
> .setFileName("/tmp/test.properties")
>   );
> try {
> FileBasedConfiguration config = builder.getConfiguration();
> System.out.println(config.getString("foo.one"));
> System.out.println(config.getString("foo.two"));
> } catch (ConfigurationException cex) {
> // pass
> }
> {code}
> The output from the above is
> {noformat}
> hello 
> prefix [hello, world] suffix
> {noformat}
> In the first case, only the first value is being matched, in the second both 
> values (and [, ]) are used.
> I'd expect the output to either be
> {noformat:title=First value only}
> hello
> prefix hello suffix
> {noformat}
> or
> {noformat:title=Both values used}
> [hello, world]
> prefix [hello, world] suffix
> {noformat}
> I can work around whichever style is chosen but think it'd be much more 
> intuitive if both cases were handled the same.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CONFIGURATION-753) Handling of interpolation is inconsistent

[Peter (Jira)]

[ 
https://issues.apache.org/jira/browse/CONFIGURATION-753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019634#comment-17019634
 ] 

Peter commented on CONFIGURATION-753:
-

[~ggregory], we're keen to update to configurations2 and this is a blocker for 
us. What would it need for this to feel more fully baked? Happy to make more 
changes to the PR.

 

> Handling of interpolation is inconsistent
> -
>
> Key: CONFIGURATION-753
> URL: https://issues.apache.org/jira/browse/CONFIGURATION-753
> Project: Commons Configuration
>  Issue Type: Bug
>  Components: Interpolation
>Affects Versions: 2.5
> Environment: Java 8, Configurations2 2.5
>Reporter: Peter
>Priority: Major
> Attachments: test.properties
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> If a key is repeated in a configuration and then used in an interpolation 
> elsewhere, the behaviour is inconsistent. There are other tickets/discussions 
> about whether it should just pick the first value or not, but I don't think 
> it should do both.
> {code:java|title=/tmp/test.properties}
> abc = hello
> abc = world
> foo.one = ${abc}
> foo.two = prefix ${abc} suffix
> {code}
> {code:java|title=Demo.java (main)}
> Parameters params = new Parameters();
> FileBasedConfigurationBuilder builder = new 
> FileBasedConfigurationBuilder(PropertiesConfiguration.class)
> .configure(params.fileBased()
> .setFileName("/tmp/test.properties")
>   );
> try {
> FileBasedConfiguration config = builder.getConfiguration();
> System.out.println(config.getString("foo.one"));
> System.out.println(config.getString("foo.two"));
> } catch (ConfigurationException cex) {
> // pass
> }
> {code}
> The output from the above is
> {noformat}
> hello 
> prefix [hello, world] suffix
> {noformat}
> In the first case, only the first value is being matched, in the second both 
> values (and [, ]) are used.
> I'd expect the output to either be
> {noformat:title=First value only}
> hello
> prefix hello suffix
> {noformat}
> or
> {noformat:title=Both values used}
> [hello, world]
> prefix [hello, world] suffix
> {noformat}
> I can work around whichever style is chosen but think it'd be much more 
> intuitive if both cases were handled the same.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (JEXL-307) Variable redeclaration option

[Henri Biestro (Jira)]

[ 
https://issues.apache.org/jira/browse/JEXL-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019577#comment-17019577
 ] 

Henri Biestro commented on JEXL-307:


Changeset: f0942719bf96487596fa71f0b7fdcc1d004c34e0
Author:henrib 
Date:  2020-01-20 17:12
Message:   JEXL-307: fixing regression, parameter declaration under lexical 
feature

> Variable redeclaration option
> -
>
> Key: JEXL-307
> URL: https://issues.apache.org/jira/browse/JEXL-307
> Project: Commons JEXL
>  Issue Type: New Feature
>Affects Versions: 3.1
>Reporter: Dmitri Blinov
>Assignee: Henri Biestro
>Priority: Minor
> Fix For: 3.2
>
>
> As of now, JEXL allows a script writer to redeclare a local variable during 
> script evaluation.
> {code:java}
> var a = 1; var a = 2;{code}
> This may lead to potential errors with misspelled names and clashed 
> variables. Checking for already defined variable is a common feature of many 
> languages. This feature can be implemented in JEXL as an additional option of 
> JexlFeatures class, enabled by default, thus allowing compatibility with 
> existing code.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (GEOMETRY-80) Rename Transform to AffineTransform

[Matt Juntunen (Jira)]

[ 
https://issues.apache.org/jira/browse/GEOMETRY-80?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019565#comment-17019565
 ] 

Matt Juntunen commented on GEOMETRY-80:
---

I closed PR #58 so I could squash the commits made during discussion on the ML. 
The new PR is https://github.com/apache/commons-geometry/pull/59. This includes 
simplification of the {{Transform}} class hierarchy and improved documentation 
but does not include any renaming of interfaces, per the ML discussion.

> Rename Transform to AffineTransform
> ---
>
> Key: GEOMETRY-80
> URL: https://issues.apache.org/jira/browse/GEOMETRY-80
> Project: Apache Commons Geometry
>  Issue Type: Improvement
>Reporter: Matt Juntunen
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> The {{Transform}} interface is intended to represent affine transforms only. 
> Perhaps, we should then rename this interface to {{AffineTransform}} to 
> distinguish it from other transform types (such as projective transforms).
> Pros:
> - Clear, unambiguous name
> - Allows possible use of other transform types later on without naming 
> conflicts
> Cons:
> - Name conflicts with {{java.awt.geom.AffineTransform}}
> - I'm not sure if the term "affine" can be used to describe this type of 
> transform in spherical space. I've only ever seen it used in discussions 
> involving Euclidean space.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[GitHub] [commons-geometry] darkma773r opened a new pull request #59: GEOMETRY-80: Simplify Transform Hierarchy

[GitBox]

darkma773r opened a new pull request #59: GEOMETRY-80: Simplify Transform 
Hierarchy
URL: https://github.com/apache/commons-geometry/pull/59
 
 
   - simplifying Transform class hierarchy
   - updating documentation


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [commons-geometry] darkma773r closed pull request #58: GEOMETRY-80: Simplify Transform Hierarchy

[GitBox]

darkma773r closed pull request #58: GEOMETRY-80: Simplify Transform Hierarchy
URL: https://github.com/apache/commons-geometry/pull/58
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [commons-geometry] darkma773r commented on issue #58: GEOMETRY-80: Simplify Transform Hierarchy

[GitBox]

darkma773r commented on issue #58: GEOMETRY-80: Simplify Transform Hierarchy
URL: https://github.com/apache/commons-geometry/pull/58#issuecomment-576325814
 
 
   Closing PR to squash commits.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Alex Herbert (Jira)]

[ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019527#comment-17019527
 ] 

Alex Herbert commented on CODEC-279:


Your stack trace is from codec 1.13. That had a bug in the validation of final 
bits so I will not cover that here.

My working shows that the exception is expected:
{noformat}
Decode blocks of 4 characters as 4 * 6 bits = 24-bits = 3 bytes

Trailing characters are decoded as:
1 =  6 bits => Ignored
2 = 12 bits => 8-bits discard 4
3 = 18 bits => 16-bits discard 2

The discarded bits are now required to be zero.

86 / 4 = 21remainder   2

Last two characters decoded to base64:
j = 35 = 100011
E =  4 = 000100
Last 12-bits = 100011 000100 
 = 10001100   0100
This has 4 trailing bits which should be zeros but they are 0100
{noformat}
Thus this is expected to throw.

Can you change the final character to 'A' and see if it still throws. I expect 
this to be fine. Then you can check that Codec returns the same bytes as the 
java.util.Base64 class.

You may ask what about 6 trailing bits when there is 1 trailing character? This 
is currently not validated. In [Codec-270] I suggested throwing an exception 
given that there should never be 6-bits left-over when decoding to 8-bit bytes. 
This would indicate a bad or truncated encoding. However this extra validation 
was not added and there is a comment in the code to determine if it should be 
validated.

The validation of trailing bytes was to ensure a round trip of 
decoding->encoding bytes produces the same bytes (see [Codec-134]). So given 
this argument I think that validation could be made more strict given that both 
Base32 and Base64 allow an entire single trailing character to be discarded but 
will throw if there are multiple trailing characters and the trailing bits to 
be discarded are non-zero.

 

> Base64.decode fails on Java11 for certain valid base 64 encoded String
> --
>
> Key: CODEC-279
> URL: https://issues.apache.org/jira/browse/CODEC-279
> Project: Commons Codec
>  Issue Type: Bug
>Affects Versions: 1.13, 1.14
> Environment: Java 11
>Reporter: Christian Kleinbölting
>Priority: Major
>
> The following lines will fail for a specific String:
> {code:java}
> String sharedSecret = 
> "cannot_share_the_affected_string_because_it_is_a_secret";
> byte[] bytes = 
> org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
> The exception I get is
> {code:java}
> java.lang.IllegalArgumentException: Last encoded character (before the 
> paddings if any) is a valid base 64 alphabet but not a possible value  at 
> org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
>   at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
>   at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
>   at 
> com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
>   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
>   at 
> 

[jira] [Commented] (JEXL-307) Variable redeclaration option

[Dmitri Blinov (Jira)]

[ 
https://issues.apache.org/jira/browse/JEXL-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019468#comment-17019468
 ] 

Dmitri Blinov commented on JEXL-307:


It seems I have found a nasty regression - the code below fails with _*x: 
variable is already defined*_
{code:java}
@Test
public void testNamed() throws Exception {
JexlFeatures f = new JexlFeatures();
f.lexical(true);
JexlEngine jexl = new JexlBuilder().strict(true).features(f).create();
JexlScript script = jexl.createScript("var i = function(x, y, z) 
{return x + y + z}; i(20,20,2)");
JexlContext jc = new MapContext();
Object result = script.execute(jc);
Assert.assertEquals(result, 42);
}
{code}
If the lexical feature is switched *off* everything works OK.

> Variable redeclaration option
> -
>
> Key: JEXL-307
> URL: https://issues.apache.org/jira/browse/JEXL-307
> Project: Commons JEXL
>  Issue Type: New Feature
>Affects Versions: 3.1
>Reporter: Dmitri Blinov
>Assignee: Henri Biestro
>Priority: Minor
> Fix For: 3.2
>
>
> As of now, JEXL allows a script writer to redeclare a local variable during 
> script evaluation.
> {code:java}
> var a = 1; var a = 2;{code}
> This may lead to potential errors with misspelled names and clashed 
> variables. Checking for already defined variable is a common feature of many 
> languages. This feature can be implemented in JEXL as an additional option of 
> JexlFeatures class, enabled by default, thus allowing compatibility with 
> existing code.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

[ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019451#comment-17019451
 ] 

Christian Kleinbölting commented on CODEC-279:
--

> Other libraries may decode this but commons codec now throws an exception.

Interestingly enough the same commons code versions work fine on Java 8 but 
fail with Java 11.

> Base64.decode fails on Java11 for certain valid base 64 encoded String
> --
>
> Key: CODEC-279
> URL: https://issues.apache.org/jira/browse/CODEC-279
> Project: Commons Codec
>  Issue Type: Bug
>Affects Versions: 1.13, 1.14
> Environment: Java 11
>Reporter: Christian Kleinbölting
>Priority: Major
>
> The following lines will fail for a specific String:
> {code:java}
> String sharedSecret = 
> "cannot_share_the_affected_string_because_it_is_a_secret";
> byte[] bytes = 
> org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
> The exception I get is
> {code:java}
> java.lang.IllegalArgumentException: Last encoded character (before the 
> paddings if any) is a valid base 64 alphabet but not a possible value  at 
> org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
>   at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
>   at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
>   at 
> com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
>   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
>   at 
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
>   at 
> 

[GitHub] [commons-csv] kensixx commented on issue #43: CSVFormat#valiadte() does not account for allowDuplicateHeaderNames

[GitBox]

kensixx commented on issue #43: CSVFormat#valiadte() does not account for 
allowDuplicateHeaderNames
URL: https://github.com/apache/commons-csv/pull/43#issuecomment-576263656
 
 
   Thanks so much for your response @garydgregory , looking forward to the next 
release! =)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Commented] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

[ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019450#comment-17019450
 ] 

Christian Kleinbölting commented on CODEC-279:
--

> If you cannot share your entire string then can you provide the length of the 
> string in ascii and the last 4 characters. From that I will be able to decode 
> the trailing bits.

The {{String.length() == 86}}. Last four characters are 
{color:#008000}6cjE{color}.

> How did you create your encoded string? Was it using Commons Codec or another 
> base 64 encoder?

I do not know how it was created.

> Base64.decode fails on Java11 for certain valid base 64 encoded String
> --
>
> Key: CODEC-279
> URL: https://issues.apache.org/jira/browse/CODEC-279
> Project: Commons Codec
>  Issue Type: Bug
>Affects Versions: 1.13, 1.14
> Environment: Java 11
>Reporter: Christian Kleinbölting
>Priority: Major
>
> The following lines will fail for a specific String:
> {code:java}
> String sharedSecret = 
> "cannot_share_the_affected_string_because_it_is_a_secret";
> byte[] bytes = 
> org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
> The exception I get is
> {code:java}
> java.lang.IllegalArgumentException: Last encoded character (before the 
> paddings if any) is a valid base 64 alphabet but not a possible value  at 
> org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
>   at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
>   at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
>   at 
> com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
>   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> 

[jira] [Updated] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

 [ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kleinbölting updated CODEC-279:
-
Description: 
The following lines will fail for a specific String:
{code:java}
String sharedSecret = "cannot_share_the_affected_string_because_it_is_a_secret";
byte[] bytes = 
org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
The exception I get is
{code:java}
java.lang.IllegalArgumentException: Last encoded character (before the paddings 
if any) is a valid base 64 alphabet but not a possible valueat 
org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
at 
com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at 
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at 

[jira] [Comment Edited] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Alex Herbert (Jira)]

[ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019449#comment-17019449
 ] 

Alex Herbert edited comment on CODEC-279 at 1/20/20 12:51 PM:
--

Hi,

This issue is related to [CODEC-270].

This occurs when there are unused bits left over from the input bytes. In this 
case it is unknown if the message has been truncated or badly encoded. The 
method requires that there are no unused bits left.

If you cannot share your entire string then can you provide the length of the 
string in ascii and the last 4 characters. From that I will be able to decode 
the trailing bits.

How did you create your encoded string? Was it using Commons Codec or another 
base 64 encoder?

If it is a random string using characters from the base 64 alphabet then this 
is not a guarantee that the string will be a valid base64 encoding. Other 
libraries may decode this but commons codec now throws an exception.





was (Author: alexherbert):
Hi,

This issue is related to [CODEC-270].

This occurs when there are unused bits left over from the input bytes. In this 
case it is unknown if the message has been truncated or badly encoded. The 
method requires that there are no unused bits left.

If you cannot share your entire string then can you provide the length of the 
string in ascii and the last 4 characters. From that I will be able to decode 
the trailing bits.

How did you create your encoded string. Was it using Commons Codec or another 
base 64 encoder?

If it is a random string using characters from the base 64 alphabet then this 
is not a guarantee that the string will be a valid base64 encoding. Othe 
libraries may decode this but commons codec now throws an exception.




> Base64.decode fails on Java11 for certain valid base 64 encoded String
> --
>
> Key: CODEC-279
> URL: https://issues.apache.org/jira/browse/CODEC-279
> Project: Commons Codec
>  Issue Type: Bug
>Affects Versions: 1.13, 1.14
> Environment: Java 11
>Reporter: Christian Kleinbölting
>Priority: Major
>
> The following lines will fail for a specific String:
> {code:java}
> String sharedSecret = 
> "cannot_share_the_affected_string_because_it_is_a_secret";
> byte[] bytes = 
> org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
> The exception I get is
> {code:java}
> java.lang.IllegalArgumentException: Last encoded character (before the 
> paddings if any) is a valid base 64 alphabet but not a possible value  at 
> org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
>   at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
>   at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
>   at 
> com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
>   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
>   at 
> 

[jira] [Commented] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Alex Herbert (Jira)]

[ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17019449#comment-17019449
 ] 

Alex Herbert commented on CODEC-279:


Hi,

This issue is related to [CODEC-270].

This occurs when there are unused bits left over from the input bytes. In this 
case it is unknown if the message has been truncated or badly encoded. The 
method requires that there are no unused bits left.

If you cannot share your entire string then can you provide the length of the 
string in ascii and the last 4 characters. From that I will be able to decode 
the trailing bits.

How did you create your encoded string. Was it using Commons Codec or another 
base 64 encoder?

If it is a random string using characters from the base 64 alphabet then this 
is not a guarantee that the string will be a valid base64 encoding. Othe 
libraries may decode this but commons codec now throws an exception.




> Base64.decode fails on Java11 for certain valid base 64 encoded String
> --
>
> Key: CODEC-279
> URL: https://issues.apache.org/jira/browse/CODEC-279
> Project: Commons Codec
>  Issue Type: Bug
>Affects Versions: 1.13, 1.14
> Environment: Java 11
>Reporter: Christian Kleinbölting
>Priority: Major
>
> The following lines will fail for a specific String:
> {code:java}
> String sharedSecret = 
> "cannot_share_the_affected_string_because_it_is_a_secret";
> byte[] bytes = 
> org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
> The exception I get is
> {code:java}
> java.lang.IllegalArgumentException: Last encoded character (before the 
> paddings if any) is a valid base 64 alphabet but not a possible value  at 
> org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
>   at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
>   at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
>   at 
> com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
>   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
>   at 
> 

[jira] [Updated] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

 [ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kleinbölting updated CODEC-279:
-
Description: 
The following lines will fail for a specific String:
{code:java}
String sharedSecret = "cannot_share_the_affected_string_because_it_is_a_secret";
byte[] bytes = 
org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
The exception I get is
{code:java}
java.lang.IllegalArgumentException: Last encoded character (before the paddings 
if any) is a valid base 64 alphabet but not a possible valueat 
org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
at 
com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at 
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at 

[jira] [Updated] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

 [ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kleinbölting updated CODEC-279:
-
Affects Version/s: 1.14

> Base64.decode fails on Java11 for certain valid base 64 encoded String
> --
>
> Key: CODEC-279
> URL: https://issues.apache.org/jira/browse/CODEC-279
> Project: Commons Codec
>  Issue Type: Bug
>Affects Versions: 1.13, 1.14
> Environment: Java 11
>Reporter: Christian Kleinbölting
>Priority: Major
>
> The following lines will fail for a specific String:
> {code:java}
> String sharedSecret = 
> "cannot_share_the_affected_string_because_it_is_a_secret";
> byte[] bytes = 
> org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
> The exception I get is
> {code:java}
> java.lang.IllegalArgumentException: Last encoded character (before the 
> paddings if any) is a valid base 64 alphabet but not a possible value  at 
> org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
>   at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
>   at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
>   at 
> com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
>   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
>   at 
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 

[jira] [Updated] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

 [ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kleinbölting updated CODEC-279:
-
Description: 
The following lines will fail for a specific String:
{code:java}
String sharedSecret = "cannot_share_the_affected_string_because_it_is_a_secret";
byte[] bytes = 
org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
The exception I get is
{code:java}
java.lang.IllegalArgumentException: Last encoded character (before the paddings 
if any) is a valid base 64 alphabet but not a possible valueat 
org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
at 
com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at 
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at 

[jira] [Updated] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

 [ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kleinbölting updated CODEC-279:
-
Affects Version/s: 1.13

> Base64.decode fails on Java11 for certain valid base 64 encoded String
> --
>
> Key: CODEC-279
> URL: https://issues.apache.org/jira/browse/CODEC-279
> Project: Commons Codec
>  Issue Type: Bug
>Affects Versions: 1.13
> Environment: Java 11
>Reporter: Christian Kleinbölting
>Priority: Major
>
> The following lines will fail for a specific String:
> {code:java}
> String sharedSecret = 
> "cannot_share_the_affected_string_because_it_is_a_secret";
> byte[] bytes = 
> org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
> The exception I get is
> {code:java}
> java.lang.IllegalArgumentException: Last encoded character (before the 
> paddings if any) is a valid base 64 alphabet but not a possible value  at 
> org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
>   at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
>   at 
> org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
>   at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
>   at 
> com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
>   at 
> org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
>   at 
> org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
>   at 
> org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
>   at 
> org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
>   at 
> org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
>   at 
> org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
>   at 
> org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
>   at 
> org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
>   at 
> 

[jira] [Created] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

Christian Kleinbölting created CODEC-279:


 Summary: Base64.decode fails on Java11 for certain valid base 64 
encoded String
 Key: CODEC-279
 URL: https://issues.apache.org/jira/browse/CODEC-279
 Project: Commons Codec
  Issue Type: Bug
 Environment: Java 11
Reporter: Christian Kleinbölting


The following lines will fail for a specific String:
{code:java}
String sharedSecret = "cannot_share_the_affected_string_because_it_is_a_secret";
byte[] bytes = 
org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
The exception I get is
{code:java}
java.lang.IllegalArgumentException: Last encoded character (before the paddings 
if any) is a valid base 64 alphabet but not a possible valueat 
org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
at 
com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at 

[jira] [Updated] (CODEC-279) Base64.decode fails on Java11 for certain valid base 64 encoded String

[Jira]

 [ 
https://issues.apache.org/jira/browse/CODEC-279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christian Kleinbölting updated CODEC-279:
-
Description: 
The following lines will fail for a specific String:
{code:java}
String sharedSecret = "cannot_share_the_affected_string_because_it_is_a_secret";
byte[] bytes = 
org.apache.commons.codec.binary.Base64.decodeBase64(sharedSecret); {code}
The exception I get is
{code:java}
java.lang.IllegalArgumentException: Last encoded character (before the paddings 
if any) is a valid base 64 alphabet but not a possible valueat 
org.apache.commons.codec.binary.Base64.validateCharacter(Base64.java:798)
at org.apache.commons.codec.binary.Base64.decode(Base64.java:472)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:412)
at 
org.apache.commons.codec.binary.BaseNCodec.decode(BaseNCodec.java:395)
at org.apache.commons.codec.binary.Base64.decodeBase64(Base64.java:694)
at 
com.rbmhtechnology.oidc.provider.spi.grants.jwt.thirdparty.rest.FooTest.foo(FooTest.java:17)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:675)
at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:125)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:132)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:124)
at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:74)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(ExecutableInvoker.java:115)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.lambda$invoke$0(ExecutableInvoker.java:105)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:104)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:62)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:43)
at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:35)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:104)
at 
org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:98)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$6(TestMethodTestDescriptor.java:202)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:198)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:135)
at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:69)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:135)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$7(NodeTestTask.java:125)
at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:135)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:123)
at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:122)
at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:80)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
at 
org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
at 

[GitHub] [commons-csv] coveralls commented on issue #55: Fixtypo performance test

[GitBox]

coveralls commented on issue #55: Fixtypo performance test
URL: https://github.com/apache/commons-csv/pull/55#issuecomment-576160158
 
 
   
   [![Coverage 
Status](https://coveralls.io/builds/28204881/badge)](https://coveralls.io/builds/28204881)
   
   Coverage remained the same at 92.928% when pulling 
**18b7fb91e5adcd9cd2d09066b1beaef7d220777a on dota17:fixtypoPerformanceTest** 
into **1d12ed9d0dc17f05fd99071ebf0bd209d08a18c8 on apache:master**.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] [commons-csv] dota17 opened a new pull request #55: Fixtypo performance test

[GitBox]

dota17 opened a new pull request #55: Fixtypo performance test
URL: https://github.com/apache/commons-csv/pull/55
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services