[jira] [Commented] (VFS-830) SFTP - moveto() throws FileSystemException: Could not set the last modified timestamp
[
https://issues.apache.org/jira/browse/VFS-830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678655#comment-17678655
]
Simon Alexander commented on VFS-830:
-
[~ggregory] as in, add a test to this folder
[commons-vfs/commons-vfs2/src/test/java/org/apache/commons/vfs2/provider/sftp
at 83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
GitHub|https://github.com/apache/commons-vfs/tree/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/test/java/org/apache/commons/vfs2/provider/sftp]
and add it to the Test Suite defined there in
[SftpProviderTestCase.java|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/test/java/org/apache/commons/vfs2/provider/sftp/SftpProviderTestCase.java]?
> SFTP - moveto() throws FileSystemException: Could not set the last modified
> timestamp
> -
>
> Key: VFS-830
> URL: https://issues.apache.org/jira/browse/VFS-830
> Project: Commons VFS
> Issue Type: Bug
>Affects Versions: 2.9.0
> Environment: RHEL Linux server connecting to AWS Transfer SFTP Server
>Reporter: Simon Alexander
>Priority: Minor
>
> I am uploading a file via a temp file, by using the following code:
>
> {code:java}
> FileSystemOptions opts = createDefaultOptions();
> BytesIdentityInfo identityInfo = new BytesIdentityInfo(sshKey.getBytes(),
> null);
> SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts,
> identityInfo);
> SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
> SftpFileSystemConfigBuilder.getInstance().setSessionTimeout(opts,
> Duration.ofMillis(1));
> SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts,
> true);
> // Create temp remote file object
> String tempFilePath = remoteFolder + FilePathSeparator + tempFileName;
> tempFileObject = remoteManager.resolveFile(new
> URI("sftp",server.getServerInfo(),server.HostName,server.Port,tempFilePath,null,null).toString(),
> opts);
> tempFileObject.copyFrom(sourceFileObject, Selectors.SELECT_SELF);
> // rename to the correct name
> tempFileObject.moveTo(remoteFileObject);} {code}
> In this code, `sourceFileObject` is on a remote linux server; and
> `tempFileObject` and `remoteFileObject` are on the AWS SFTP Transfer server.
> The exec channel is disabled on the server, so I've disabled its use here.
> When I run this code, the creation of the temp file runs successfully (using
> `copyFrom()`), but then the `moveTo()` call fails with the following
> exception:
> *java.io.IOException: copyFileBetweenServersUsingTempFile() - Could not set
> the last modified timestamp of "testRemoteFileName.txt"*
>
> I was trying to understand why the moveTo() call would fail in this way, so I
> started digging into the Apache code. As far as I can see, the call to
> `setLastModifiedTime()` only happens if the code thinks that the source and
> target filesystems are different - see [commons-vfs/AbstractFileObject.java
> at 83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
> GitHub|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/AbstractFileObject.java#L1726]
> {code:java}
> if (fileSystem == newfile.getFileSystem()) // canRenameTo()
> {
> ...
> }
> else
> {
> ...
> destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
> } {code}
> The issue, I think, is the `==` in the canRenameTo() method - because I am
> actually moving from the temp file to the final file on the same file system,
> which means this should be returning true not false, right? presumably we
> should be using `.equals()` here, and overriding equals in the appropriate
> type of `FileSystem` object?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (VFS-830) SFTP - moveto() throws FileSystemException: Could not set the last modified timestamp
[
https://issues.apache.org/jira/browse/VFS-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Alexander updated VFS-830:
Description:
I am uploading a file via a temp file, by using the following code:
{code:java}
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts,
identityInfo);
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
SftpFileSystemConfigBuilder.getInstance().setSessionTimeout(opts,
Duration.ofMillis(1));
SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts,
true);
// Create temp remote file object
String tempFilePath = remoteFolder + FilePathSeparator + tempFileName;
tempFileObject = remoteManager.resolveFile(new
URI("sftp",server.getServerInfo(),server.HostName,server.Port,tempFilePath,null,null).toString(),
opts);
tempFileObject.copyFrom(sourceFileObject, Selectors.SELECT_SELF);
// rename to the correct name
tempFileObject.moveTo(remoteFileObject);} {code}
In this code, `sourceFileObject` is on a remote linux server; and
`tempFileObject` and `remoteFileObject` are on the AWS SFTP Transfer server.
The exec channel is disabled on the server, so I've disabled its use here.
When I run this code, the creation of the temp file runs successfully (using
`copyFrom()`), but then the `moveTo()` call fails with the following exception:
*java.io.IOException: copyFileBetweenServersUsingTempFile() - Could not set the
last modified timestamp of "testRemoteFileName.txt"*
I was trying to understand why the moveTo() call would fail in this way, so I
started digging into the Apache code. As far as I can see, the call to
`setLastModifiedTime()` only happens if the code thinks that the source and
target filesystems are different - see [commons-vfs/AbstractFileObject.java at
83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
GitHub|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/AbstractFileObject.java#L1726]
{code:java}
if (fileSystem == newfile.getFileSystem()) // canRenameTo()
{
...
}
else
{
...
destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
} {code}
The issue, I think, is the `==` in the canRenameTo() method - because I am
actually moving from the temp file to the final file on the same file system,
which means this should be returning true not false, right? presumably we
should be using `.equals()` here, and overriding equals in the appropriate type
of `FileSystem` object?
was:
I am uploading a file via a temp file, by using the following code:
{code:java}
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts,
identityInfo);
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
SftpFileSystemConfigBuilder.getInstance().setSessionTimeout(opts,
Duration.ofMillis(1));
SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts,
true);
// Create temp remote file object
String tempFilePath = remoteFolder + FilePathSeparator + tempFileName;
tempFileObject = remoteManager.resolveFile(new
URI("sftp",server.getServerInfo(),server.HostName,server.Port,tempFilePath,null,null).toString(),
opts);
tempFileObject.copyFrom(sourceFileObject, Selectors.SELECT_SELF);
// rename to the correct name
tempFileObject.moveTo(remoteFileObject);} {code}
In this code, `sourceFileObject` is on a remote linux server; and
`tempFileObject` and `remoteFileObject` are on the AWS SFTP Transfer server.
When I run this code, the creation of the temp file runs successfully (using
`copyFrom()`), but then the `moveTo()` call fails with the following exception:
*java.io.IOException: copyFileBetweenServersUsingTempFile() - Could not set the
last modified timestamp of "testRemoteFileName.txt"*
I was trying to understand why the moveTo() call would fail in this way, so I
started digging into the Apache code. As far as I can see, the call to
`setLastModifiedTime()` only happens if the code thinks that the source and
target filesystems are different - see [commons-vfs/AbstractFileObject.java at
83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
GitHub|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/AbstractFileObject.java#L1726]
{code:java}
if (fileSystem == newfile.getFileSystem()) // canRenameTo()
{
...
}
else
{
...
destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
} {code}
The issue, I think, is the `==` in the
[jira] [Updated] (VFS-830) SFTP - moveto() throws FileSystemException: Could not set the last modified timestamp
[
https://issues.apache.org/jira/browse/VFS-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Alexander updated VFS-830:
Description:
I am uploading a file via a temp file, by using the following code:
{code:java}
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts,
identityInfo);
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
SftpFileSystemConfigBuilder.getInstance().setSessionTimeout(opts,
Duration.ofMillis(1));
SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts,
true);
// Create temp remote file object
String tempFilePath = remoteFolder + FilePathSeparator + tempFileName;
tempFileObject = remoteManager.resolveFile(new
URI("sftp",server.getServerInfo(),server.HostName,server.Port,tempFilePath,null,null).toString(),
opts);
tempFileObject.copyFrom(sourceFileObject, Selectors.SELECT_SELF);
// rename to the correct name
tempFileObject.moveTo(remoteFileObject);} {code}
In this code, `sourceFileObject` is on a remote linux server; and
`tempFileObject` and `remoteFileObject` are on the AWS SFTP Transfer server.
When I run this code, the creation of the temp file runs successfully (using
`copyFrom()`), but then the `moveTo()` call fails with the following exception:
*java.io.IOException: copyFileBetweenServersUsingTempFile() - Could not set the
last modified timestamp of "testRemoteFileName.txt"*
I was trying to understand why the moveTo() call would fail in this way, so I
started digging into the Apache code. As far as I can see, the call to
`setLastModifiedTime()` only happens if the code thinks that the source and
target filesystems are different - see [commons-vfs/AbstractFileObject.java at
83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
GitHub|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/AbstractFileObject.java#L1726]
{code:java}
if (fileSystem == newfile.getFileSystem()) // canRenameTo()
{
...
}
else
{
...
destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
} {code}
The issue, I think, is the `==` in the canRenameTo() method - because I am
actually moving from the temp file to the final file on the same file system,
which means this should be returning true not false, right? presumably we
should be using `.equals()` here, and overriding equals in the appropriate type
of `FileSystem` object?
was:
I am uploading a file via a temp file, by using the following code:
{code:java}
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts,
identityInfo);
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
SftpFileSystemConfigBuilder.getInstance().setSessionTimeout(opts,
Duration.ofMillis(1));
SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts,
true);
// Create temp remote file object
String tempFilePath = remoteFolder + FilePathSeparator + tempFileName;
tempFileObject = remoteManager.resolveFile(new
URI("sftp",server.getServerInfo(),server.HostName,server.Port,tempFilePath,null,null).toString(),
opts);
tempFileObject.copyFrom(sourceFileObject, Selectors.SELECT_SELF);
// rename to the correct name tempFileObject.moveTo(remoteFileObject);} {code}
In this code, `sourceFileObject` is on a remote linux server; and
`tempFileObject` and `remoteFileObject` are on the AWS SFTP Transfer server.
When I run this code, the creation of the temp file runs successfully (using
`copyFrom()`), but then the `moveTo()` call fails with the following exception:
*java.io.IOException: copyFileBetweenServersUsingTempFile() - Could not set the
last modified timestamp of "testRemoteFileName.txt"*
I was trying to understand why the moveTo() call would fail in this way, so I
started digging into the Apache code. As far as I can see, the call to
`setLastModifiedTime()` only happens if the code thinks that the source and
target filesystems are different - see [commons-vfs/AbstractFileObject.java at
83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
GitHub|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/AbstractFileObject.java#L1726]
{code:java}
if (fileSystem == newfile.getFileSystem()) // canRenameTo()
{
...
}
else
{
...
destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
} {code}
The issue, I think, is the `==` in the canRenameTo() method - because I am
actually moving from the temp file to the fi
[jira] [Created] (VFS-830) SFTP - moveto() throws FileSystemException: Could not set the last modified timestamp
Simon Alexander created VFS-830:
---
Summary: SFTP - moveto() throws FileSystemException: Could not set
the last modified timestamp
Key: VFS-830
URL: https://issues.apache.org/jira/browse/VFS-830
Project: Commons VFS
Issue Type: Bug
Affects Versions: 2.9.0
Environment: RHEL Linux server connecting to AWS Transfer SFTP Server
Reporter: Simon Alexander
I am uploading a file via a temp file, by using the following code:
{code:java}
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts,
identityInfo);
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
SftpFileSystemConfigBuilder.getInstance().setSessionTimeout(opts,
Duration.ofMillis(1));
SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts,
true);
// Create temp remote file object
String tempFilePath = remoteFolder + FilePathSeparator + tempFileName;
tempFileObject = remoteManager.resolveFile(new
URI("sftp",server.getServerInfo(),server.HostName,server.Port,tempFilePath,null,null).toString(),
opts);
tempFileObject.copyFrom(sourceFileObject, Selectors.SELECT_SELF);
// rename to the correct name tempFileObject.moveTo(remoteFileObject);} {code}
In this code, `sourceFileObject` is on a remote linux server; and
`tempFileObject` and `remoteFileObject` are on the AWS SFTP Transfer server.
When I run this code, the creation of the temp file runs successfully (using
`copyFrom()`), but then the `moveTo()` call fails with the following exception:
*java.io.IOException: copyFileBetweenServersUsingTempFile() - Could not set the
last modified timestamp of "testRemoteFileName.txt"*
I was trying to understand why the moveTo() call would fail in this way, so I
started digging into the Apache code. As far as I can see, the call to
`setLastModifiedTime()` only happens if the code thinks that the source and
target filesystems are different:
[AbstractFileObject Source|[commons-vfs/AbstractFileObject.java at
83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
GitHub|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/AbstractFileObject.java#L1726]]
{code:java}
if (fileSystem == newfile.getFileSystem()) // canRenameTo()
{
...
}
else
{
...
destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
} {code}
The issue, I think, is the `==` in the canRenameTo() method - because I am
actually moving from the temp file to the final file on the same file system,
which means this should be returning true not false, right? presumably we
should be using `.equals()` here, and overriding equals in the appropriate type
of `FileSystem` object?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (VFS-830) SFTP - moveto() throws FileSystemException: Could not set the last modified timestamp
[
https://issues.apache.org/jira/browse/VFS-830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Alexander updated VFS-830:
Description:
I am uploading a file via a temp file, by using the following code:
{code:java}
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts,
identityInfo);
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
SftpFileSystemConfigBuilder.getInstance().setSessionTimeout(opts,
Duration.ofMillis(1));
SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts,
true);
// Create temp remote file object
String tempFilePath = remoteFolder + FilePathSeparator + tempFileName;
tempFileObject = remoteManager.resolveFile(new
URI("sftp",server.getServerInfo(),server.HostName,server.Port,tempFilePath,null,null).toString(),
opts);
tempFileObject.copyFrom(sourceFileObject, Selectors.SELECT_SELF);
// rename to the correct name tempFileObject.moveTo(remoteFileObject);} {code}
In this code, `sourceFileObject` is on a remote linux server; and
`tempFileObject` and `remoteFileObject` are on the AWS SFTP Transfer server.
When I run this code, the creation of the temp file runs successfully (using
`copyFrom()`), but then the `moveTo()` call fails with the following exception:
*java.io.IOException: copyFileBetweenServersUsingTempFile() - Could not set the
last modified timestamp of "testRemoteFileName.txt"*
I was trying to understand why the moveTo() call would fail in this way, so I
started digging into the Apache code. As far as I can see, the call to
`setLastModifiedTime()` only happens if the code thinks that the source and
target filesystems are different - see [commons-vfs/AbstractFileObject.java at
83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
GitHub|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/AbstractFileObject.java#L1726]
{code:java}
if (fileSystem == newfile.getFileSystem()) // canRenameTo()
{
...
}
else
{
...
destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
} {code}
The issue, I think, is the `==` in the canRenameTo() method - because I am
actually moving from the temp file to the final file on the same file system,
which means this should be returning true not false, right? presumably we
should be using `.equals()` here, and overriding equals in the appropriate type
of `FileSystem` object?
was:
I am uploading a file via a temp file, by using the following code:
{code:java}
FileSystemOptions opts = createDefaultOptions();
BytesIdentityInfo identityInfo = new BytesIdentityInfo(sshKey.getBytes(), null);
SftpFileSystemConfigBuilder.getInstance().setIdentityProvider(opts,
identityInfo);
SftpFileSystemConfigBuilder.getInstance().setUserDirIsRoot(opts, false);
SftpFileSystemConfigBuilder.getInstance().setSessionTimeout(opts,
Duration.ofMillis(1));
SftpFileSystemConfigBuilder.getInstance().setDisableDetectExecChannel(opts,
true);
// Create temp remote file object
String tempFilePath = remoteFolder + FilePathSeparator + tempFileName;
tempFileObject = remoteManager.resolveFile(new
URI("sftp",server.getServerInfo(),server.HostName,server.Port,tempFilePath,null,null).toString(),
opts);
tempFileObject.copyFrom(sourceFileObject, Selectors.SELECT_SELF);
// rename to the correct name tempFileObject.moveTo(remoteFileObject);} {code}
In this code, `sourceFileObject` is on a remote linux server; and
`tempFileObject` and `remoteFileObject` are on the AWS SFTP Transfer server.
When I run this code, the creation of the temp file runs successfully (using
`copyFrom()`), but then the `moveTo()` call fails with the following exception:
*java.io.IOException: copyFileBetweenServersUsingTempFile() - Could not set the
last modified timestamp of "testRemoteFileName.txt"*
I was trying to understand why the moveTo() call would fail in this way, so I
started digging into the Apache code. As far as I can see, the call to
`setLastModifiedTime()` only happens if the code thinks that the source and
target filesystems are different:
[AbstractFileObject Source|[commons-vfs/AbstractFileObject.java at
83514069293cbf80644f1d47dd3eceaaf4e6954b · apache/commons-vfs ·
GitHub|https://github.com/apache/commons-vfs/blob/83514069293cbf80644f1d47dd3eceaaf4e6954b/commons-vfs2/src/main/java/org/apache/commons/vfs2/provider/AbstractFileObject.java#L1726]]
{code:java}
if (fileSystem == newfile.getFileSystem()) // canRenameTo()
{
...
}
else
{
...
destFile.getContent().setLastModifiedTime(this.getContent().getLastModifiedTime());
} {code}
The issue, I think, is the `==` in the canRenameTo() method - because I am
actually moving from the
[jira] [Commented] (CONFIGURATION-793) Error reading a list of complex objects from JSON after 2.3
[
https://issues.apache.org/jira/browse/CONFIGURATION-793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258275#comment-17258275
]
Simon Billingsley commented on CONFIGURATION-793:
-
Hello,
There is now a NIST CVE report on my version (2.2):
[https://nvd.nist.gov/vuln/detail/CVE-2020-1953]
The report recommends upgrading to at least v2.7 but I cannot do that as I am
affected by this bug.
> Error reading a list of complex objects from JSON after 2.3
> ---
>
> Key: CONFIGURATION-793
> URL: https://issues.apache.org/jira/browse/CONFIGURATION-793
> Project: Commons Configuration
> Issue Type: Bug
>Affects Versions: 2.3
> Environment: JDK 11.0.4
>Reporter: Cliff Evans
>Priority: Major
>
> We have just attempted to move from version 2.2 to 2.7 and have encountered
> and issue whilst attempting to read a List from a JSON formatted
> configuration file.
> The issue appears to have been introduced in 2.3.
> Adding the following tests to
> org.apache.commons.configuration2.TestJSONConfiguration will demonstrate the
> change. These tests will pass against 2.2 but fail agains 2.3 (and 2.7.)
> https://issues.apache.org/jira/browse/CONFIGURATION-686 appears to have fixed
> access to lists of complex objects using the dot notation for strings (see
> testGetProperty_dictionaryInList() in TestJSONConfiguration) but has broken
> access to the list of complex objects.
>
> {noformat}
> @Test
> public void testGetList_nested_with_list()
> {
> assertEquals(Arrays.asList("col1", "col2"),
> jsonConfiguration.getList(String.class, "key4.key5"));
> }
> @Test
> public void testGetList() {
> final List configList = jsonConfiguration.getList(Map.class,
> "capitals");
> assertEquals(configList.size(), 2);
> assertEquals(configList.get(0).get("country"), "USA");
> assertEquals(configList.get(0).get("capital"), "Washington");
> assertEquals(configList.get(1).get("country"), "UK");
> assertEquals(configList.get(1).get("capital"), "London");
> }
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JCS-204) Release JCS 3.0
[ https://issues.apache.org/jira/browse/JCS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17183534#comment-17183534 ] Simon Legner commented on JCS-204: -- Thanks a lot for your work on the JCS 3.0 release! Just now, I've upgraded JOSM to use the 3.0 release via https://josm.openstreetmap.de/ticket/19681 and https://josm.openstreetmap.de/changeset/16925/josm > Release JCS 3.0 > --- > > Key: JCS-204 > URL: https://issues.apache.org/jira/browse/JCS-204 > Project: Commons JCS > Issue Type: Task >Reporter: Simon Legner >Assignee: Thomas Vandahl >Priority: Major > Labels: release > Fix For: jcs-3.0 > > > Many improvements have been made since the last 2.2.1 release in 2017: > https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master > [JOSM|https://josm.openstreetmap.de/] always has been using the latest SVN > version of JCS without any known problems. We would like to switch to Apache > Ivy for dependency management and thus require a published release. > Can you craft a new 2.3 release of JCS? Are there any blockers? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JCS-205) commons-jcs3-core-3.0-20200526.132410-4-sources.jar lacks META-INF/services/org.apache.commons.jcs3.log.LogFactory
[
https://issues.apache.org/jira/browse/JCS-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17135208#comment-17135208
]
Simon Legner commented on JCS-205:
--
Great, thank you!
> commons-jcs3-core-3.0-20200526.132410-4-sources.jar lacks
> META-INF/services/org.apache.commons.jcs3.log.LogFactory
> --
>
> Key: JCS-205
> URL: https://issues.apache.org/jira/browse/JCS-205
> Project: Commons JCS
> Issue Type: Bug
>Affects Versions: jcs-3.0
>Reporter: Simon Legner
>Assignee: Thomas Vandahl
>Priority: Major
> Fix For: jcs-3.0
>
>
> The source JAR
> https://repository.apache.org/content/repositories/snapshots/org/apache/commons/commons-jcs3-core/3.0-SNAPSHOT/commons-jcs3-core-3.0-20200526.132410-4-sources.jar
> lacks the services file
> {{META-INF/services/org.apache.commons.jcs3.log.LogFactory}}. Instead, the
> file
> {{/home/jenkins/jenkins-slave/workspace/commons-jcs/commons-jcs-core/target/classes/META-INF/services/org.apache.commons.jcs3.log.LogFactory}}
> is present.
> JOSM bug: https://josm.openstreetmap.de/ticket/19335
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (JCS-205) commons-jcs3-core-3.0-20200526.132410-4-sources.jar lacks META-INF/services/org.apache.commons.jcs3.log.LogFactory
Simon Legner created JCS-205:
Summary: commons-jcs3-core-3.0-20200526.132410-4-sources.jar lacks
META-INF/services/org.apache.commons.jcs3.log.LogFactory
Key: JCS-205
URL: https://issues.apache.org/jira/browse/JCS-205
Project: Commons JCS
Issue Type: Bug
Affects Versions: jcs-3.0
Reporter: Simon Legner
The source JAR
https://repository.apache.org/content/repositories/snapshots/org/apache/commons/commons-jcs3-core/3.0-SNAPSHOT/commons-jcs3-core-3.0-20200526.132410-4-sources.jar
lacks the services file
{{META-INF/services/org.apache.commons.jcs3.log.LogFactory}}. Instead, the file
{{/home/jenkins/jenkins-slave/workspace/commons-jcs/commons-jcs-core/target/classes/META-INF/services/org.apache.commons.jcs3.log.LogFactory}}
is present.
JOSM bug: https://josm.openstreetmap.de/ticket/19335
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JCS-204) Release JCS 3.0
[
https://issues.apache.org/jira/browse/JCS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125261#comment-17125261
]
Simon Legner commented on JCS-204:
--
We've managed to complete the migration to {{package org.apache.commons.jcs3}}
tracked in https://josm.openstreetmap.de/ticket/19208. Updating JOSM core was
easy. Reaching out to 7 plugin maintainers and getting the affected plugins
updated, took more effort.
> Release JCS 3.0
> ---
>
> Key: JCS-204
> URL: https://issues.apache.org/jira/browse/JCS-204
> Project: Commons JCS
> Issue Type: Task
>Reporter: Simon Legner
>Assignee: Thomas Vandahl
>Priority: Major
> Labels: release
>
> Many improvements have been made since the last 2.2.1 release in 2017:
> https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master
> [JOSM|https://josm.openstreetmap.de/] always has been using the latest SVN
> version of JCS without any known problems. We would like to switch to Apache
> Ivy for dependency management and thus require a published release.
> Can you craft a new 2.3 release of JCS? Are there any blockers?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JCS-204) Release JCS 3.0
[ https://issues.apache.org/jira/browse/JCS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086531#comment-17086531 ] Simon Legner commented on JCS-204: -- Actually, most of the JCS code in JOSM has been written by [~wiktorn]. The switch between Block and Indexed mode has been implemented for [bug #12909|https://josm.openstreetmap.de/ticket/12909]. > Release JCS 3.0 > --- > > Key: JCS-204 > URL: https://issues.apache.org/jira/browse/JCS-204 > Project: Commons JCS > Issue Type: Task >Reporter: Simon Legner >Assignee: Thomas Vandahl >Priority: Major > Labels: release > > Many improvements have been made since the last 2.2.1 release in 2017: > https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master > [JOSM|https://josm.openstreetmap.de/] always has been using the latest SVN > version of JCS without any known problems. We would like to switch to Apache > Ivy for dependency management and thus require a published release. > Can you craft a new 2.3 release of JCS? Are there any blockers? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JCS-204) Release JCS 3.0
[ https://issues.apache.org/jira/browse/JCS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17086379#comment-17086379 ] Simon Legner commented on JCS-204: -- Thanks, done via [https://josm.openstreetmap.de/ticket/19113] > Release JCS 3.0 > --- > > Key: JCS-204 > URL: https://issues.apache.org/jira/browse/JCS-204 > Project: Commons JCS > Issue Type: Task >Reporter: Simon Legner >Assignee: Thomas Vandahl >Priority: Major > Labels: release > > Many improvements have been made since the last 2.2.1 release in 2017: > https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master > [JOSM|https://josm.openstreetmap.de/] always has been using the latest SVN > version of JCS without any known problems. We would like to switch to Apache > Ivy for dependency management and thus require a published release. > Can you craft a new 2.3 release of JCS? Are there any blockers? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JCS-204) Release JCS 3.0
[ https://issues.apache.org/jira/browse/JCS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081829#comment-17081829 ] Simon Legner commented on JCS-204: -- Very nice – the warning is gone, still works with JOSM. :) > Release JCS 3.0 > --- > > Key: JCS-204 > URL: https://issues.apache.org/jira/browse/JCS-204 > Project: Commons JCS > Issue Type: Task >Reporter: Simon Legner >Assignee: Thomas Vandahl >Priority: Major > Labels: release > > Many improvements have been made since the last 2.2.1 release in 2017: > https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master > [JOSM|https://josm.openstreetmap.de/] always has been using the latest SVN > version of JCS without any known problems. We would like to switch to Apache > Ivy for dependency management and thus require a published release. > Can you craft a new 2.3 release of JCS? Are there any blockers? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JCS-201) Tune logging: "Using hardcoded default" as info
[ https://issues.apache.org/jira/browse/JCS-201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081830#comment-17081830 ] Simon Legner commented on JCS-201: -- Thank you for looking into this issue! :) > Tune logging: "Using hardcoded default" as info > --- > > Key: JCS-201 > URL: https://issues.apache.org/jira/browse/JCS-201 > Project: Commons JCS > Issue Type: Improvement >Reporter: Simon Legner >Assignee: Thomas Vandahl >Priority: Minor > Labels: logging > Fix For: jcs-3.0 > > > https://github.com/apache/commons-jcs/pull/9 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (JCS-204) Release JCS 3.0
[
https://issues.apache.org/jira/browse/JCS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081524#comment-17081524
]
Simon Legner commented on JCS-204:
--
Hi [~tv], testing the {{jcs-122-logging-abstraction}} branch in JOSM (via
[https://josm.openstreetmap.de/browser/josm/trunk/src/org/openstreetmap/josm/data/cache/JCSCacheManager.java])
didn't reveal any problems:
{code:java}
2020-04-11 20:55:15.359 INFO: Instance is null, returning unconfigured instance
2020-04-11 20:55:15.430 WARNING: No configuration settings found. Using
hardcoded default values for all pools.
2020-04-11 20:55:15.458 INFO: Setting default auxiliaries to "null"
2020-04-11 20:55:15.466 INFO: setting defaultCompositeCacheAttributes to [
useLateral = true, useRemote = true, useDisk = true, maxObjs = 1000,
maxSpoolPerRun = -1, diskUsagePattern = UPDATE, spoolChunkSize = 2 ]
2020-04-11 20:55:15.483 INFO: setting defaultElementAttributes to [ IS_LATERAL
= true, IS_SPOOL = true, IS_REMOTE = true, IS_ETERNAL = false, MaxLifeSeconds =
-1, IdleTime = -1, CreateTime = 1586631315467, LastAccessTime = 1586631315467,
getTimeToLiveSeconds() = -1, createTime = 1586631315467 ]
2020-04-11 20:55:15.488 INFO: Parsed regions []
2020-04-11 20:55:15.489 INFO: Finished configuration in 32 ms.
2020-04-11 20:55:15.489 INFO: No special ElementAttribute class defined for key
[jcs.region.TMS.elementattributes], using default class.
2020-04-11 20:55:15.503 INFO: initialized MemoryCache for TMS
2020-04-11 20:55:15.503 INFO: Constructed cache with name [TMS] and cache
attributes [ useLateral = true, useRemote = true, useDisk = true, maxObjs = 0,
maxSpoolPerRun = -1, diskUsagePattern = UPDATE, spoolChunkSize = 2 ]
2020-04-11 20:55:15.540 INFO: Region [TMS_BLOCK_v2] : Constructing
BlockDiskCache with attributes
BlockDiskAttributes
DiskPath [/tmp/.josm/cache/tiles]
MaxKeySize [524288]
MaxPurgatorySize [5000]
BlockSizeBytes [4096]
KeyPersistenceIntervalSeconds [300]
DiskLimitType [SIZE]
2020-04-11 20:55:15.540 INFO: Region [TMS_BLOCK_v2] : Cache file root
directory: [/tmp/.josm/cache/tiles]
2020-04-11 20:55:15.541 INFO: Constructing BlockDisk, blockSizeBytes [4,096]
2020-04-11 20:55:15.543 INFO: Region [TMS_BLOCK_v2] : Cache file root directory
[/tmp/.josm/cache/tiles]
2020-04-11 20:55:15.543 INFO: Region [TMS_BLOCK_v2] : Key File
[/tmp/.josm/cache/tiles/TMS_BLOCK_v2.key]
2020-04-11 20:55:15.544 INFO: Region [TMS_BLOCK_v2] : Loading keys for
/tmp/.josm/cache/tiles/TMS_BLOCK_v2.key
2020-04-11 20:55:15.544 INFO: Region [TMS_BLOCK_v2] : Set maxKeySize to:
"524,288"
2020-04-11 20:55:15.557 INFO: Region [TMS_BLOCK_v2] : Loaded keys from
[TMS_BLOCK_v2], key count: 371; up to 524,288 will be available.
2020-04-11 20:55:15.649 INFO: Region [TMS_BLOCK_v2] : Block Disk Cache is alive.
{code}
The only flaw is that the following {{WARNING}} ist still present, see JCS-201:
{code:java}
WARNING: No configuration settings found. Using hardcoded default values for
all pools.{code}
> Release JCS 3.0
> ---
>
> Key: JCS-204
> URL: https://issues.apache.org/jira/browse/JCS-204
> Project: Commons JCS
> Issue Type: Task
>Reporter: Simon Legner
>Assignee: Thomas Vandahl
>Priority: Major
> Labels: release
>
> Many improvements have been made since the last 2.2.1 release in 2017:
> https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master
> [JOSM|https://josm.openstreetmap.de/] always has been using the latest SVN
> version of JCS without any known problems. We would like to switch to Apache
> Ivy for dependency management and thus require a published release.
> Can you craft a new 2.3 release of JCS? Are there any blockers?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (JCS-204) Release JCS 3.0
[ https://issues.apache.org/jira/browse/JCS-204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Legner updated JCS-204: - Summary: Release JCS 3.0 (was: Craft a new 2.3 release) > Release JCS 3.0 > --- > > Key: JCS-204 > URL: https://issues.apache.org/jira/browse/JCS-204 > Project: Commons JCS > Issue Type: Task > Reporter: Simon Legner >Priority: Major > Labels: release > > Many improvements have been made since the last 2.2.1 release in 2017: > https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master > [JOSM|https://josm.openstreetmap.de/] always has been using the latest SVN > version of JCS without any known problems. We would like to switch to Apache > Ivy for dependency management and thus require a published release. > Can you craft a new 2.3 release of JCS? Are there any blockers? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (JCS-204) Craft a new 2.3 release
Simon Legner created JCS-204: Summary: Craft a new 2.3 release Key: JCS-204 URL: https://issues.apache.org/jira/browse/JCS-204 Project: Commons JCS Issue Type: Task Reporter: Simon Legner Many improvements have been made since the last 2.2.1 release in 2017: https://github.com/apache/commons-jcs/compare/commons-jcs-2.2.1...master [JOSM|https://josm.openstreetmap.de/] always has been using the latest SVN version of JCS without any known problems. We would like to switch to Apache Ivy for dependency management and thus require a published release. Can you craft a new 2.3 release of JCS? Are there any blockers? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (JCS-202) Tune logging: assume that info is enabled
Simon Legner created JCS-202: Summary: Tune logging: assume that info is enabled Key: JCS-202 URL: https://issues.apache.org/jira/browse/JCS-202 Project: Commons JCS Issue Type: Improvement Reporter: Simon Legner https://github.com/apache/commons-jcs/pull/10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (JCS-200) Tune logging: "Resetting cache" as info
Simon Legner created JCS-200: Summary: Tune logging: "Resetting cache" as info Key: JCS-200 URL: https://issues.apache.org/jira/browse/JCS-200 Project: Commons JCS Issue Type: Improvement Reporter: Simon Legner https://github.com/apache/commons-jcs/pull/8 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (JCS-201) Tune logging: "Using hardcoded default" as info
Simon Legner created JCS-201: Summary: Tune logging: "Using hardcoded default" as info Key: JCS-201 URL: https://issues.apache.org/jira/browse/JCS-201 Project: Commons JCS Issue Type: Improvement Reporter: Simon Legner https://github.com/apache/commons-jcs/pull/9 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEXT-109) Implement or document how to use edit distances that consider the keyboard layout
[
https://issues.apache.org/jira/browse/TEXT-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984654#comment-16984654
]
Simon poortman commented on TEXT-109:
-
{code:java}
// Some comments here
public String getFoo()
{
return foo;
}
{code}
> Implement or document how to use edit distances that consider the keyboard
> layout
> -
>
> Key: TEXT-109
> URL: https://issues.apache.org/jira/browse/TEXT-109
> Project: Commons Text
> Issue Type: New Feature
>Reporter: Bruno P. Kinoshita
>Priority: Minor
> Labels: discussion, edit-distance, help-wanted
> Attachments: 328DADB9-2465-45E4-B36C-953BFF7C2B9F.jpeg,
> 63B28BEB-F040-46D9-8997-3E09DA2C94C5.jpeg,
> 8A14350C-87F1-4A58-8AF0-4E1742ED8D64.jpeg
>
>
> Most edit distances take into consideration number of "changes" required in
> one string to match with another string. And they give you a value that
> represent the distance between the words.
> While it is helpful, when working with datasets and corpora that have been
> created with keyboards (e.g. SMS, e-mail, transcripts) it is common to have
> mistakes. In some cases a letter was accidentally mistyped. But the character
> used is normally close to the correct character.
> For example, given the word "one", and two incorrect misspellings "onr" and
> "oni". The Levenshtein distance for both would be 1. But if you are aware
> that the keyboard layout is English with the QUERTY layout (notice the E and
> the R), so the distance between "one" and "onr", would be greater than the
> distance between "one" and "oni", because in the English keyboard the letter
> 'E' is neighbouring 'R'. Whereas 'I' is not even covered by the left hand,
> but by the right hand.
> Here's some reference links for further research.
> * https://findsomethingnewtoday.wordpress.com/2013/07/20/986/
> * https://www.joyofdata.de/blog/comparison-of-string-distance-algorithms/
> * http://www.nada.kth.se/~ann/exjobb/axel_samuelsson.pdf
> * https://github.com/wsong/Typo-Distance
> *
> https://stackoverflow.com/questions/29233888/edit-distance-such-as-levenshtein-taking-into-account-proximity-on-keyboard
> Ideally such edit distance would be extensible to support other keyboard
> layouts.
> There is some indication that perhaps an existing edit distance like
> levenshtein could be extended to take into consideration the keyboard layout.
> So perhaps a new edit distance is not entirely necessary.
> We could come with the the decision that it is too hard to implement, and it
> would be better done in a spell checker, or that it would require some
> statistics and would be out of the scope of Text. Or we could simply add
> documentation on how to do it, without adding any code. Or, perhaps we add a
> new edit distance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (TEXT-109) Implement or document how to use edit distances that consider the keyboard layout
[ https://issues.apache.org/jira/browse/TEXT-109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon poortman updated TEXT-109: Attachment: 328DADB9-2465-45E4-B36C-953BFF7C2B9F.jpeg 8A14350C-87F1-4A58-8AF0-4E1742ED8D64.jpeg 63B28BEB-F040-46D9-8997-3E09DA2C94C5.jpeg > Implement or document how to use edit distances that consider the keyboard > layout > - > > Key: TEXT-109 > URL: https://issues.apache.org/jira/browse/TEXT-109 > Project: Commons Text > Issue Type: New Feature >Reporter: Bruno P. Kinoshita >Priority: Minor > Labels: discussion, edit-distance, help-wanted > Attachments: 328DADB9-2465-45E4-B36C-953BFF7C2B9F.jpeg, > 63B28BEB-F040-46D9-8997-3E09DA2C94C5.jpeg, > 8A14350C-87F1-4A58-8AF0-4E1742ED8D64.jpeg > > > Most edit distances take into consideration number of "changes" required in > one string to match with another string. And they give you a value that > represent the distance between the words. > While it is helpful, when working with datasets and corpora that have been > created with keyboards (e.g. SMS, e-mail, transcripts) it is common to have > mistakes. In some cases a letter was accidentally mistyped. But the character > used is normally close to the correct character. > For example, given the word "one", and two incorrect misspellings "onr" and > "oni". The Levenshtein distance for both would be 1. But if you are aware > that the keyboard layout is English with the QUERTY layout (notice the E and > the R), so the distance between "one" and "onr", would be greater than the > distance between "one" and "oni", because in the English keyboard the letter > 'E' is neighbouring 'R'. Whereas 'I' is not even covered by the left hand, > but by the right hand. > Here's some reference links for further research. > * https://findsomethingnewtoday.wordpress.com/2013/07/20/986/ > * https://www.joyofdata.de/blog/comparison-of-string-distance-algorithms/ > * http://www.nada.kth.se/~ann/exjobb/axel_samuelsson.pdf > * https://github.com/wsong/Typo-Distance > * > https://stackoverflow.com/questions/29233888/edit-distance-such-as-levenshtein-taking-into-account-proximity-on-keyboard > Ideally such edit distance would be extensible to support other keyboard > layouts. > There is some indication that perhaps an existing edit distance like > levenshtein could be extended to take into consideration the keyboard layout. > So perhaps a new edit distance is not entirely necessary. > We could come with the the decision that it is too hard to implement, and it > would be better done in a spell checker, or that it would require some > statistics and would be out of the scope of Text. Or we could simply add > documentation on how to do it, without adding any code. Or, perhaps we add a > new edit distance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TEXT-109) Implement or document how to use edit distances that consider the keyboard layout
[ https://issues.apache.org/jira/browse/TEXT-109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16984652#comment-16984652 ] Simon poortman commented on TEXT-109: - Begin war in eu union benelux come for 1 country nederland België luxenburg = 1 bundesland > Implement or document how to use edit distances that consider the keyboard > layout > - > > Key: TEXT-109 > URL: https://issues.apache.org/jira/browse/TEXT-109 > Project: Commons Text > Issue Type: New Feature >Reporter: Bruno P. Kinoshita >Priority: Minor > Labels: discussion, edit-distance, help-wanted > > Most edit distances take into consideration number of "changes" required in > one string to match with another string. And they give you a value that > represent the distance between the words. > While it is helpful, when working with datasets and corpora that have been > created with keyboards (e.g. SMS, e-mail, transcripts) it is common to have > mistakes. In some cases a letter was accidentally mistyped. But the character > used is normally close to the correct character. > For example, given the word "one", and two incorrect misspellings "onr" and > "oni". The Levenshtein distance for both would be 1. But if you are aware > that the keyboard layout is English with the QUERTY layout (notice the E and > the R), so the distance between "one" and "onr", would be greater than the > distance between "one" and "oni", because in the English keyboard the letter > 'E' is neighbouring 'R'. Whereas 'I' is not even covered by the left hand, > but by the right hand. > Here's some reference links for further research. > * https://findsomethingnewtoday.wordpress.com/2013/07/20/986/ > * https://www.joyofdata.de/blog/comparison-of-string-distance-algorithms/ > * http://www.nada.kth.se/~ann/exjobb/axel_samuelsson.pdf > * https://github.com/wsong/Typo-Distance > * > https://stackoverflow.com/questions/29233888/edit-distance-such-as-levenshtein-taking-into-account-proximity-on-keyboard > Ideally such edit distance would be extensible to support other keyboard > layouts. > There is some indication that perhaps an existing edit distance like > levenshtein could be extended to take into consideration the keyboard layout. > So perhaps a new edit distance is not entirely necessary. > We could come with the the decision that it is too hard to implement, and it > would be better done in a spell checker, or that it would require some > statistics and would be out of the scope of Text. Or we could simply add > documentation on how to do it, without adding any code. Or, perhaps we add a > new edit distance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (DAEMON-408) PROCRUN 1.2.x x64 crash on Windows Server 2008 R2
[ https://issues.apache.org/jira/browse/DAEMON-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Wimmesberger updated DAEMON-408: -- Description: Since procrun version 1.2.0 the application crashes when executed on Windows Server 2008 R2: Problemsignatur: Problem Event Name: BEX64 Application Name: procrun.exe Application Version: 1.2.1.0 Application Timestamp: 5d6fdd78 Fault Module Name: StackHash_25f1 Fault Module Version: 0.0.0.0 Fault Module Timestamp: Exception Offset: 004f8aa2 Exception Code: c005 Exception Data: 0008 OS Version: 6.1.7600.2.0.0.1296.17 Additional Information 1: 25f1 Additional Information 2: 25f1bf9a72a93f5658ee9429e86c7409 Additional Information 3: f2ca Additional Information 4: f2ca14b6934ede5b7e664f76bc87ba68 I also have a crash dump available but for security/sensitive reasons I will only provide it privately when necessary. With version 1.1.0 I can't reproduce the issue. was: Since procrun version 1.2.0 the application crashes when executed on Windows Server 2008 R2: Problemsignatur: Error name: BEX64 Application name: procrun.exe Application version: 1.2.1.0 Application timestamp: 5d6fdd78 Error module name: StackHash_25f1 Error module version: 0.0.0.0 Error module timestamp: Error offset: 004f8aa2 Error code: c005 Error data: 0008 OS Version: 6.1.7600.2.0.0.1296.17 Additional info 1: 25f1 Additional info 2: 25f1bf9a72a93f5658ee9429e86c7409 Additional info 3: f2ca Additional info 4: f2ca14b6934ede5b7e664f76bc87ba68 I also have a crash dump available but for security/sensitive reasons I will only provide it privately when necessary. With version 1.1.0 I can't reproduce the issue. > PROCRUN 1.2.x x64 crash on Windows Server 2008 R2 > - > > Key: DAEMON-408 > URL: https://issues.apache.org/jira/browse/DAEMON-408 > Project: Commons Daemon > Issue Type: Bug > Components: Procrun >Affects Versions: 1.2.0, 1.2.1 >Reporter: Simon Wimmesberger >Priority: Major > > Since procrun version 1.2.0 the application crashes when executed on Windows > Server 2008 R2: > Problemsignatur: > Problem Event Name: BEX64 > Application Name: procrun.exe > Application Version: 1.2.1.0 > Application Timestamp: 5d6fdd78 > Fault Module Name: StackHash_25f1 > Fault Module Version: 0.0.0.0 > Fault Module Timestamp: > Exception Offset: 004f8aa2 > Exception Code: c005 > Exception Data: 0008 > OS Version: 6.1.7600.2.0.0.1296.17 > Additional Information 1: 25f1 > Additional Information 2: 25f1bf9a72a93f5658ee9429e86c7409 > Additional Information 3: f2ca > Additional Information 4: f2ca14b6934ede5b7e664f76bc87ba68 > > I also have a crash dump available but for security/sensitive reasons I will > only provide it privately when necessary. > > With version 1.1.0 I can't reproduce the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (DAEMON-408) PROCRUN 1.2.x x64 crash on Windows Server 2008 R2
Simon Wimmesberger created DAEMON-408: - Summary: PROCRUN 1.2.x x64 crash on Windows Server 2008 R2 Key: DAEMON-408 URL: https://issues.apache.org/jira/browse/DAEMON-408 Project: Commons Daemon Issue Type: Bug Components: Procrun Affects Versions: 1.2.0, 1.2.1 Reporter: Simon Wimmesberger Since procrun version 1.2.0 the application crashes when executed on Windows Server 2008 R2: Problemsignatur: Error name: BEX64 Application name: procrun.exe Application version: 1.2.1.0 Application timestamp: 5d6fdd78 Error module name: StackHash_25f1 Error module version: 0.0.0.0 Error module timestamp: Error offset: 004f8aa2 Error code: c005 Error data: 0008 OS Version: 6.1.7600.2.0.0.1296.17 Additional info 1: 25f1 Additional info 2: 25f1bf9a72a93f5658ee9429e86c7409 Additional info 3: f2ca Additional info 4: f2ca14b6934ede5b7e664f76bc87ba68 I also have a crash dump available but for security/sensitive reasons I will only provide it privately when necessary. With version 1.1.0 I can't reproduce the issue. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TEXT-160) Of server
Simon poortman created TEXT-160: --- Summary: Of server Key: TEXT-160 URL: https://issues.apache.org/jira/browse/TEXT-160 Project: Commons Text Issue Type: Bug Reporter: Simon poortman Fix For: 2.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (TEXT-159) Delete of server
Simon poortman created TEXT-159: --- Summary: Delete of server Key: TEXT-159 URL: https://issues.apache.org/jira/browse/TEXT-159 Project: Commons Text Issue Type: Bug Reporter: Simon poortman Fix For: 1.0-beta-1 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (LANG-1432) FastDateFormat doesn't parse time string with timezone correctly
[
https://issues.apache.org/jira/browse/LANG-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16795676#comment-16795676
]
Simon Xianyu commented on LANG-1432:
Yes, I'm in China and the default timezone is Asia/Shanghai.
I think " choose the most likely interpretation if the parser's timezone is in
the same region" is better
> FastDateFormat doesn't parse time string with timezone correctly
>
>
> Key: LANG-1432
> URL: https://issues.apache.org/jira/browse/LANG-1432
> Project: Commons Lang
> Issue Type: Bug
> Components: lang.time.*
>Affects Versions: 3.8.1
> Environment: jdk 1.8
>Reporter: Simon Xianyu
>Priority: Major
>
> FastDateFormat doesn't parse the pattern correctly :
> "EEE MMM dd HH:mm:ss zzz "
> The test code is as following:
>
> {code:java}
> // code placeholder
> public class TestFastDateFormat {
> public static void main(String[] args) throws ParseException {
> Date d = new Date();
> String dstr = d.toString();
> System.out.println(dstr);
> String pattern = "EEE MMM dd HH:mm:ss zzz ";
> FastDateFormat fdf = FastDateFormat.getInstance(pattern,
> Locale.US);
> Date d1 = fdf.parse(dstr);
> System.out.println(d1);
> // Use SimpleDateFormat to parse string
> SimpleDateFormat sdf = new SimpleDateFormat(pattern);
> Date d2 = sdf.parse(dstr);
> System.out.println(d2);
> }
> }
> {code}
> The test result example:
> Tue Feb 12 16:07:53 CST 2019 /// this is origin result
> Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
> Tue Feb 12 16:07:53 CST 2019 /// this is result of
> SimpleDateFormat.parse()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (LANG-1432) FastDateFormat doesn't parse time string with timezone correctly
[
https://issues.apache.org/jira/browse/LANG-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Xianyu updated LANG-1432:
---
Description:
FastDateFormat doesn't parse the pattern correctly :
"EEE MMM dd HH:mm:ss zzz "
The test code is as following:
{code:java}
// code placeholder
public class TestFastDateFormat {
public static void main(String[] args) throws ParseException {
Date d = new Date();
String dstr = d.toString();
System.out.println(dstr);
String pattern = "EEE MMM dd HH:mm:ss zzz ";
FastDateFormat fdf = FastDateFormat.getInstance(pattern,
Locale.US);
Date d1 = fdf.parse(dstr);
System.out.println(d1);
// Use SimpleDateFormat to parse string
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
Date d2 = sdf.parse(dstr);
System.out.println(d2);
}
}
{code}
The test result example:
Tue Feb 12 16:07:53 CST 2019 /// this is origin result
Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
Tue Feb 12 16:07:53 CST 2019 /// this is result of SimpleDateFormat.parse()
was:
FastDateFormat doesn't parse the pattern correctly :
"EEE MMM dd HH:mm:ss zzz "
The test code is as following:
public class TestFastDateFormat {
public static void main(String[] args) throws ParseException {
Date d = new Date();
String dstr = d.toString();
System.out.println(dstr);
String pattern = "EEE MMM dd HH:mm:ss zzz ";
FastDateFormat fdf = FastDateFormat.getInstance(pattern,
Locale.US);
Date d1 = fdf.parse(dstr);
System.out.println(d1);
// Use SimpleDateFormat to parse string
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
Date d2 = sdf.parse(dstr);
System.out.println(d2);
}
}
The test result example:
Tue Feb 12 16:07:53 CST 2019 /// this is origin result
Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
Tue Feb 12 16:07:53 CST 2019 /// this is result of SimpleDateFormat.parse()
> FastDateFormat doesn't parse time string with timezone correctly
>
>
> Key: LANG-1432
> URL: https://issues.apache.org/jira/browse/LANG-1432
> Project: Commons Lang
> Issue Type: Bug
> Components: lang.time.*
>Affects Versions: 3.8.1
> Environment: jdk 1.8
>Reporter: Simon Xianyu
>Priority: Major
>
> FastDateFormat doesn't parse the pattern correctly :
> "EEE MMM dd HH:mm:ss zzz "
> The test code is as following:
>
>
> {code:java}
> // code placeholder
> public class TestFastDateFormat {
> public static void main(String[] args) throws ParseException {
> Date d = new Date();
> String dstr = d.toString();
> System.out.println(dstr);
> String pattern = "EEE MMM dd HH:mm:ss zzz ";
> FastDateFormat fdf = FastDateFormat.getInstance(pattern,
> Locale.US);
> Date d1 = fdf.parse(dstr);
> System.out.println(d1);
> // Use SimpleDateFormat to parse string
> SimpleDateFormat sdf = new SimpleDateFormat(pattern);
> Date d2 = sdf.parse(dstr);
> System.out.println(d2);
> }
> }
> {code}
> The test result example:
> Tue Feb 12 16:07:53 CST 2019 /// this is origin result
> Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
> Tue Feb 12 16:07:53 CST 2019 /// this is result of
> SimpleDateFormat.parse()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (LANG-1432) FastDateFormat doesn't parse time string with timezone correctly
[
https://issues.apache.org/jira/browse/LANG-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Xianyu updated LANG-1432:
---
Description:
FastDateFormat doesn't parse the pattern correctly :
"EEE MMM dd HH:mm:ss zzz "
The test code is as following:
{code:java}
// code placeholder
public class TestFastDateFormat {
public static void main(String[] args) throws ParseException {
Date d = new Date();
String dstr = d.toString();
System.out.println(dstr);
String pattern = "EEE MMM dd HH:mm:ss zzz ";
FastDateFormat fdf = FastDateFormat.getInstance(pattern,
Locale.US);
Date d1 = fdf.parse(dstr);
System.out.println(d1);
// Use SimpleDateFormat to parse string
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
Date d2 = sdf.parse(dstr);
System.out.println(d2);
}
}
{code}
The test result example:
Tue Feb 12 16:07:53 CST 2019 /// this is origin result
Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
Tue Feb 12 16:07:53 CST 2019 /// this is result of SimpleDateFormat.parse()
was:
FastDateFormat doesn't parse the pattern correctly :
"EEE MMM dd HH:mm:ss zzz "
The test code is as following:
{code:java}
// code placeholder
public class TestFastDateFormat {
public static void main(String[] args) throws ParseException {
Date d = new Date();
String dstr = d.toString();
System.out.println(dstr);
String pattern = "EEE MMM dd HH:mm:ss zzz ";
FastDateFormat fdf = FastDateFormat.getInstance(pattern,
Locale.US);
Date d1 = fdf.parse(dstr);
System.out.println(d1);
// Use SimpleDateFormat to parse string
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
Date d2 = sdf.parse(dstr);
System.out.println(d2);
}
}
{code}
The test result example:
Tue Feb 12 16:07:53 CST 2019 /// this is origin result
Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
Tue Feb 12 16:07:53 CST 2019 /// this is result of SimpleDateFormat.parse()
> FastDateFormat doesn't parse time string with timezone correctly
>
>
> Key: LANG-1432
> URL: https://issues.apache.org/jira/browse/LANG-1432
> Project: Commons Lang
> Issue Type: Bug
> Components: lang.time.*
>Affects Versions: 3.8.1
> Environment: jdk 1.8
>Reporter: Simon Xianyu
>Priority: Major
>
> FastDateFormat doesn't parse the pattern correctly :
> "EEE MMM dd HH:mm:ss zzz "
> The test code is as following:
>
> {code:java}
> // code placeholder
> public class TestFastDateFormat {
> public static void main(String[] args) throws ParseException {
> Date d = new Date();
> String dstr = d.toString();
> System.out.println(dstr);
> String pattern = "EEE MMM dd HH:mm:ss zzz ";
> FastDateFormat fdf = FastDateFormat.getInstance(pattern,
> Locale.US);
> Date d1 = fdf.parse(dstr);
> System.out.println(d1);
> // Use SimpleDateFormat to parse string
> SimpleDateFormat sdf = new SimpleDateFormat(pattern);
> Date d2 = sdf.parse(dstr);
> System.out.println(d2);
> }
> }
> {code}
> The test result example:
> Tue Feb 12 16:07:53 CST 2019 /// this is origin result
> Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
> Tue Feb 12 16:07:53 CST 2019 /// this is result of
> SimpleDateFormat.parse()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (LANG-1432) FastDateFormat doesn't parse time string with timezone correctly
[
https://issues.apache.org/jira/browse/LANG-1432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Xianyu updated LANG-1432:
---
Summary: FastDateFormat doesn't parse time string with timezone correctly
(was: FastDateFormat doesn't parse time string with timezone)
> FastDateFormat doesn't parse time string with timezone correctly
>
>
> Key: LANG-1432
> URL: https://issues.apache.org/jira/browse/LANG-1432
> Project: Commons Lang
> Issue Type: Bug
> Components: lang.time.*
>Affects Versions: 3.8.1
> Environment: jdk 1.8
>Reporter: Simon Xianyu
>Priority: Major
>
> FastDateFormat doesn't parse the pattern correctly :
> "EEE MMM dd HH:mm:ss zzz "
> The test code is as following:
> public class TestFastDateFormat {
> public static void main(String[] args) throws ParseException {
> Date d = new Date();
> String dstr = d.toString();
> System.out.println(dstr);
> String pattern = "EEE MMM dd HH:mm:ss zzz ";
> FastDateFormat fdf = FastDateFormat.getInstance(pattern,
> Locale.US);
> Date d1 = fdf.parse(dstr);
> System.out.println(d1);
> // Use SimpleDateFormat to parse string
> SimpleDateFormat sdf = new SimpleDateFormat(pattern);
> Date d2 = sdf.parse(dstr);
> System.out.println(d2);
> }
> }
>
> The test result example:
> Tue Feb 12 16:07:53 CST 2019 /// this is origin result
> Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
> Tue Feb 12 16:07:53 CST 2019 /// this is result of SimpleDateFormat.parse()
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (LANG-1432) FastDateFormat doesn't parse time string with timezone
Simon Xianyu created LANG-1432:
--
Summary: FastDateFormat doesn't parse time string with timezone
Key: LANG-1432
URL: https://issues.apache.org/jira/browse/LANG-1432
Project: Commons Lang
Issue Type: Bug
Components: lang.time.*
Affects Versions: 3.8.1
Environment: jdk 1.8
Reporter: Simon Xianyu
FastDateFormat doesn't parse the pattern correctly :
"EEE MMM dd HH:mm:ss zzz "
The test code is as following:
public class TestFastDateFormat {
public static void main(String[] args) throws ParseException {
Date d = new Date();
String dstr = d.toString();
System.out.println(dstr);
String pattern = "EEE MMM dd HH:mm:ss zzz ";
FastDateFormat fdf = FastDateFormat.getInstance(pattern,
Locale.US);
Date d1 = fdf.parse(dstr);
System.out.println(d1);
// Use SimpleDateFormat to parse string
SimpleDateFormat sdf = new SimpleDateFormat(pattern);
Date d2 = sdf.parse(dstr);
System.out.println(d2);
}
}
The test result example:
Tue Feb 12 16:07:53 CST 2019 /// this is origin result
Wed Feb 13 06:07:53 CST 2019 /// this is result of FastDateTime.parse()
Tue Feb 12 16:07:53 CST 2019 /// this is result of SimpleDateFormat.parse()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (IO-577) Add readers to filter out given characters: CharacterSetFilterReader and CharacterFilterReader.
[ https://issues.apache.org/jira/browse/IO-577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504882#comment-16504882 ] Simon Spero commented on IO-577: A few comments : 1. apis like java.lang.stream and rxjava have filter methods that work in the opposite sense to the filter method introduced here - they select items that match the test, rather than excluding them. 2. The documentation refers to "codepoints"; however, the read method in java.io.FilterReader returns UTF-16 characters. This makes a difference for characters that aren't in the BMP, and which are represented in Java as surrogate pairs. The current implementation can't filter codepoints like 😭 (U+1F62D) because it only sees the UTF-16 surrogates. Working with codepoints would potentially require interposing a pushback reader to handle the case where the input contains a codepoint encoded in more than one char, which is not rejected. 3. commons IO is currently using Java 7. If the source level were to change to Java 8 then the filter method could be replaced by an IntPredicate / Predicate (passed in when the class is constructed). The current cases could be handled using a method reference. / Predicate.isEquals. > Add readers to filter out given characters: CharacterSetFilterReader and > CharacterFilterReader. > --- > > Key: IO-577 > URL: https://issues.apache.org/jira/browse/IO-577 > Project: Commons IO > Issue Type: New Feature > Components: Filters >Reporter: Gary Gregory >Assignee: Gary Gregory >Priority: Major > Fix For: 2.7 > > Attachments: commons-io-577.patch > > > Add readers to filter out given characters, handy to remove known junk > characters from CSV files for example. Please see attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (VALIDATOR-438) IBANValidator fails for El Salvador
Simon Marti created VALIDATOR-438:
-
Summary: IBANValidator fails for El Salvador
Key: VALIDATOR-438
URL: https://issues.apache.org/jira/browse/VALIDATOR-438
Project: Commons Validator
Issue Type: Bug
Affects Versions: 1.6
Reporter: Simon Marti
The {{IBANValidator}} does not validate the IBANs from El Salvador, even though
they are part of the IBAN registry.
E.g. {{SV62CENR00700025}} fails but should pass.
See https://www.iban.com/structure.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CODEC-245) RegEx for verifying salts in ShaCrypt is incorrect
[
https://issues.apache.org/jira/browse/CODEC-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16369090#comment-16369090
]
Simon Levermann commented on CODEC-245:
---
Is there interest in a PR that aligns the behaviour to be more similar to the
original crypt function, or should this behaviour (since honestly, I found it
quite unexpected) not be ported to this library?
> RegEx for verifying salts in ShaCrypt is incorrect
> --
>
> Key: CODEC-245
> URL: https://issues.apache.org/jira/browse/CODEC-245
> Project: Commons Codec
> Issue Type: Bug
>Affects Versions: 1.11
>Reporter: Simon Levermann
>Priority: Major
>
> The regex/code that extracts the salt from a given salt string in Sha2Crypt
> treats some invalid salt formats as valid:
> {code:java}
> ^\$([56])\$(rounds=(\d+)\$)?([\.\/a-zA-Z0-9]{1,16}).*
> {code}
> The code then goes on to use capture group 3 (the round count) to determine
> how many rounds are used, and capture group 4 (the actual salt) to use as
> salt data.
> However, for an input that contains an invalid salt specification like this:
> {code:java}
> $5$notrounds=1000$asdfasdf
> {code}
> This string is treated as valid. The operation then uses "notrounds" as the
> salt for hashing:
> {code:java}
> System.out.println(Sha2Crypt.sha256Crypt(new byte[100],
> "$5$notrounds=1000$asdfasdf"));
> {code}
> The above code prints
> {code:java}
> $5$notrounds$aAEx6EzUBfc.aIIeiItVjUREbj/ar4xlb/qcMxD90NA
> {code}
> This code should probably throw an exception. Additionally, other invalid
> salt strings like
>
> {code:java}
> $6$rounds=1000$äöüäöü
> {code}
> Result in hashes like:
> {code:java}
> $5$rounds$ZIRdMgi25kK5Zmi1whKjcnFN/Pe8QCAOlTmxMfwAoq5
> {code}
> Completely ignoring the rounds parameter, and using the literal string
> "rounds" as the salt for hashing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CODEC-245) RegEx for verifying salts in ShaCrypt is incorrect
[
https://issues.apache.org/jira/browse/CODEC-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361183#comment-16361183
]
Simon Levermann commented on CODEC-245:
---
Though I suppose it's still *somewhat* incorrect: Apparently, *all bytes* are
valid salts, not just base64 characters. For example, invoking the libc crypt
function like this:
{code:java}
char *foo = crypt("asdfasdf", "$6$mounds=1000$asdf$");
{code}
results in the following hash:
{code:java}
$6$mounds=1000$A9qEzceIj1cqGxb4w8WdBvHHotUYOmx.xnedWJRwQQJCZCHKpSHX4qwcY9DIvXN.ggWMhKRW4pE93oGgPOOBy0
{code}
That is: "mounds=1000" is the literal salt here, and the actually supplied
salt, "asdf" is discarded. Apparently the specification does NOT mandate that
salts must be B64-encoded. It seems like any bytes (even non-printable
characters) are valid for salts, but only ever the first 16 bytes are used.
Thoughts on how to continue here?
> RegEx for verifying salts in ShaCrypt is incorrect
> --
>
> Key: CODEC-245
> URL: https://issues.apache.org/jira/browse/CODEC-245
> Project: Commons Codec
> Issue Type: Bug
>Affects Versions: 1.11
>Reporter: Simon Levermann
>Priority: Major
>
> The regex/code that extracts the salt from a given salt string in Sha2Crypt
> treats some invalid salt formats as valid:
> {code:java}
> ^\$([56])\$(rounds=(\d+)\$)?([\.\/a-zA-Z0-9]{1,16}).*
> {code}
> The code then goes on to use capture group 3 (the round count) to determine
> how many rounds are used, and capture group 4 (the actual salt) to use as
> salt data.
> However, for an input that contains an invalid salt specification like this:
> {code:java}
> $5$notrounds=1000$asdfasdf
> {code}
> This string is treated as valid. The operation then uses "notrounds" as the
> salt for hashing:
> {code:java}
> System.out.println(Sha2Crypt.sha256Crypt(new byte[100],
> "$5$notrounds=1000$asdfasdf"));
> {code}
> The above code prints
> {code:java}
> $5$notrounds$aAEx6EzUBfc.aIIeiItVjUREbj/ar4xlb/qcMxD90NA
> {code}
> This code should probably throw an exception. Additionally, other invalid
> salt strings like
>
> {code:java}
> $6$rounds=1000$äöüäöü
> {code}
> Result in hashes like:
> {code:java}
> $5$rounds$ZIRdMgi25kK5Zmi1whKjcnFN/Pe8QCAOlTmxMfwAoq5
> {code}
> Completely ignoring the rounds parameter, and using the literal string
> "rounds" as the salt for hashing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CODEC-245) RegEx for verifying salts in ShaCrypt is incorrect
[
https://issues.apache.org/jira/browse/CODEC-245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16361180#comment-16361180
]
Simon Levermann commented on CODEC-245:
---
Well it turns out it was I who was wrong. The spec for shacrypt was apparently
written with the following thought in mind: "Invalid inputs do not exist. Every
input is valid *no matter what*". So, apparently, this behaviour, despite being
HIGHLY confusing and obviously insane, is correct. Not quite sure what to do
about this.
> RegEx for verifying salts in ShaCrypt is incorrect
> --
>
> Key: CODEC-245
> URL: https://issues.apache.org/jira/browse/CODEC-245
> Project: Commons Codec
> Issue Type: Bug
>Affects Versions: 1.11
>Reporter: Simon Levermann
>Priority: Major
>
> The regex/code that extracts the salt from a given salt string in Sha2Crypt
> treats some invalid salt formats as valid:
> {code:java}
> ^\$([56])\$(rounds=(\d+)\$)?([\.\/a-zA-Z0-9]{1,16}).*
> {code}
> The code then goes on to use capture group 3 (the round count) to determine
> how many rounds are used, and capture group 4 (the actual salt) to use as
> salt data.
> However, for an input that contains an invalid salt specification like this:
> {code:java}
> $5$notrounds=1000$asdfasdf
> {code}
> This string is treated as valid. The operation then uses "notrounds" as the
> salt for hashing:
> {code:java}
> System.out.println(Sha2Crypt.sha256Crypt(new byte[100],
> "$5$notrounds=1000$asdfasdf"));
> {code}
> The above code prints
> {code:java}
> $5$notrounds$aAEx6EzUBfc.aIIeiItVjUREbj/ar4xlb/qcMxD90NA
> {code}
> This code should probably throw an exception. Additionally, other invalid
> salt strings like
>
> {code:java}
> $6$rounds=1000$äöüäöü
> {code}
> Result in hashes like:
> {code:java}
> $5$rounds$ZIRdMgi25kK5Zmi1whKjcnFN/Pe8QCAOlTmxMfwAoq5
> {code}
> Completely ignoring the rounds parameter, and using the literal string
> "rounds" as the salt for hashing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (CODEC-245) RegEx for verifying salts in ShaCrypt is incorrect
[
https://issues.apache.org/jira/browse/CODEC-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Levermann updated CODEC-245:
--
Description:
The regex/code that extracts the salt from a given salt string in Sha2Crypt
treats some invalid salt formats as valid:
{code:java}
^\$([56])\$(rounds=(\d+)\$)?([\.\/a-zA-Z0-9]{1,16}).*
{code}
The code then goes on to use capture group 3 (the round count) to determine how
many rounds are used, and capture group 4 (the actual salt) to use as salt data.
However, for an input that contains an invalid salt specification like this:
{code:java}
$5$notrounds=1000$asdfasdf
{code}
This string is treated as valid. The operation then uses "notrounds" as the
salt for hashing:
{code:java}
System.out.println(Sha2Crypt.sha256Crypt(new byte[100],
"$5$notrounds=1000$asdfasdf"));
{code}
The above code prints
{code:java}
$5$notrounds$aAEx6EzUBfc.aIIeiItVjUREbj/ar4xlb/qcMxD90NA
{code}
This code should probably throw an exception. Additionally, other invalid salt
strings like
{code:java}
$6$rounds=1000$äöüäöü
{code}
Result in hashes like:
{code:java}
$5$rounds$ZIRdMgi25kK5Zmi1whKjcnFN/Pe8QCAOlTmxMfwAoq5
{code}
Completely ignoring the rounds parameter, and using the literal string "rounds"
as the salt for hashing.
was:
The regex/code that extracts the salt from a given salt string in Sha2Crypt
treats some invalid salt formats as valid:
{code:java}
^\$([56])\$(rounds=(\d+)\$)?([\.\/a-zA-Z0-9]{1,16}).*
{code}
The code then goes on to use capture group 3 (the round count) to determine how
many rounds are used, and capture group 4 (the actual salt) to use as salt data.
However, for an input that contains an invalid salt specification like this:
{code:java}
$5$notrounds=1000$asdfasdf
{code}
This string is treated as valid. The operation then uses "notrounds" as the
salt for hashing:
{code:java}
System.out.println(Sha2Crypt.sha256Crypt(new byte[100],
"$5$notrounds=1000$asdfasdf"));
{code}
The above code prints
{code:java}
$5$notrounds$aAEx6EzUBfc.aIIeiItVjUREbj/ar4xlb/qcMxD90NA
{code}
This code should probably throw an exceptions. Additionally, other invalid salt
strings like
{code:java}
$6$rounds=1000$äöüäöü
{code}
Result in hashes like:
{code:java}
$5$rounds$ZIRdMgi25kK5Zmi1whKjcnFN/Pe8QCAOlTmxMfwAoq5
{code}
Completely ignoring the rounds parameter, and using the literal string "rounds"
as the salt for hashing.
> RegEx for verifying salts in ShaCrypt is incorrect
> --
>
> Key: CODEC-245
> URL: https://issues.apache.org/jira/browse/CODEC-245
> Project: Commons Codec
> Issue Type: Bug
>Affects Versions: 1.11
>Reporter: Simon Levermann
>Priority: Major
>
> The regex/code that extracts the salt from a given salt string in Sha2Crypt
> treats some invalid salt formats as valid:
> {code:java}
> ^\$([56])\$(rounds=(\d+)\$)?([\.\/a-zA-Z0-9]{1,16}).*
> {code}
> The code then goes on to use capture group 3 (the round count) to determine
> how many rounds are used, and capture group 4 (the actual salt) to use as
> salt data.
> However, for an input that contains an invalid salt specification like this:
> {code:java}
> $5$notrounds=1000$asdfasdf
> {code}
> This string is treated as valid. The operation then uses "notrounds" as the
> salt for hashing:
> {code:java}
> System.out.println(Sha2Crypt.sha256Crypt(new byte[100],
> "$5$notrounds=1000$asdfasdf"));
> {code}
> The above code prints
> {code:java}
> $5$notrounds$aAEx6EzUBfc.aIIeiItVjUREbj/ar4xlb/qcMxD90NA
> {code}
> This code should probably throw an exception. Additionally, other invalid
> salt strings like
>
> {code:java}
> $6$rounds=1000$äöüäöü
> {code}
> Result in hashes like:
> {code:java}
> $5$rounds$ZIRdMgi25kK5Zmi1whKjcnFN/Pe8QCAOlTmxMfwAoq5
> {code}
> Completely ignoring the rounds parameter, and using the literal string
> "rounds" as the salt for hashing.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (CODEC-245) RegEx for verifying salts in ShaCrypt is incorrect
Simon Levermann created CODEC-245:
-
Summary: RegEx for verifying salts in ShaCrypt is incorrect
Key: CODEC-245
URL: https://issues.apache.org/jira/browse/CODEC-245
Project: Commons Codec
Issue Type: Bug
Affects Versions: 1.11
Reporter: Simon Levermann
The regex/code that extracts the salt from a given salt string in Sha2Crypt
treats some invalid salt formats as valid:
{code:java}
^\$([56])\$(rounds=(\d+)\$)?([\.\/a-zA-Z0-9]{1,16}).*
{code}
The code then goes on to use capture group 3 (the round count) to determine how
many rounds are used, and capture group 4 (the actual salt) to use as salt data.
However, for an input that contains an invalid salt specification like this:
{code:java}
$5$notrounds=1000$asdfasdf
{code}
This string is treated as valid. The operation then uses "notrounds" as the
salt for hashing:
{code:java}
System.out.println(Sha2Crypt.sha256Crypt(new byte[100],
"$5$notrounds=1000$asdfasdf"));
{code}
The above code prints
{code:java}
$5$notrounds$aAEx6EzUBfc.aIIeiItVjUREbj/ar4xlb/qcMxD90NA
{code}
This code should probably throw an exceptions. Additionally, other invalid salt
strings like
{code:java}
$6$rounds=1000$äöüäöü
{code}
Result in hashes like:
{code:java}
$5$rounds$ZIRdMgi25kK5Zmi1whKjcnFN/Pe8QCAOlTmxMfwAoq5
{code}
Completely ignoring the rounds parameter, and using the literal string "rounds"
as the salt for hashing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (CODEC-171) Add support for CRC32-C
[
https://issues.apache.org/jira/browse/CODEC-171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075134#comment-16075134
]
Simon Spero commented on CODEC-171:
---
This class becomes redundant in JDK 9, which introduces
[java.util.zip.CRC32C|http://download.java.net/java/jdk9/docs/api/java/util/zip/CRC32C.html],
which as one might expect, implements j.u.z.Checksum.
Both implementations use the same basic approach. The main differences are (a)
that the hadoop/Commons Codec implementation has hardcoded lookup tables,
whilst the oracle one calculates them on startup, and (b) a few methods like
these :
{code:java}
/** Updates the CRC-32C checksum with the specified array of bytes. */
@HotSpotIntrinsicCandidate
private static int updateBytes(int crc, byte[] b, int off, int end) {
[...]
}
/**Updates the CRC-32C checksum reading from the specified address. */
@HotSpotIntrinsicCandidate
private static int updateDirectByteBuffer(int crc, long address, int off,
int end) {
[...]
}
{code}
Notice the HotSpotIntrinsicCandidate annotations... on hotspot, the java code
for these methods will almost certainly not get called; instead calls to the
methods will be replaced by architecture specific code, using any available
hardware support.
The commons codec code is useful for jdk 8 and earlier, but should not be used
on 9.
One possible approach would be to generate a multi-release jar with a JDK 9+
specific version of the code that delegates to the native implementation (the
class if final, so delegation is required). Current maven support for MR jars
is... incomplete...
> Add support for CRC32-C
> ---
>
> Key: CODEC-171
> URL: https://issues.apache.org/jira/browse/CODEC-171
> Project: Commons Codec
> Issue Type: Improvement
>Reporter: Brett Okken
> Fix For: 1.11
>
>
> Add a java CRC32-C java.util.zip.Checksum implementation.
> There is a PureJavaCrc32C implementation that is part of
> apache-hadoop-common. It would be great to have that more easily consummable
> with far fewer transitive dependencies.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (CODEC-238) DigestUtils should make it easier to generate hex strings from MessageDigests and Digest Streams
[ https://issues.apache.org/jira/browse/CODEC-238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16075065#comment-16075065 ] Simon Spero commented on CODEC-238: --- [Pull Request|https://github.com/apache/commons-codec/pull/8] submitted on github. > DigestUtils should make it easier to generate hex strings from > MessageDigests and Digest Streams > -- > > Key: CODEC-238 > URL: https://issues.apache.org/jira/browse/CODEC-238 > Project: Commons Codec > Issue Type: New Feature >Reporter: Simon Spero >Priority: Minor > Fix For: 1.11 > > > Digests are the usual reason why I need Hex :-) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (CODEC-237) README.md does not have project specific JIRA link
[ https://issues.apache.org/jira/browse/CODEC-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Spero closed CODEC-237. - Is good (Got here from github page :) > README.md does not have project specific JIRA link > -- > > Key: CODEC-237 > URL: https://issues.apache.org/jira/browse/CODEC-237 > Project: Commons Codec > Issue Type: Improvement > Reporter: Simon Spero >Priority: Blocker > Fix For: 1.11 > > > Did the property not expand correctly when the file was generated? > As things are there's no way to reportbugs > /me downgrades from BLOCKER -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (CODEC-237) README.md does not have project specific JIRA link
[ https://issues.apache.org/jira/browse/CODEC-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Spero resolved CODEC-237. --- Resolution: Fixed Fix Version/s: 1.11 Looks good > README.md does not have project specific JIRA link > -- > > Key: CODEC-237 > URL: https://issues.apache.org/jira/browse/CODEC-237 > Project: Commons Codec > Issue Type: Improvement > Reporter: Simon Spero >Priority: Blocker > Fix For: 1.11 > > > Did the property not expand correctly when the file was generated? > As things are there's no way to reportbugs > /me downgrades from BLOCKER -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CODEC-224) Add convenience API org.apache.commons.codec.binary.Hex.encodeHexString(byte[]|ByteBuffer, boolean)
[
https://issues.apache.org/jira/browse/CODEC-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16074089#comment-16074089
]
Simon Spero commented on CODEC-224:
---
Oops- wrong issue
> Add convenience API
> org.apache.commons.codec.binary.Hex.encodeHexString(byte[]|ByteBuffer,
> boolean)
> ---
>
> Key: CODEC-224
> URL: https://issues.apache.org/jira/browse/CODEC-224
> Project: Commons Codec
> Issue Type: Improvement
>Reporter: Gary Gregory
>Assignee: Gary Gregory
> Fix For: 1.11
>
>
> Add convenience APIs:
> {{org.apache.commons.codec.binary.Hex.encodeHexString(byte[], boolean)}}:
> {code:java}
> /**
> * Converts an array of bytes into a String representing the hexadecimal
> values of each byte in order. The returned
> * String will be double the length of the passed array, as it takes two
> characters to represent any given byte.
> *
> * @param data
> *a byte[] to convert to Hex characters
> * @param toLowerCase
> *true converts to lowercase, false
> to uppercase
> * @return A String containing lower-case hexadecimal characters
> * @since 1.11
> */
> public static String encodeHexString(final byte[] data, boolean
> toLowerCase)
> {code}
> {{org.apache.commons.codec.binary.Hex.encodeHexString(ByteBuffer, boolean)}}:
> {code:java}
> /**
> * Converts a byte buffer into a String representing the hexadecimal
> values of each byte in order. The returned
> * String will be double the length of the passed array, as it takes two
> characters to represent any given byte.
> *
> * @param data
> *a byte buffer to convert to Hex characters
> * @param toLowerCase
> *true converts to lowercase, false
> to uppercase
> * @return A String containing lower-case hexadecimal characters
> * @since 1.11
> */
> public static String encodeHexString(final ByteBuffer data, boolean
> toLowerCase)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CODEC-238) DigestUtils should make it easier to generate hex strings from MessageDigests and Digest Streams
Simon Spero created CODEC-238: - Summary: DigestUtils should make it easier to generate hex strings from MessageDigests and Digest Streams Key: CODEC-238 URL: https://issues.apache.org/jira/browse/CODEC-238 Project: Commons Codec Issue Type: New Feature Reporter: Simon Spero Priority: Minor Fix For: 1.11 Digests are the usual reason why I need Hex :-) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (CODEC-224) Add convenience API org.apache.commons.codec.binary.Hex.encodeHexString(byte[]|ByteBuffer, boolean)
[
https://issues.apache.org/jira/browse/CODEC-224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073994#comment-16073994
]
Simon Spero commented on CODEC-224:
---
This commit also added lower/upper case configuration for encodeHexString,
which I was about to add because I needed it for coveralls.
> Add convenience API
> org.apache.commons.codec.binary.Hex.encodeHexString(byte[]|ByteBuffer,
> boolean)
> ---
>
> Key: CODEC-224
> URL: https://issues.apache.org/jira/browse/CODEC-224
> Project: Commons Codec
> Issue Type: Improvement
>Reporter: Gary Gregory
>Assignee: Gary Gregory
> Fix For: 1.11
>
>
> Add convenience APIs:
> {{org.apache.commons.codec.binary.Hex.encodeHexString(byte[], boolean)}}:
> {code:java}
> /**
> * Converts an array of bytes into a String representing the hexadecimal
> values of each byte in order. The returned
> * String will be double the length of the passed array, as it takes two
> characters to represent any given byte.
> *
> * @param data
> *a byte[] to convert to Hex characters
> * @param toLowerCase
> *true converts to lowercase, false
> to uppercase
> * @return A String containing lower-case hexadecimal characters
> * @since 1.11
> */
> public static String encodeHexString(final byte[] data, boolean
> toLowerCase)
> {code}
> {{org.apache.commons.codec.binary.Hex.encodeHexString(ByteBuffer, boolean)}}:
> {code:java}
> /**
> * Converts a byte buffer into a String representing the hexadecimal
> values of each byte in order. The returned
> * String will be double the length of the passed array, as it takes two
> characters to represent any given byte.
> *
> * @param data
> *a byte buffer to convert to Hex characters
> * @param toLowerCase
> *true converts to lowercase, false
> to uppercase
> * @return A String containing lower-case hexadecimal characters
> * @since 1.11
> */
> public static String encodeHexString(final ByteBuffer data, boolean
> toLowerCase)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (CODEC-237) README.md does not have project specific JIRA link
Simon Spero created CODEC-237: - Summary: README.md does not have project specific JIRA link Key: CODEC-237 URL: https://issues.apache.org/jira/browse/CODEC-237 Project: Commons Codec Issue Type: Improvement Reporter: Simon Spero Priority: Blocker Did the property not expand correctly when the file was generated? As things are there's no way to reportbugs /me downgrades from BLOCKER -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16073504#comment-16073504 ] Simon Spero commented on COMPRESS-416: -- There was an oddity before with IBM java 8,which is earlier, but it was an incredibly suspicious off by a second thing, with a little timezone too. I will see if it all behaves. > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > Attachments: surefire-reports.zip > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-210) handle zip extra field 0x5455 - Extended Timestamp
[
https://issues.apache.org/jira/browse/COMPRESS-210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072825#comment-16072825
]
Simon Spero commented on COMPRESS-210:
--
zip extended timestamps are made signed! Doh!
> handle zip extra field 0x5455 - Extended Timestamp
> --
>
> Key: COMPRESS-210
> URL: https://issues.apache.org/jira/browse/COMPRESS-210
> Project: Commons Compress
> Issue Type: Improvement
> Components: Archivers
>Reporter: Julius Davies
>Assignee: Julius Davies
>Priority: Minor
> Fix For: 1.5
>
> Attachments: COMPRESS-210.patch
>
>
> From Info Zip's "zip-3.0" source archive (proginfo/extrafld.txt):
> -Extended Timestamp Extra Field:
> ==
> The following is the layout of the extended-timestamp extra block.
> (Last Revision 19970118)
> {noformat}
> Local-header version:
> Value SizeDescription
> - ---
> (time) 0x5455Short tag for this extra block type ("UT")
> TSize Short total data size for this block
> Flags Byteinfo bits
> (ModTime) Longtime of last modification (UTC/GMT)
> (AcTime) Longtime of last access (UTC/GMT)
> (CrTime) Longtime of original creation (UTC/GMT)
> Central-header version:
> Value SizeDescription
> - ---
> (time) 0x5455Short tag for this extra block type ("UT")
> TSize Short total data size for this block
> Flags Byteinfo bits (refers to local header!)
> (ModTime) Longtime of last modification (UTC/GMT)
> {noformat}
> The central-header extra field contains the modification time only,
> or no timestamp at all. TSize is used to flag its presence or
> absence. But note:
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072800#comment-16072800 ] Simon Spero commented on COMPRESS-416: -- http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/48b296b645bd#l1.103 Not mentioned in the commit log, not backported to jdk8 > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > Attachments: surefire-reports.zip > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[
https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072786#comment-16072786
]
Simon Spero commented on COMPRESS-416:
--
But... after going back to the original path, I see that the actual definition
of the extended timestamps states that:
So COMMONS-COMPRESS and the Oracle JDK were both doing it wrong.
So this goes back to the original patch in COMMONS-220
{noformat}
The time values are in standard Unix signed-long format, indicating the number
of seconds since 1 January 1970 00:00:00.
{noformat}
[https://opensource.apple.com/source/zip/zip-6/unzip/unzip/proginfo/extra.fld]
> Tests failing under jdk 9 : one reflection issue, one change to ZipEntry
> related issue
> --
>
> Key: COMPRESS-416
> URL: https://issues.apache.org/jira/browse/COMPRESS-416
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.14, 1.15
> Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be
> the RC, or if not, what would have been RC).
> java.runtime.version = 9-internal+0-adhoc.ses.jdk9
> java.specification.version = 9
> java.version = 9-internal
> java.vm.specification.version = 9
> os.arch = amd64
> os.name = Linux
> os.version = 4.4.0-81-generic
>Reporter: Simon Spero
> Fix For: 1.15
>
> Attachments: surefire-reports.zip
>
>
> X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to
> be a bogus value returned from getTime(). It seems like the test failure
> might be due to the changes introduced for this:
> https://bugs.openjdk.java.net/browse/JDK-8073497
> Tests were run using intelliJ TestRunner, using the openjdk9 build from the
> tip of the jdk9 tree (not dev). I believe that this is at most one commit
> away from what will be the RC (which was delayed at the last minute due to
> two issues, one of which was javadoc related, and the other hotspot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072764#comment-16072764 ] Simon Spero commented on COMPRESS-416: -- Found it! Changed a method signature from long to int and didn't deal with sign extension. Now I can finish the bug report. > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > Attachments: surefire-reports.zip > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072714#comment-16072714 ] Simon Spero commented on COMPRESS-416: -- I have a draft open in another window. One special cause for speculation: zip file parsing moved from C to java in JDK 9. Also, I discovered that I had off-by-one src.zips in my jdk9 build, which explains why debugging was confusing the heck out of me. So I'm rebuilding so I can check to see what's going on in the central directory parse. Worst thing about trying to debug j.u.z.* is loading stuff from jars. > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > Attachments: surefire-reports.zip > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[
https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072647#comment-16072647
]
Simon Spero commented on COMPRESS-416:
--
It's an oracle issue.
{code:java}
@Test
public void testJUZTimes() throws Exception {
final File archive = getFile("COMPRESS-210_unix_time_zip_test.zip");
try (java.util.zip.ZipFile zf = new java.util.zip.ZipFile(archive)) {
ZipEntry entry = zf.getEntry("COMPRESS-210_unix_time_zip_test/2105");
Calendar girl = Calendar.getInstance(TimeZone.getTimeZone("GMT"));
girl.setTime(new Date(entry.getTime()));
int year = girl.get(Calendar.YEAR);
assertEquals(2105,year);
}
}
{code}
java.lang.AssertionError:
Expected :2105
Actual :1968
1968-11-24T00:00:00.000Z to be precise. So I'm weeks away from being
conceived, and the Nixon transition team is busy picking out hidden tape
recorders.
> Tests failing under jdk 9 : one reflection issue, one change to ZipEntry
> related issue
> --
>
> Key: COMPRESS-416
> URL: https://issues.apache.org/jira/browse/COMPRESS-416
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.14, 1.15
> Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be
> the RC, or if not, what would have been RC).
> java.runtime.version = 9-internal+0-adhoc.ses.jdk9
> java.specification.version = 9
> java.version = 9-internal
> java.vm.specification.version = 9
> os.arch = amd64
> os.name = Linux
> os.version = 4.4.0-81-generic
>Reporter: Simon Spero
> Fix For: 1.15
>
> Attachments: surefire-reports.zip
>
>
> X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to
> be a bogus value returned from getTime(). It seems like the test failure
> might be due to the changes introduced for this:
> https://bugs.openjdk.java.net/browse/JDK-8073497
> Tests were run using intelliJ TestRunner, using the openjdk9 build from the
> tip of the jdk9 tree (not dev). I believe that this is at most one commit
> away from what will be the RC (which was delayed at the last minute due to
> two issues, one of which was javadoc related, and the other hotspot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072561#comment-16072561 ] Simon Spero commented on COMPRESS-416: -- >From the mercurial logs I think this it is definitely related - possibly fixing a performance regression there was an issue with locales being initialized too eagerly. It is indeed very sensitive to jdk(since it seems to be randomized each release). The extended header is parsed correctly (and the zip entry itself has the right FileTime). It's the dosxtime that seems messed up. Now to see where that's set (of course, extended header test should be OK just testing itself). If this wasn't the only known jdk9 fail issue it wouldn't be bugging me :) > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > Attachments: surefire-reports.zip > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072408#comment-16072408 ] Simon Spero commented on COMPRESS-416: -- I think I mostly understand the issue now. # There are three different formats in which Zip can store times # Two of these formats will overflow around the year 2107-8 # JDK 9 transparently switches from using dos times or xtra dos times to using NTFS times # JDK 9 now switches over to NTFS times at the end of 2099. # Since some of this logic is in the set code, there may be a bogus timestamp stored. # I haven't checked to see if the problem can happen with java utils ZipInputStream on the same test file, or if the problem comes from commons-compress. If the problem occurs using the test file with the java libs, then it's definitely a jdk bug. # If the test is hacked to say that xtended time is always used, the tests all pass. # This may > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > Attachments: surefire-reports.zip > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16072408#comment-16072408 ] Simon Spero edited comment on COMPRESS-416 at 7/3/17 1:07 PM: -- I think I mostly understand the issue now. # There are three different formats in which Zip can store times # Two of these formats will overflow around the year 2107-8 # JDK 9 transparently switches from using dos times or xtra dos times to using NTFS times # JDK 9 now switches over to NTFS times at the end of 2099. # Since some of this logic is in the set code, there may be a bogus timestamp stored. # I haven't checked to see if the problem can happen with java utils ZipInputStream on the same test file, or if the problem comes from commons-compress. If the problem occurs using the test file with the java libs, then it's definitely a jdk bug. # If the test is hacked to say that xtended time is always used, the tests all pass. was (Author: sesuncedu): I think I mostly understand the issue now. # There are three different formats in which Zip can store times # Two of these formats will overflow around the year 2107-8 # JDK 9 transparently switches from using dos times or xtra dos times to using NTFS times # JDK 9 now switches over to NTFS times at the end of 2099. # Since some of this logic is in the set code, there may be a bogus timestamp stored. # I haven't checked to see if the problem can happen with java utils ZipInputStream on the same test file, or if the problem comes from commons-compress. If the problem occurs using the test file with the java libs, then it's definitely a jdk bug. # If the test is hacked to say that xtended time is always used, the tests all pass. # This may > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > Attachments: surefire-reports.zip > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-413) Travis build redundantly repeats compilation and tests redundantly
[
https://issues.apache.org/jira/browse/COMPRESS-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071324#comment-16071324
]
Simon Spero commented on COMPRESS-413:
--
I will try a few sample builds to to check for effect on build time to
completion. If there's no important effect then this is fine; if it is
important then I may reorder the matrix.
I will submit an issue with a patch for changes that belong in parent
(unless parent is now in git.
The indicated errors are the ones I expect to see with the final 9-ea. The
illegal access from the mocker in ZCompressorInputStreamTest is reduced to
a warning in RC, which is acceptable for testing and can be left for now.
The timestamp behavior is an error, but I am not sure if the new behavior
in jdk 9 is wrong and needs a workaround, or right and needs a workaround,
or is wrong but should be ignored as it seems to be a deliberate decision
to not hack around DOS years after 2099. Adapting to follow jdk behavior
seems to be the best bet for now, since the error is being detected in a
situation where there are correct timestamps.
I could file an issue in this for jdk core-libs but I think this might be
scheduled for a fix in jdk 49).
(My current thinking is to deprecate the use of DOS times for creating
archives after that year, and to ensure that any consent forms for
replicating or storing any simulation of what "I" "perceive" as "my"
"consciousness" stipulate that no such contents may be stored in zip
format.
It will go next to the clause refusing permission to transform me into a
brain in a vat unless it's VAT 69)
On Jun 30, 2017 7:05 PM, "Gary Gregory (JIRA)" wrote:
[ https://issues.apache.org/jira/browse/COMPRESS-413?page=
com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Gregory resolved COMPRESS-413.
---
Resolution: Fixed
Thank you for the PR. Patch applied except for the removal of Oracle 7 and
8 from the Travis build.
Please verify and close or continue.
I only see 1 error and one failure now instead of a bunch:
{noformat}
Failed tests:
X5455_ExtendedTimestampTest.testSampleFile:185
expected:<[2105-01-01/00:00:02]
+> but was:<[1968-11-24/17:31:45] +>
Tests in error:
ZCompressorInputStreamTest.testFailsToCreateZCompressorInputStreamAndThrowsIOException
╗
Tests run: 842, Failures: 1, Errors: 1, Skipped: 4
{noformat}
phase.
tests.
retested; this time with
run tests with coverage during the build phase.
if the service is unreachable, so forks that don't have jacoco enabled
won't always have their builds fail.
properly at the moment, so installing a jdk7 doesn't work properly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
> Travis build redundantly repeats compilation and tests redundantly
> --
>
> Key: COMPRESS-413
> URL: https://issues.apache.org/jira/browse/COMPRESS-413
> Project: Commons Compress
> Issue Type: Improvement
> Components: Build
>Affects Versions: 1.14
> Environment: Travis
>Reporter: Simon Spero
>Priority: Minor
> Labels: CI
> Fix For: 1.15
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> The Travis build setup is suboptimal.
> At the moment, code is compiled and installed by the default install phase.
> Then the default build phase is executed, which compiles and runs the tests.
> If the tests succeed, then the build is cleaned, recompiled, and retested;
> this time with
> coverage enabled.
> The .travis.yml file could be changed to skip the install phase, and to run
> tests with coverage during the build phase.
> The coveralls plugin can be configured in the pom to not fail the build if
> the service is unreachable, so forks that don't have jacoco enabled won't
> always have their builds fail.
> Also, the jdk switching in the trusty container seems to be not working
> properly at the moment, so installing a jdk7 doesn't work properly.
> These changes evolved as I was poking jenkins last night.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-413) Travis build redundantly repeats compilation and tests redundantly
[ https://issues.apache.org/jira/browse/COMPRESS-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070784#comment-16070784 ] Simon Spero commented on COMPRESS-413: -- Oracle JDK 7 is EOL and is missing critical security patches. OpenJDK 7 is at least lightly patched. Oracle JDK and OpenJDK are in a different situation to IBM java. Open/Oracle JDKs both use the same virtual machine, and the roughly the same source trees, whereas IBM java uses a completely different VM implementation (J9), and also has a few differences in the runtime libraries (mostly openjdk, but there are a few differences). Any code that uses private interfaces not covered by the TCK is having problems. I haven't had any On Fri, Jun 30, 2017 at 5:23 PM, ASF GitHub Bot (JIRA) > Travis build redundantly repeats compilation and tests redundantly > -- > > Key: COMPRESS-413 > URL: https://issues.apache.org/jira/browse/COMPRESS-413 > Project: Commons Compress > Issue Type: Improvement > Components: Build >Affects Versions: 1.14 > Environment: Travis >Reporter: Simon Spero >Priority: Minor > Labels: CI > Fix For: 1.15 > > Original Estimate: 0h > Remaining Estimate: 0h > > The Travis build setup is suboptimal. > At the moment, code is compiled and installed by the default install phase. > Then the default build phase is executed, which compiles and runs the tests. > If the tests succeed, then the build is cleaned, recompiled, and retested; > this time with > coverage enabled. > The .travis.yml file could be changed to skip the install phase, and to run > tests with coverage during the build phase. > The coveralls plugin can be configured in the pom to not fail the build if > the service is unreachable, so forks that don't have jacoco enabled won't > always have their builds fail. > Also, the jdk switching in the trusty container seems to be not working > properly at the moment, so installing a jdk7 doesn't work properly. > These changes evolved as I was poking jenkins last night. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070616#comment-16070616 ] Simon Spero commented on COMPRESS-416: -- I've replaced the use of mocks in SevenZNativeHeapTest with delegation. I'm not sure whether this test is really testing much, but it's easy enough to just convert a couple of anonymous classes to named package private ones, and to generate trivial delegates. I can see where the bad code is in the mock libs, but that's not my problem anymore :-) All JDK 9 changes are going into the fixed travis tree, since I've learned my lesson and won't try splitting PRs that depend on other dependencies. > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > Attachments: surefire-reports.zip > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[
https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16070536#comment-16070536
]
Simon Spero commented on COMPRESS-416:
--
The issues are being caused by jacoco making a class unloadable by twiddling
with a final field outside a constructor.
This is fixed by using a newer version of jacoco.
I have made a few changes to the enhanced travis branch, which reduces the, er,
non-successes to three tests.
See: https://github.com/apache/commons-compress/pull/43
Some of these are changes to outdated plugin versions. I blame the parents.
There are two "failures" and one "error". Quotation marks are used because one
of the failures is actually an error.
The real failure is the change in the handling of zip dates in the jdk
ZipEntry. This is an actual incompatibility.
The second failure is really an illegal access from a mocker; it just happens
to take place in test for an expected exception. This presents as a warning
in the (probable) RC -
{noformat}
Running org.apache.commons.compress.compressors.z.ZCompressorInputStreamTest
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.mockito.cglib.core.ReflectUtils$2
(file:/home/ses/.m2/repository/org/mockito/mockito-core/1.10.19/mockito-core-1.10.19.jar)
to method
java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
WARNING: Please consider reporting this to the maintainers of
org.mockito.cglib.core.ReflectUtils$2
WARNING: Use --illegal-access=warn to enable warnings of further illegal
reflective access operations
WARNING: All illegal access operations will be denied in a future release
{noformat}
I'm not sure how I feel about having to locate the maintainers of an anonymous
inner class :-)
> Tests failing under jdk 9 : one reflection issue, one change to ZipEntry
> related issue
> --
>
> Key: COMPRESS-416
> URL: https://issues.apache.org/jira/browse/COMPRESS-416
> Project: Commons Compress
> Issue Type: Bug
>Affects Versions: 1.14, 1.15
> Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be
> the RC, or if not, what would have been RC).
> java.runtime.version = 9-internal+0-adhoc.ses.jdk9
> java.specification.version = 9
> java.version = 9-internal
> java.vm.specification.version = 9
> os.arch = amd64
> os.name = Linux
> os.version = 4.4.0-81-generic
>Reporter: Simon Spero
> Fix For: 1.15
>
> Attachments: surefire-reports.zip
>
>
> X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to
> be a bogus value returned from getTime(). It seems like the test failure
> might be due to the changes introduced for this:
> https://bugs.openjdk.java.net/browse/JDK-8073497
> Tests were run using intelliJ TestRunner, using the openjdk9 build from the
> tip of the jdk9 tree (not dev). I believe that this is at most one commit
> away from what will be the RC (which was delayed at the last minute due to
> two issues, one of which was javadoc related, and the other hotspot.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-414) Provide API for Channel(-ish) Seekable, Possibly Read/Write, Possibly Transactional Compressors & Archivers,
[ https://issues.apache.org/jira/browse/COMPRESS-414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069474#comment-16069474 ] Simon Spero commented on COMPRESS-414: -- I think that this could use a fair chunk of the prototyping in the version 2 branch. It ought to be possible to evolve an API along the lines discussed above and in the 2 branch, with the existing api becoming deprecated for removal (less removal than becoming the implementation classes becoming not exported to OSGI and Jigsaw users) Much of the implementation code could slide into new, private packages already; the current API could mostly be delegates, and the most of the test suite that use the public API could become kind of integration tests (with the other tests following the implementation.) Things would work cleaner with a multiproject build, with projects for the general apis, with a separate project for each archiver / compressor. Implementations of formats would be exposed as services (OSGI declarative and module-info / META-INF/services) There should also be a combined single dependency thing (a shadowed / shaded jar, and a Karaf feature.xml?) The artifacts for each format could be released on separate schedules (e.g. a bug-fix or performance tweak for 7z could be released without requiring new releases of everything else (so people just wanting tar don't have to be affected). > Provide API for Channel(-ish) Seekable, Possibly Read/Write, Possibly > Transactional Compressors & Archivers, > -- > > Key: COMPRESS-414 > URL: https://issues.apache.org/jira/browse/COMPRESS-414 > Project: Commons Compress > Issue Type: Improvement >Reporter: Simon Spero > > It would be very useful to have APIs to support random-access, possibly > read-write archives & compressors. > These interfaces should probably be based around Seekable, Scatter, and > Gather ByteChannels. > It might be necessary or desirable to have these classes extend FileChannel, > since there are some important methods that first make their appearance as > abstract methods on that class. > The methods of interest are the absolute positioned read and write methods, > plus force/sync. > The use case I'm thinking of for the the non-relative read/write would be > for a random-access compressed "file", where it may be useful to keep a cache > of decompressed blocks, and where a task may require access to several parts > of the file simultaneously. Nesting this deeper, there might be (well, > there are) a tar.xz file containing a sequence of versions of a maven > artifact jar, and with an analysis process that needs access to two or > more of the files simultaneously. > A use case requiring read/write + transactional + sync support would be to > support in-place appending new items to a tar.xz file. It can be quite > useful to be able to perform such updates "in-place"; for safety this > requires a log the parts of the existing file that are going to be changed, > so that the contents can be recovered in the event of failure. > If there is related metadata in a separate transactional resource, it is good > to have both stores remain in sync. The complicated parts are the job of a > transaction manager (e.g. Aries); basic XAResource support is relatively > simple.There are other bits of glue that are useful for handling simple > transactions with no transaction manager around, but those can be optional > imports. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16069451#comment-16069451 ] Simon Spero commented on COMPRESS-416: -- Greg - can you attach the sure xml , or something with stacktraces? There seems to be something going on in the locate setup , and I think many of those places are setting up encoders. I think I read some descriptions of something similar (which didn't stick in brain L2). (I do have access to a windows environment, but I'll be in trouble if my partner wakes up and catches me (say - if I install cygwin and sshd, her laptop only has an SSD, so there's no HD rattling to give me away.. :) It's probably safer to see if I can get a travis jdk9 to produce the same errors from maven (the branch with travis tweaking has mvnw for 3.5.0; I forgot which version of 9 is in the new trusty stable container on travis, but I think it's the last oracle ea). > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
[ https://issues.apache.org/jira/browse/COMPRESS-416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Spero updated COMPRESS-416: - Affects Version/s: 1.15 Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be the RC, or if not, what would have been RC). java.runtime.version = 9-internal+0-adhoc.ses.jdk9 java.specification.version = 9 java.version = 9-internal java.vm.specification.version = 9 os.arch = amd64 os.name = Linux os.version = 4.4.0-81-generic was:JDK 9 Fix Version/s: 1.15 Description: X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to be a bogus value returned from getTime(). It seems like the test failure might be due to the changes introduced for this: https://bugs.openjdk.java.net/browse/JDK-8073497 Tests were run using intelliJ TestRunner, using the openjdk9 build from the tip of the jdk9 tree (not dev). I believe that this is at most one commit away from what will be the RC (which was delayed at the last minute due to two issues, one of which was javadoc related, and the other hotspot. was: X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to be a bogus value returned from getTime(). It seems like the test failure might be due to the changes introduced for this: https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests failing under jdk 9 : one reflection issue, one change to ZipEntry > related issue > -- > > Key: COMPRESS-416 > URL: https://issues.apache.org/jira/browse/COMPRESS-416 > Project: Commons Compress > Issue Type: Bug >Affects Versions: 1.14, 1.15 > Environment: JDK 9 ( jdk9 tree tip - I believe this is what will be > the RC, or if not, what would have been RC). > java.runtime.version = 9-internal+0-adhoc.ses.jdk9 > java.specification.version = 9 > java.version = 9-internal > java.vm.specification.version = 9 > os.arch = amd64 > os.name = Linux > os.version = 4.4.0-81-generic >Reporter: Simon Spero > Fix For: 1.15 > > > X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to > be a bogus value returned from getTime(). It seems like the test failure > might be due to the changes introduced for this: > https://bugs.openjdk.java.net/browse/JDK-8073497 > Tests were run using intelliJ TestRunner, using the openjdk9 build from the > tip of the jdk9 tree (not dev). I believe that this is at most one commit > away from what will be the RC (which was delayed at the last minute due to > two issues, one of which was javadoc related, and the other hotspot. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (COMPRESS-416) Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue
Simon Spero created COMPRESS-416: Summary: Tests failing under jdk 9 : one reflection issue, one change to ZipEntry related issue Key: COMPRESS-416 URL: https://issues.apache.org/jira/browse/COMPRESS-416 Project: Commons Compress Issue Type: Bug Affects Versions: 1.14 Environment: JDK 9 Reporter: Simon Spero X5455_ExtendedTimestampTest is failing under JDK 9 , due to what appears to be a bogus value returned from getTime(). It seems like the test failure might be due to the changes introduced for this: https://bugs.openjdk.java.net/browse/JDK-8073497 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (LANG-1342) JavaVersion is not consistent with JEP-223
[
https://issues.apache.org/jira/browse/LANG-1342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16065494#comment-16065494
]
Simon Spero commented on LANG-1342:
---
Of course, JEP 223 is a little non-semantic-versiony on the semantics of
specification.version . A minor version may include non-breaking api changes
from MR of specifications, but those versions aren't specification versions.
{panel:title=Minor}
$MINOR --- The minor version number, incremented for a minor update release
that may contain compatible bug fixes, revisions to standard APIs mandated by a
Maintenance Release of the relevant Platform Specification, and implementation
features outside the scope of that Specification such as new JDK-specific APIs,
additional service providers, new garbage collectors, and ports to new hardware
architectures.
{panel}
*Major (GA)*
{noformat}
java.version1.9.0 9
java.runtime.version1.9.0-b1009+100
java.vm.version 1.9.0-b1009+100
java.specification.version 1.9 9
java.vm.specification.version 1.9 9
{noformat}
*Minor #1 (GA)*
{noformat}
java.version1.9.0_20 9.1.2
java.runtime.version1.9.0_20-b62 9.1.2+62
java.vm.version 1.9.0_20-b62 9.1.2+62
java.specification.version 1.9 9
java.vm.specification.version 1.9 9
{noformat}
> JavaVersion is not consistent with JEP-223
> --
>
> Key: LANG-1342
> URL: https://issues.apache.org/jira/browse/LANG-1342
> Project: Commons Lang
> Issue Type: Bug
> Components: lang.*
>Affects Versions: 3.6
>Reporter: Simon Spero
>Priority: Minor
>
> [JEP 223: New Version-String Syntax|http://openjdk.java.net/jeps/223]
> introduces a new syntax for version strings.
> This syntax is sort-of not incompatible with JavaVersion in 3.5, but this is
> likely to change as soon as someone finds a bug in JDK 9 after it's released.
> There is a third specified version number component for security releases.
> Unlike the rest of the full version string, this component is significant for
> ordering, and cannot reliably be mapped to float.
> JEP 223 defines some regular expressions. It also defines an api class
> (java.lang.Runtime.Version). This class can't be used directly (since it's
> new with JDK-9), and the API can't be directly emulated, as it requires
> java.lang.Optional. However, it's not rocket science (it is however regular
> expressions, so now you have problems 2.0).
> I'll take a crack at a class implementing JEP-223 Versions and try to get
> them to play with JavaVersion.
> The existing mapping to float is unsuited to minor versions, so there has to
> be some new hackery for java 9 even for two components. ( JDK 9.11 should
> be > JDK 9.2 )
> Since there's no java minor releases yet, there's a bit of freedom. If the
> range of minor and patch versions is restricted to values between 0 and 99
> there should be a way to make things work.
> Ideally the floaty stuff can be deprecated for removal before that matters.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (LANG-1342) JavaVersion is not consistent with JEP-223
Simon Spero created LANG-1342: - Summary: JavaVersion is not consistent with JEP-223 Key: LANG-1342 URL: https://issues.apache.org/jira/browse/LANG-1342 Project: Commons Lang Issue Type: Bug Components: lang.* Affects Versions: 3.6 Reporter: Simon Spero Priority: Minor [JEP 223: New Version-String Syntax|http://openjdk.java.net/jeps/223] introduces a new syntax for version strings. This syntax is sort-of not incompatible with JavaVersion in 3.5, but this is likely to change as soon as someone finds a bug in JDK 9 after it's released. There is a third specified version number component for security releases. Unlike the rest of the full version string, this component is significant for ordering, and cannot reliably be mapped to float. JEP 223 defines some regular expressions. It also defines an api class (java.lang.Runtime.Version). This class can't be used directly (since it's new with JDK-9), and the API can't be directly emulated, as it requires java.lang.Optional. However, it's not rocket science (it is however regular expressions, so now you have problems 2.0). I'll take a crack at a class implementing JEP-223 Versions and try to get them to play with JavaVersion. The existing mapping to float is unsuited to minor versions, so there has to be some new hackery for java 9 even for two components. ( JDK 9.11 should be > JDK 9.2 ) Since there's no java minor releases yet, there's a bit of freedom. If the range of minor and patch versions is restricted to values between 0 and 99 there should be a way to make things work. Ideally the floaty stuff can be deprecated for removal before that matters. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-400) It should be possible for users to create and access extra PAX headers to tar archives
[ https://issues.apache.org/jira/browse/COMPRESS-400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062427#comment-16062427 ] Simon Spero commented on COMPRESS-400: -- Just added a commit to remote abstracted support for extended attributes. The commit message has some extended notes, which I am adding to the work log should this need to be revisited (since a lot of this stuff is not documented except in 'read the source code, not the spec'. If this needs to be revisited, the following information is useful: 1. pax header keywords should be ascii. star/gnutar (SCHILY.xattr.* ) do not check for this. libarchive/bsdtar (LIBARCHIVE.xattr.*) use URL-Encoding. 2. pax header values should be encoded as UTF-8 characters (including \0). star/gnutar (SCHILY.xattr.*) do not check for this. libarchive/bsdtar (LIBARCHIVE.xattr.*) encode values using Base64. libarchive/bsdtar will read SCHILY.xattr headers, but will not generate them. gnutar will complain about LIBARCHIVE.xattr (and any other unknown) headers and will neither encode nor decode them. I believe that star is the same. Proper support for handling extended attributes may require knowing which convention should be used on output. It may also require additional methods that handle byte[] values appropriately. Should Path based initalization of entry metadata be supported, it should be noted that Java's abstraction of extended attributes treats handles them as binary values (which must fit in a single ByteBuffer, and thus are of maximum size MAX_INT). nio file attribute views may be a suitable basis handling xattrs (and other related metadata). https://docs.oracle.com/javase/7/docs/api/java/nio/file/attribute/UserDefinedFileAttributeView.html > It should be possible for users to create and access extra PAX headers to tar > archives > --- > > Key: COMPRESS-400 > URL: https://issues.apache.org/jira/browse/COMPRESS-400 > Project: Commons Compress > Issue Type: Improvement > Components: Archivers >Reporter: Simon Spero >Priority: Minor > > It is very useful to be able to add extra PAX headers to tar entries. For > example, a tar file containing maven artifacts could have extra headers > carrying the groupid, artifactid, version, classifier, etc. > If the appropriate prefixes are used, these headers can be extracted to posix > extended attributes by gnu and bsdtar. > This change requires adding a map to TarArchiveEntry to carry the extra > headers, plus modifications to the TarArchive*Stream to save unrecognized > headers when reading, and to add any extra headers when writing. > I have created a prototype implementation, but have not merged it into my > fork of the project. I don't have full tests written, because I was using > gnutar as an oracle. > I have also ignored the issue of writing values to standard headers like > size, though since the PAX specification states that doing things like > setting size=100 (if the real size is not in fact 100) is undefined, so I'm > technically in compliance. The temptation is to do what was asked, then on > close pop up a "Were you sure?" dialog, but that's mean. I guess I could use > this to set the appropriate entry fields if doing so makes sense, but the > easiest approach is to block setting any headers that would be consumed by > the tar implementation when reading. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-413) Travis build redundantly repeats compilation and tests redundantly
[ https://issues.apache.org/jira/browse/COMPRESS-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062176#comment-16062176 ] Simon Spero commented on COMPRESS-413: -- I'll strip out the AsiExtraField foo. I just have the strategic reserve of coverage increasers for coverall-shutting-up. My main issue with Java 7 compatibility is that the primary use case now is some older android releases (which can't do Java 8 default/static methods), but android compatibility testing at its most basic requires using a different animal-sniffer scent (anything else is difficult without gradle). > Travis build redundantly repeats compilation and tests redundantly > -- > > Key: COMPRESS-413 > URL: https://issues.apache.org/jira/browse/COMPRESS-413 > Project: Commons Compress > Issue Type: Improvement > Components: Build >Affects Versions: 1.14 > Environment: Travis >Reporter: Simon Spero >Priority: Minor > Labels: CI > Fix For: 1.15 > > Original Estimate: 0h > Remaining Estimate: 0h > > The Travis build setup is suboptimal. > At the moment, code is compiled and installed by the default install phase. > Then the default build phase is executed, which compiles and runs the tests. > If the tests succeed, then the build is cleaned, recompiled, and retested; > this time with > coverage enabled. > The .travis.yml file could be changed to skip the install phase, and to run > tests with coverage during the build phase. > The coveralls plugin can be configured in the pom to not fail the build if > the service is unreachable, so forks that don't have jacoco enabled won't > always have their builds fail. > Also, the jdk switching in the trusty container seems to be not working > properly at the moment, so installing a jdk7 doesn't work properly. > These changes evolved as I was poking jenkins last night. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (COMPRESS-414) Provide API for Channel(-ish) Seekable, Possibly Read/Write, Possibly Transactional Compressors & Archivers,
Simon Spero created COMPRESS-414: Summary: Provide API for Channel(-ish) Seekable, Possibly Read/Write, Possibly Transactional Compressors & Archivers, Key: COMPRESS-414 URL: https://issues.apache.org/jira/browse/COMPRESS-414 Project: Commons Compress Issue Type: Improvement Reporter: Simon Spero It would be very useful to have APIs to support random-access, possibly read-write archives & compressors. These interfaces should probably be based around Seekable, Scatter, and Gather ByteChannels. It might be necessary or desirable to have these classes extend FileChannel, since there are some important methods that first make their appearance as abstract methods on that class. The methods of interest are the absolute positioned read and write methods, plus force/sync. The use case I'm thinking of for the the non-relative read/write would be for a random-access compressed "file", where it may be useful to keep a cache of decompressed blocks, and where a task may require access to several parts of the file simultaneously. Nesting this deeper, there might be (well, there are) a tar.xz file containing a sequence of versions of a maven artifact jar, and with an analysis process that needs access to two or more of the files simultaneously. A use case requiring read/write + transactional + sync support would be to support in-place appending new items to a tar.xz file. It can be quite useful to be able to perform such updates "in-place"; for safety this requires a log the parts of the existing file that are going to be changed, so that the contents can be recovered in the event of failure. If there is related metadata in a separate transactional resource, it is good to have both stores remain in sync. The complicated parts are the job of a transaction manager (e.g. Aries); basic XAResource support is relatively simple.There are other bits of glue that are useful for handling simple transactions with no transaction manager around, but those can be optional imports. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (COMPRESS-408) If block size is unspecified, use 512 bytes
[ https://issues.apache.org/jira/browse/COMPRESS-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Spero closed COMPRESS-408. Work done as part of COMPRESS-407 > If block size is unspecified, use 512 bytes > --- > > Key: COMPRESS-408 > URL: https://issues.apache.org/jira/browse/COMPRESS-408 > Project: Commons Compress > Issue Type: Sub-task > Components: Archivers >Reporter: Simon Spero > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-407) Validate Block and Record Sizes
[ https://issues.apache.org/jira/browse/COMPRESS-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062074#comment-16062074 ] Simon Spero commented on COMPRESS-407: -- Maxi-Min For The Not-losing-as-badly-as-possible :-) I will also close COMPRESS-408, since that sub-task got addressed as part of this one. > Validate Block and Record Sizes > --- > > Key: COMPRESS-407 > URL: https://issues.apache.org/jira/browse/COMPRESS-407 > Project: Commons Compress > Issue Type: Sub-task > Components: Archivers >Reporter: Simon Spero > Fix For: 1.15 > > > Reject record sizes not equal to 512 bytes; require block sizes to be > multiples of 512 bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-408) If block size is unspecified, use 512 bytes
[ https://issues.apache.org/jira/browse/COMPRESS-408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16062075#comment-16062075 ] Simon Spero commented on COMPRESS-408: -- Completed as part of COMPRESS-407 > If block size is unspecified, use 512 bytes > --- > > Key: COMPRESS-408 > URL: https://issues.apache.org/jira/browse/COMPRESS-408 > Project: Commons Compress > Issue Type: Sub-task > Components: Archivers >Reporter: Simon Spero > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (COMPRESS-408) If block size is unspecified, use 512 bytes
[ https://issues.apache.org/jira/browse/COMPRESS-408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Spero resolved COMPRESS-408. -- Resolution: Fixed > If block size is unspecified, use 512 bytes > --- > > Key: COMPRESS-408 > URL: https://issues.apache.org/jira/browse/COMPRESS-408 > Project: Commons Compress > Issue Type: Sub-task > Components: Archivers >Reporter: Simon Spero > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-407) Validate Block and Record Sizes
[
https://issues.apache.org/jira/browse/COMPRESS-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16060227#comment-16060227
]
Simon Spero commented on COMPRESS-407:
--
It is definitely an error to attempt to create a PAX or USTAR format file a
record size that is not 512 bytes. Any attempt to use a PAX or USTAR feature
with a record size not equal to 512 bytes will have to be modified to throw an
exception.
[The tar format documentation in libarchive|
https://github.com/libarchive/libarchive/blob/master/libarchive/tar.5] shows
that this goes all the way back to V7.
{panel:title=tar (5) }
*General Format*
A tar archive consists of a series of 512-byte records. Each file system
object requires a header record which stores basic metadata (pathname, owner,
permissions, etc.) and zero or more records containing any file data. The end
of the archive is indicated by two records consisting entirely of zero bytes.
For compatibility with tape drives that use fixed block sizes, programs
that read or write tar files always read or write a fixed number of records
with each I/O operation. These “blocks” are always a multiple of the record
size. The maximum block size supported by early implementations was 10240
bytes or 20 records. This is still the default for most implementations
although block sizes of 1MiB (2048 records) or larger are commonly used with
modern high-speed tape drives. (Note: the terms “block” and “record” here are
not entirely standard; this document follows the convention established by John
Gilmore in documenting pdtar.)
{panel}
If alternative record sizes are to be permitted then these will have to be 512
bytes or greater in size. If such uses are permitted then there needs to a
warning logged. Which means there has to be logging added.
I will add a second pull request on top of this one that adds a logging
framework, reduces the errors to a warning, and removes the tests.
As an alternative, I will add a third pull request that uses a system property
to enable invalid tar record sizes.
I would still advise applying the PR as submitted and rejecting either of the
above listed alternatives.
> Validate Block and Record Sizes
> ---
>
> Key: COMPRESS-407
> URL: https://issues.apache.org/jira/browse/COMPRESS-407
> Project: Commons Compress
> Issue Type: Sub-task
> Components: Archivers
>Reporter: Simon Spero
>
> Reject record sizes not equal to 512 bytes; require block sizes to be
> multiples of 512 bytes.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-413) Travis build redundantly repeats compilation and tests redundantly
[ https://issues.apache.org/jira/browse/COMPRESS-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059842#comment-16059842 ] Simon Spero commented on COMPRESS-413: -- Switching to trusty container, and using openjdk7 and openjdk8 as jdks [Build stats| https://travis-ci.org/apache/commons-compress/builds/245906995] - Ran for 2 min 57 sec - Total time 4 min 12 sec Note that the changed container means that caches for this build were reset. > Travis build redundantly repeats compilation and tests redundantly > -- > > Key: COMPRESS-413 > URL: https://issues.apache.org/jira/browse/COMPRESS-413 > Project: Commons Compress > Issue Type: Improvement > Components: Build >Affects Versions: 1.14 > Environment: Travis >Reporter: Simon Spero >Priority: Minor > Labels: CI > Fix For: 1.15 > > Original Estimate: 0h > Remaining Estimate: 0h > > The Travis build setup is suboptimal. > At the moment, code is compiled and installed by the default install phase. > Then the default build phase is executed, which compiles and runs the tests. > If the tests succeed, then the build is cleaned, recompiled, and retested; > this time with > coverage enabled. > The .travis.yml file could be changed to skip the install phase, and to run > tests with coverage during the build phase. > The coveralls plugin can be configured in the pom to not fail the build if > the service is unreachable, so forks that don't have jacoco enabled won't > always have their builds fail. > Also, the jdk switching in the trusty container seems to be not working > properly at the moment, so installing a jdk7 doesn't work properly. > These changes evolved as I was poking jenkins last night. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-413) Travis build redundantly repeats compilation and tests redundantly
[ https://issues.apache.org/jira/browse/COMPRESS-413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16059790#comment-16059790 ] Simon Spero commented on COMPRESS-413: -- Compare: [Master| https://travis-ci.org/apache/commons-compress/builds/244058294] - Ran for 6 min 13 sec - Total time 14 min 15 sec [PR| https://travis-ci.org/apache/commons-compress/builds/245887614 ] - Ran for 4 min 31 sec - Total time 8 min 22 sec Further, Oracle JDK 7 hit EOL *over 2 years* ago! OpenJDK 7 has seen only sporadic patching, and is no longer part of any ubuntu LTS release. It is probably worth only testing under at most one of these environments (also, current tests run against really, really old versions of jdk8) > Travis build redundantly repeats compilation and tests redundantly > -- > > Key: COMPRESS-413 > URL: https://issues.apache.org/jira/browse/COMPRESS-413 > Project: Commons Compress > Issue Type: Improvement > Components: Build >Affects Versions: 1.14 > Environment: Travis >Reporter: Simon Spero >Priority: Minor > Labels: CI > Fix For: 1.15 > > Original Estimate: 0h > Remaining Estimate: 0h > > The Travis build setup is suboptimal. > At the moment, code is compiled and installed by the default install phase. > Then the default build phase is executed, which compiles and runs the tests. > If the tests succeed, then the build is cleaned, recompiled, and retested; > this time with > coverage enabled. > The .travis.yml file could be changed to skip the install phase, and to run > tests with coverage during the build phase. > The coveralls plugin can be configured in the pom to not fail the build if > the service is unreachable, so forks that don't have jacoco enabled won't > always have their builds fail. > Also, the jdk switching in the trusty container seems to be not working > properly at the moment, so installing a jdk7 doesn't work properly. > These changes evolved as I was poking jenkins last night. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (COMPRESS-413) Travis build redundantly repeats compilation and tests redundantly
Simon Spero created COMPRESS-413: Summary: Travis build redundantly repeats compilation and tests redundantly Key: COMPRESS-413 URL: https://issues.apache.org/jira/browse/COMPRESS-413 Project: Commons Compress Issue Type: Improvement Components: Build Affects Versions: 1.14 Environment: Travis Reporter: Simon Spero Priority: Minor Fix For: 1.15 The Travis build setup is suboptimal. At the moment, code is compiled and installed by the default install phase. Then the default build phase is executed, which compiles and runs the tests. If the tests succeed, then the build is cleaned, recompiled, and retested; this time with coverage enabled. The .travis.yml file could be changed to skip the install phase, and to run tests with coverage during the build phase. The coveralls plugin can be configured in the pom to not fail the build if the service is unreachable, so forks that don't have jacoco enabled won't always have their builds fail. Also, the jdk switching in the trusty container seems to be not working properly at the moment, so installing a jdk7 doesn't work properly. These changes evolved as I was poking jenkins last night. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-281) Archives don't contain resource fork on Macs
[
https://issues.apache.org/jira/browse/COMPRESS-281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16056488#comment-16056488
]
Simon Spero commented on COMPRESS-281:
--
Things changed so that now this info is stored as an xattr, with the legacy
"..namedfork/rsrc" being just another attribute. Before, extended attributes
were stored as a special fork.
In
{{#define XATTR_RESOURCEFORK_NAME "com.apple.ResourceFork"}}
The MacOS release of openjdk doesn't an extended attribute (UserDefined) file
view. An implementation would be quite similar to the Linux version; however,
no-one has added it since JDK 7 came out; support isn't included in JDK-9
either. You could implement the required code, and write a shim around default
FileSystemProvider to do the extra attribute wrangling.
The default provider can be replaced using the mechanism discussed in
[FileSystems::getDefault |
https://docs.oracle.com/javase/7/docs/api/java/nio/file/FileSystems.html#getDefault()],
which requires setting a system property. This probably has to be done on the
command line, since by the time any user code is executed the default file
system will probably have been instantiated. Note that the value of the
property can be a list of classes; these will be instantiated in the order in
which they are given. Each instance will be passed as an argument to the
next, with the first class being passed an instance of the system default
classloader; the final class will become the actual default.
The wrapper implementation delegate most methods to the passed in provider.
The obvious places where changes are needed
are in the attribute related methods. The wrapper also needs to create a
mostly delegating instance of FileSystem, with changes to return the right
provider and add "user" to the list of supported file views.
It probably also needs to provide an implementation of
UserDefinedFileAttributeView, and some native code to invoke the appropriate
system call.
The Linux implementation, which needs a few extra system call wrappers in its
Dispatcher:
[LinuxUserDefinedFileAttributeView.java|http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/e03f9868f7df/src/solaris/classes/sun/nio/fs/LinuxUserDefinedFileAttributeView.java]
which relies on a few NIO methods in
[LinuxNativeDispatcher.c|
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/e03f9868f7df/src/solaris/native/sun/nio/fs/LinuxNativeDispatcher.c#l83
]
Or the Solaris implementation, which uses a solaris-specific flag - the jdk 8
sources are a little solaris biased, as you may have spotted from the url path.
[SolarisUserDefineFileAttributeView.java|http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/e03f9868f7df/src/solaris/classes/sun/nio/fs/SolarisUserDefinedFileAttributeView.java]
This is probably a bit out of scope.
An alternative disgustingly hacky approach would be to use /usr/bin/xattr,
which can make the system calls, and can read/dump the contents of an attribute
in hex. Did I mention that /usr/bin/xattr is a python script? Feel the
performance flow through you.
> Archives don't contain resource fork on Macs
>
>
> Key: COMPRESS-281
> URL: https://issues.apache.org/jira/browse/COMPRESS-281
> Project: Commons Compress
> Issue Type: Wish
>Reporter: Dimple Rohara
>Priority: Critical
>
> Currently, we are able to create archive on windows. But for Mac,it doesn't
> save the resource fork information. I wish there was more compatibility among
> the archives created on different Operating systems and could be used easily
> cross-platform.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-411) TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
[ https://issues.apache.org/jira/browse/COMPRESS-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053720#comment-16053720 ] Simon Spero commented on COMPRESS-411: -- The mailing list suffers from the billions of duplicated emails from jira vs github (featuring coveralls). Wait - I think I see your point. The volume is such that I have missed important emails that need reply (eg I need to reply to an email of yours about how users would identify versions to report in OSGI situations (it really needs illustration from Kafka /Felix (which in turn requires a faux compress to add multi versions, and multi issues.) The conclusolory statement that "its not a problem" is not helpful. Showing what would be sent is. The real confusion comes from bad declarations ; other issues comes from failed resolution. This former can be pom tracing hell ; the latter is interpreting constraint satisfaction problem errors. Both get better with more precise versions :) But that's just me saying that. > TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary > --- > > Key: COMPRESS-411 > URL: https://issues.apache.org/jira/browse/COMPRESS-411 > Project: Commons Compress > Issue Type: Bug > Components: Archivers >Affects Versions: 1.14 >Reporter: Simon Spero >Priority: Minor > Fix For: 1.15 > > > if the length < 9, formatLongBinary is executed, then overwritten by the > results of formatBigIntegerBinary. > If the results are not ignored, a unit test would fail. > Also, do the binary hacks need to support negative numbers? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-411) TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
[ https://issues.apache.org/jira/browse/COMPRESS-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053266#comment-16053266 ] Simon Spero commented on COMPRESS-411: -- Everything I've opened so far has been something I'm working on, or will be shortly :-) Annoyingly, JIRA doesn't support workflow triggers from github integration until a Pull-Request is created (which the examples discuss as being suitable for moving into an "In Review" state (merging a PR is the closeout state). I'm not sure if that works if the merge is done wrt the asf git (I've always just assumed the github repository is a one-way mirror). What I really want is some way to only not receive a billion different notifications for every commit or comment - I really wish I hadn't found the rat issue that was breaking coveralls :-) [https://confluence.atlassian.com/adminjiraserver071/configuring-workflow-triggers-802592811.html] > TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary > --- > > Key: COMPRESS-411 > URL: https://issues.apache.org/jira/browse/COMPRESS-411 > Project: Commons Compress > Issue Type: Bug > Components: Archivers >Affects Versions: 1.14 >Reporter: Simon Spero >Priority: Minor > Fix For: 1.15 > > > if the length < 9, formatLongBinary is executed, then overwritten by the > results of formatBigIntegerBinary. > If the results are not ignored, a unit test would fail. > Also, do the binary hacks need to support negative numbers? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (COMPRESS-411) TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
[
https://issues.apache.org/jira/browse/COMPRESS-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053045#comment-16053045
]
Simon Spero edited comment on COMPRESS-411 at 6/18/17 2:19 AM:
---
I have a full set of fixes (apart from checking when decoding that for base256
values the most significant bytes for lengths > 8 octets are just sign
extension (i.e. 0x080 + 0x00{p} or 0xff + 0xff{p}.
The changes are mixed in with a bunch of other changes to TarUtils that are
needed for direct encoding/decoding to ByteBuffers. I needed to make sure the
byte array versions were behaving correctly as they are needed for tests.
Unfortunately most of the methods in TarUtils are declared public, as is the
class, so the API has to be maintained, but my plan is to deprecate them, move
their bodies into a package private class in test, and delegate the
functionality to the new methods, which can remain package private (I doubt
there is any especially pressing need for classes outside of the tar
implementation to call these methods.
was (Author: sesuncedu):
I have a full set of fixes (apart from checking when decoding that for base256
values the most significant bytes for lengths > 8 octets are just sign
extension (i.e. 0x080 + 0x00{p} or 0xff + 0xff{p}.
The changes are mixed in with a bunch of other changes to TarUtils that are
needed for direct encoding/decoding to ByteBuffers.
> TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
> ---
>
> Key: COMPRESS-411
> URL: https://issues.apache.org/jira/browse/COMPRESS-411
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.14
>Reporter: Simon Spero
>Priority: Minor
> Fix For: 1.15
>
>
> if the length < 9, formatLongBinary is executed, then overwritten by the
> results of formatBigIntegerBinary.
> If the results are not ignored, a unit test would fail.
> Also, do the binary hacks need to support negative numbers?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-411) TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
[
https://issues.apache.org/jira/browse/COMPRESS-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053047#comment-16053047
]
Simon Spero commented on COMPRESS-411:
--
Can you grant me {{Assign issues}}, {{Assignable user}}, {{Transition issues}},
and {{Resolve issues}} on the project (assuming that the review and
commit/merge phase of the workflow take place starting in RESOLVED). This
doesn't include {{Close issues}}.
[https://confluence.atlassian.com/adminjiraserver073/managing-project-permissions-861253293.html]
I keep wanting to self assign and mark "IN PROGRESS", so that effort isn't
duplicated.
I _thing_ that requires assign,assignable & transition, though it depends on
how the workflow is set up.
> TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
> ---
>
> Key: COMPRESS-411
> URL: https://issues.apache.org/jira/browse/COMPRESS-411
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.14
>Reporter: Simon Spero
>Priority: Minor
> Fix For: 1.15
>
>
> if the length < 9, formatLongBinary is executed, then overwritten by the
> results of formatBigIntegerBinary.
> If the results are not ignored, a unit test would fail.
> Also, do the binary hacks need to support negative numbers?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-411) TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
[
https://issues.apache.org/jira/browse/COMPRESS-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053045#comment-16053045
]
Simon Spero commented on COMPRESS-411:
--
I have a full set of fixes (apart from checking when decoding that for base256
values the most significant bytes for lengths > 8 octets are just sign
extension (i.e. 0x080 + 0x00{p} or 0xff + 0xff{p}.
The changes are mixed in with a bunch of other changes to TarUtils that are
needed for direct encoding/decoding to ByteBuffers.
> TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
> ---
>
> Key: COMPRESS-411
> URL: https://issues.apache.org/jira/browse/COMPRESS-411
> Project: Commons Compress
> Issue Type: Bug
> Components: Archivers
>Affects Versions: 1.14
>Reporter: Simon Spero
>Priority: Minor
> Fix For: 1.15
>
>
> if the length < 9, formatLongBinary is executed, then overwritten by the
> results of formatBigIntegerBinary.
> If the results are not ignored, a unit test would fail.
> Also, do the binary hacks need to support negative numbers?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (COMPRESS-411) TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
[ https://issues.apache.org/jira/browse/COMPRESS-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16052382#comment-16052382 ] Simon Spero commented on COMPRESS-411: -- I have fixed various issues in the handling of base256 encoded long numbers (I saw that negative numbers are used for STAR timestamps from before the epoch). One of the issues involved the maximum size of a number. The number of bits available to represent the number is one less than the total number of bits available. Thus, an eight byte field can hold up to a 63 bit 2's-complement integer. Further, the range of such a number is from -2 ^n-1^ to 2 ^n-1^ -1 , which can cause range overflows. > TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary > --- > > Key: COMPRESS-411 > URL: https://issues.apache.org/jira/browse/COMPRESS-411 > Project: Commons Compress > Issue Type: Bug > Components: Archivers >Affects Versions: 1.14 >Reporter: Simon Spero >Priority: Minor > > if the length < 9, formatLongBinary is executed, then overwritten by the > results of formatBigIntegerBinary. > If the results are not ignored, a unit test would fail. > Also, do the binary hacks need to support negative numbers? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-400) It should be possible for users to create and access extra PAX headers to tar archives
[ https://issues.apache.org/jira/browse/COMPRESS-400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16050726#comment-16050726 ] Simon Spero commented on COMPRESS-400: -- If you have xattrs in the file system, and you tar with - - xattrs set, they get added with that header by Gnu tar. Libarchive uses a different prefix but reads both. Gnu tar complains if the libarchive headers are set. There's some Extra Extended magic for names and values that I didn't code up, because there are two sets of hacks that aren't standardized, and it's only really important when extracting to the fs, but I left the tech debt. Also, not sure if I want to handle binary values, but ought to. Annoyed enough fixing bugs in the star mode base256 stuff :) On Thu, Jun 15, 2017, 12:08 PM Stefan Bodewig (JIRA) > It should be possible for users to create and access extra PAX headers to tar > archives > --- > > Key: COMPRESS-400 > URL: https://issues.apache.org/jira/browse/COMPRESS-400 > Project: Commons Compress > Issue Type: Improvement > Components: Archivers >Reporter: Simon Spero >Priority: Minor > > It is very useful to be able to add extra PAX headers to tar entries. For > example, a tar file containing maven artifacts could have extra headers > carrying the groupid, artifactid, version, classifier, etc. > If the appropriate prefixes are used, these headers can be extracted to posix > extended attributes by gnu and bsdtar. > This change requires adding a map to TarArchiveEntry to carry the extra > headers, plus modifications to the TarArchive*Stream to save unrecognized > headers when reading, and to add any extra headers when writing. > I have created a prototype implementation, but have not merged it into my > fork of the project. I don't have full tests written, because I was using > gnutar as an oracle. > I have also ignored the issue of writing values to standard headers like > size, though since the PAX specification states that doing things like > setting size=100 (if the real size is not in fact 100) is undefined, so I'm > technically in compliance. The temptation is to do what was asked, then on > close pop up a "Were you sure?" dialog, but that's mean. I guess I could use > this to set the appropriate entry fields if doing so makes sense, but the > easiest approach is to block setting any headers that would be consumed by > the tar implementation when reading. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (COMPRESS-411) TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary
Simon Spero created COMPRESS-411: Summary: TarUtils.formatLongOctalOrBinaryBytes never uses result of formatLongBinary Key: COMPRESS-411 URL: https://issues.apache.org/jira/browse/COMPRESS-411 Project: Commons Compress Issue Type: Bug Components: Archivers Affects Versions: 1.14 Reporter: Simon Spero Priority: Minor if the length < 9, formatLongBinary is executed, then overwritten by the results of formatBigIntegerBinary. If the results are not ignored, a unit test would fail. Also, do the binary hacks need to support negative numbers? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (COMPRESS-407) Validate Block and Record Sizes
[ https://issues.apache.org/jira/browse/COMPRESS-407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16046995#comment-16046995 ] Simon Spero commented on COMPRESS-407: -- I've rebased 405 and 407 onto my personal 403 branch. It make make more sense to address all these changes as a whole. > Validate Block and Record Sizes > --- > > Key: COMPRESS-407 > URL: https://issues.apache.org/jira/browse/COMPRESS-407 > Project: Commons Compress > Issue Type: Sub-task > Components: Archivers >Reporter: Simon Spero > > Reject record sizes not equal to 512 bytes; require block sizes to be > multiples of 512 bytes. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (COMPRESS-410) Remove ZipEncoding implementations introduced to work around problems in jdk 1.4
Simon Spero created COMPRESS-410: Summary: Remove ZipEncoding implementations introduced to work around problems in jdk 1.4 Key: COMPRESS-410 URL: https://issues.apache.org/jira/browse/COMPRESS-410 Project: Commons Compress Issue Type: Improvement Components: Archivers Affects Versions: 1.14 Reporter: Simon Spero Priority: Minor There are several implementations of ZipEncoding that are provided to work around missing character sets in JDK 1.4. Since commons-compress requires JDK7 or later, this code is no longer needed. As the classes are package private, they can be removed without affecting the API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (COMPRESS-405) Create Fixed Length Block OutputStream /Channel
[ https://issues.apache.org/jira/browse/COMPRESS-405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Simon Spero updated COMPRESS-405: - Complete, PR generated > Create Fixed Length Block OutputStream /Channel > --- > > Key: COMPRESS-405 > URL: https://issues.apache.org/jira/browse/COMPRESS-405 > Project: Commons Compress > Issue Type: Sub-task > Components: Archivers >Reporter: Simon Spero > Fix For: 1.15 > > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (COMPRESS-409) Remove buffering code from TarArchiveOutputStream. Replace by record-length padding and use of FixedLengthBlockOutputStream
Simon Spero created COMPRESS-409: Summary: Remove buffering code from TarArchiveOutputStream. Replace by record-length padding and use of FixedLengthBlockOutputStream Key: COMPRESS-409 URL: https://issues.apache.org/jira/browse/COMPRESS-409 Project: Commons Compress Issue Type: Sub-task Reporter: Simon Spero -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (COMPRESS-408) If block size is unspecified, use 512 bytes
Simon Spero created COMPRESS-408: Summary: If block size is unspecified, use 512 bytes Key: COMPRESS-408 URL: https://issues.apache.org/jira/browse/COMPRESS-408 Project: Commons Compress Issue Type: Sub-task Reporter: Simon Spero -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (COMPRESS-407) Validate Block and Record Sizes
Simon Spero created COMPRESS-407: Summary: Validate Block and Record Sizes Key: COMPRESS-407 URL: https://issues.apache.org/jira/browse/COMPRESS-407 Project: Commons Compress Issue Type: Sub-task Reporter: Simon Spero Reject record sizes not equal to 512 bytes; require block sizes to be multiples of 512 bytes. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (COMPRESS-406) BUILDING.md is missing license header
Simon Spero created COMPRESS-406: Summary: BUILDING.md is missing license header Key: COMPRESS-406 URL: https://issues.apache.org/jira/browse/COMPRESS-406 Project: Commons Compress Issue Type: Bug Reporter: Simon Spero There is no license header in BUILDING.md. This angers the rat -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (COMPRESS-405) Create Fixed Length Block OutputStream /Channel
Simon Spero created COMPRESS-405: Summary: Create Fixed Length Block OutputStream /Channel Key: COMPRESS-405 URL: https://issues.apache.org/jira/browse/COMPRESS-405 Project: Commons Compress Issue Type: Sub-task Reporter: Simon Spero Fix For: 1.15 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (COMPRESS-404) Add Path equivalents to TarArchiveEntry, with methods using File delegating to equivalents
[ https://issues.apache.org/jira/browse/COMPRESS-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045685#comment-16045685 ] Simon Spero commented on COMPRESS-404: -- On a Linux box, getting unix:* for "/" - rdev=0 - ctime=2017-06-07T15:01:25.692266Z - nlink=28 - fileKey=(dev=832,ino=2) - dev=2098 - ino=2 - mode=16877 - uid=0 - gid=0 - size=4096 - owner=root - group=root - lastAccessTime=2017-06-09T20:23:38.182067Z - lastModifiedTime=2017-06-07T15:01:25.692266Z - creationTime=2017-06-07T15:01:25.692266Z - isRegularFile=false - isSymbolicLink=false - permissions=[OTHERS_READ, OTHERS_EXECUTE, GROUP_EXECUTE, OWNER_EXECUTE, OWNER_WRITE, GROUP_READ, OWNER_READ] - isOther=false - isDirectory=true > Add Path equivalents to TarArchiveEntry, with methods using File delegating > to equivalents > -- > > Key: COMPRESS-404 > URL: https://issues.apache.org/jira/browse/COMPRESS-404 > Project: Commons Compress > Issue Type: Sub-task >Reporter: Simon Spero >Priority: Minor > > (This is a reasonable place to start, as Paths give better access to > tar-relevant metadata on unix system). > Symlink info is easier to obtain > Unix based Paths allow access to useful attributes under "unix:*" > - numeric uid and gid > - string owner and group names > - mtime,ctime,atime > - numeric mode > - numeric dev and inode > - numeric rdev > - Linux, Solaris and Windows allow access to extended attributes > (MacOS X xattrs aren't supported as of jdk 9) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (COMPRESS-404) Add Path equivalents to TarArchiveEntry, with methods using File delegating to equivalents
[ https://issues.apache.org/jira/browse/COMPRESS-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045685#comment-16045685 ] Simon Spero edited comment on COMPRESS-404 at 6/10/17 8:28 PM: --- On a Linux box, getting "unix:*" for "/" - rdev=0 - ctime=2017-06-07T15:01:25.692266Z - nlink=28 - fileKey=(dev=832,ino=2) - dev=2098 - ino=2 - mode=16877 - uid=0 - gid=0 - size=4096 - owner=root - group=root - lastAccessTime=2017-06-09T20:23:38.182067Z - lastModifiedTime=2017-06-07T15:01:25.692266Z - creationTime=2017-06-07T15:01:25.692266Z - isRegularFile=false - isSymbolicLink=false - permissions=[OTHERS_READ, OTHERS_EXECUTE, GROUP_EXECUTE, OWNER_EXECUTE, OWNER_WRITE, GROUP_READ, OWNER_READ] - isOther=false - isDirectory=true was (Author: sesuncedu): On a Linux box, getting unix:* for "/" - rdev=0 - ctime=2017-06-07T15:01:25.692266Z - nlink=28 - fileKey=(dev=832,ino=2) - dev=2098 - ino=2 - mode=16877 - uid=0 - gid=0 - size=4096 - owner=root - group=root - lastAccessTime=2017-06-09T20:23:38.182067Z - lastModifiedTime=2017-06-07T15:01:25.692266Z - creationTime=2017-06-07T15:01:25.692266Z - isRegularFile=false - isSymbolicLink=false - permissions=[OTHERS_READ, OTHERS_EXECUTE, GROUP_EXECUTE, OWNER_EXECUTE, OWNER_WRITE, GROUP_READ, OWNER_READ] - isOther=false - isDirectory=true > Add Path equivalents to TarArchiveEntry, with methods using File delegating > to equivalents > -- > > Key: COMPRESS-404 > URL: https://issues.apache.org/jira/browse/COMPRESS-404 > Project: Commons Compress > Issue Type: Sub-task >Reporter: Simon Spero >Priority: Minor > > (This is a reasonable place to start, as Paths give better access to > tar-relevant metadata on unix system). > Symlink info is easier to obtain > Unix based Paths allow access to useful attributes under "unix:*" > - numeric uid and gid > - string owner and group names > - mtime,ctime,atime > - numeric mode > - numeric dev and inode > - numeric rdev > - Linux, Solaris and Windows allow access to extended attributes > (MacOS X xattrs aren't supported as of jdk 9) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (COMPRESS-403) Block and Record Size issues in TarArchiveOutputStream
[ https://issues.apache.org/jira/browse/COMPRESS-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045677#comment-16045677 ] Simon Spero commented on COMPRESS-403: -- At the moment the implementation is doing the wrong thing when it doesn't matter, then adding padding at the end to make it look like you were doing the right thing all along, except that if doing the right thing were the right thing to do, it would be too late by the time you did it :) Did that make things worse enough? :-) If you are insufficiently confused, read the [Gnu Tar Documentation on Blocking|https://www.gnu.org/software/tar/manual/html_section/tar_77.html#SEC159], where the author explains why the gnu tar options and documentation uses block and record to refer to what posix calls record and block... h3. Specification # The specification says that writes to the output stream may be in block-size chunks (which should be multiples of 512, up to 32K), and _if_ this is done, _and_ the destination is a tape drive, the result must be a single block. It also says that the last physical block must be the # The specification says that the final block should be padded out to a full block size, after the end-of-archive marker has been written. # The specification says that implementations may modify the block-size, but should be careful about portability. h3. Current Implementation # The current implementation writes to the output stream in 512 byte chunks (actually record size). # The record size is supposed to be always 512 bytes, but the implementation allows this to be altered. # If, after writing the end-of-archive marker (2x512 records filled with nulls), the file size is not a multiple of the specified block size, the implementation adds sufficient padding to bring it up to the appropriate size. # The output from the implementation goes in to an arbitrary OutputStream, which most likely is doing its own buffering or compressing. # The implementation can't tell if the output is a tape-drive. If anyone has been using the implementation so far, relying on block sizes being defaulted correctly, then it hasn't been working :) # Header records and file data have to be padded to units of 512 bytes. h3. Changes. # The implementation should throw an IllegalArgumentException if an attempt is made to set a record size other than 512 bytes # The implementation should reject block sizes that aren't multiples of 512. # If a block-size is not specified, the implementation should pick a default (since it isn't known if the archive will be PAX or ustar at that point, there it is simultaneously 5120 and 10240 bytes), then immediately "alter" it to 512 bytes :-) # The TarArchiveOutputStream should not handle buffering; it should only perform padding for headers and data files to 512 byte boundaries. # For cases where an explicit blocking size is requested, the implementation should wrap the destination OutputStream with a new class that handles only fixed block length buffering. This class should ignore flush(), and pad a partial buffer the full block length on close(). This will provide correct behavior where block sizes greater than one record are needed # If the minimal block size of 512 bytes is used, such that the end-of-archive marker is the final 1024 bytes of output, this should be recorded in an extended header at the beginning of the archive. This will allow faster appending to archives that support some degree of random access. # A suitable header name might be COMMONS-COMPRESS.blockSize, with the value being the block size in bytes as a utf-8 decimal string. Or roman numerals. Does that sound like a reasonable work-plan? If so, I'll take a crack at it. Interestingly, bsdtar and gnutar will add the last-block padding after any compression headers. Except that gnutar will skip the padding if the file is being written directly to disk. Actually, that's not very interesting at all. But it does provide further justification for taking the blocking out of Tar's hands, and putting it in a separate class for all the (zero?) people using COMMONS-COMPRESS to write directly to tape drives. > Block and Record Size issues in TarArchiveOutputStream > > > Key: COMPRESS-403 > URL: https://issues.apache.org/jira/browse/COMPRESS-403 > Project: Commons Compress > Issue Type: Improvement > Components: Archivers >Affects Versions: 1.14 >Reporter: Simon Spero >Priority: Minor > > According to the pax spec > [§4.100.13.01| > http://pubs.opengroup.org/onlinepubs/009695399/utilities/pax.html#tag_04_100_13_01] > > bq. A pax archive tape or file pr
[jira] [Created] (COMPRESS-404) Add Path equivalents to TarArchiveEntry, with methods using File delegating to equivalents
Simon Spero created COMPRESS-404: Summary: Add Path equivalents to TarArchiveEntry, with methods using File delegating to equivalents Key: COMPRESS-404 URL: https://issues.apache.org/jira/browse/COMPRESS-404 Project: Commons Compress Issue Type: Sub-task Reporter: Simon Spero Priority: Minor (This is a reasonable place to start, as Paths give better access to tar-relevant metadata on unix system). Symlink info is easier to obtain Unix based Paths allow access to useful attributes under "unix:*" - numeric uid and gid - string owner and group names - mtime,ctime,atime - numeric mode - numeric dev and inode - numeric rdev - Linux, Solaris and Windows allow access to extended attributes (MacOS X xattrs aren't supported as of jdk 9) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (COMPRESS-403) Block and Record Size issues in TarArchiveOutputStream
Simon Spero created COMPRESS-403: Summary: Block and Record Size issues in TarArchiveOutputStream Key: COMPRESS-403 URL: https://issues.apache.org/jira/browse/COMPRESS-403 Project: Commons Compress Issue Type: Improvement Components: Archivers Affects Versions: 1.14 Reporter: Simon Spero Priority: Minor According to the pax spec [§4.100.13.01| http://pubs.opengroup.org/onlinepubs/009695399/utilities/pax.html#tag_04_100_13_01] bq. A pax archive tape or file produced in the -x pax format shall contain a series of blocks. The physical layout of the archive shall be identical to the ustar format [§ 4.100.13.06| http://pubs.opengroup.org/onlinepubs/009695399/utilities/pax.html#tag_04_100_13_06] bq. A ustar archive tape or file shall contain a series of logical records. Each logical record shall be a fixed-size logical record of 512 octets. ... bq. The logical records *may* be grouped for physical I/O operations, as described under the -b blocksize and -x ustar options. Each group of logical records *may* be written with a single operation equivalent to the write() function. On magnetic tape, the result of this write *shall* be a single tape physical block. The last physical block *shall* always be the full size, so logical records after the two zero logical records *may* contain undefined data. .bq pax. The default blocksize for this format for character special archive files *shall* be 5120. Implementations *shall* support all blocksize values less than or equal to 32256 that are multiples of 512. ustar. The default blocksize for this format for character special archive files *shall* be 10240. Implementations *shall* support all blocksize values less than or equal to 32256 that are multiples of 512. .bq Implementations are permitted to modify the block-size value based on the archive format or the device to which the archive is being written. This is to provide implementations with the opportunity to take advantage of special types of devices, and it should not be used without a great deal of consideration as it almost certainly decreases archive portability. The current implementation of TarArchiveOutputStream # Allows the logical record size to be altered # Has a default block size of 10240 # has two separate logical-record size buffers, and frequently double buffers in order to write to the wrapped outputstream in units of a logical record, rather than a physical block. I would hazard a guess that very few users commons-compress are writing directly to a tape drive, where the block-size is of great import. It is also not possible to guarantee that a subordinate output stream won't buffer in chunks of a different size (5120 and 10240 bytes aren't ideal for modern hard drives with 4096 byte sectors, or filesystems like ZFS with a default recordsize of 128K). The main effect of the record and block size have is the extra padding they require. For the purposes of the java output device, the optimal blocksize to modify to is probably just a single record; since all implementations must handle 512 byte blocks, and must detect block size on input (or simulate same), this cannot affect compatibility. Fixed length blocking in multiples of 512 Bytes can be supported by wrapping the destination output stream in a modified BufferedOutputStream that does not permit flushing of partial blocks, and pads on close. This would only be used as necessary. If a record size of 512 bytes is being used, it could be useful to store that information in an extended header at the start of the file. That allows for in-place appending to an archive without having to read the entire archive first (as long as the original end-of-archive location is journaled to support recovery). There is even an advantage for xz compressed files, as every block but the last can be copied without having to decompress then recompress, In the latter scenario, it would be useful to be able to signal to the subordinate layer to start a new block before writing the final 1024 nulls; in that situation, either a new block can be started overwriting the EOA and xz index blocks, with the saved index info saved at the end; or the block immediately preceding the EOA markers can be decompressed and recompressed, which will rebuild the dictionary and index structures to allow the block to be continued. That's a different issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
