[jira] [Resolved] (DERBY-6821) Include XMLOptimizerTraceTest in XMLSuite

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton resolved DERBY-6821.

   Resolution: Fixed
Fix Version/s: 10.12.0.0

I've completed the work I intended for this issue.

> Include XMLOptimizerTraceTest in XMLSuite
> -
>
> Key: DERBY-6821
> URL: https://issues.apache.org/jira/browse/DERBY-6821
> Project: Derby
>  Issue Type: Sub-task
>  Components: Test
>    Reporter: Bryan Pendleton
>Priority: Minor
> Fix For: 10.12.0.0
>
> Attachments: moveTest.diff
>
>
> There is a sub-suite called XMLSuite that runs the XML tests,
> but it does not currently run the XMLOptimizerTraceTest.
> Since that test suite is a significant quite for exercising
> Derby's XML support, it seems worth including that suite in
> the XMLSuite.
> This might require a bit of suite re-arrangement, to make sure
> that we don't run XMLOptimizerTraceTest multiple times during larger suites.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6821) Include XMLOptimizerTraceTest in XMLSuite

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6821:
---
Attachment: moveTest.diff

It seems straightforward to move the XMLOptimizerTraceTest
class from o.a.derbyTesting.functionTests.tests.lang._Suite
to o.a.derbyTesting.functionTests.suites.XMLSuite. Since
the latter is included by the former, there is no change in
test execution coverage, just a change in the ordering of
the tests as XMLOptimizerTraceTest moves slightly in the
overall ordering of the tests in the lang suite.

Attached proposed moveTest.diff

> Include XMLOptimizerTraceTest in XMLSuite
> -
>
> Key: DERBY-6821
> URL: https://issues.apache.org/jira/browse/DERBY-6821
> Project: Derby
>  Issue Type: Sub-task
>  Components: Test
>    Reporter: Bryan Pendleton
>Priority: Minor
> Attachments: moveTest.diff
>
>
> There is a sub-suite called XMLSuite that runs the XML tests,
> but it does not currently run the XMLOptimizerTraceTest.
> Since that test suite is a significant quite for exercising
> Derby's XML support, it seems worth including that suite in
> the XMLSuite.
> This might require a bit of suite re-arrangement, to make sure
> that we don't run XMLOptimizerTraceTest multiple times during larger suites.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6801:
---
Attachment: unsealDerbyNetJar.diff

My test failures involved running with jars, and it turns out
that we also need to unseal o.a.d.shared.common.error
in derbynet.jar, because the latest patches introduce use
of that package from derbynet.jar classes.

Since o.a.d.shared.common.error classes are designed
to be shared freely among both client and server code,
this is as expected, so the attached unsealDerbyNetJar.diff
unseals the package in that jar.

And now the derbynet tests are running better on my system.

Abhinav, have you had an opportunity to investigate the
revised MessageUtils patch I attached yesterday?

> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: MessageUtilsBryan.diff, MessageUtils_2.diff, 
> MessageUtils_3.diff, MessageUtils_StandardEx_1.diff, 
> MessageUtils_StandardExcep.diff, build.out, messageUtils.diff, 
> messageUtils_1.diff, subPart2.diff, unsealDerbyNetJar.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14630668#comment-14630668
 ] 

Bryan Pendleton commented on DERBY-6801:


The network tests do NOT run cleanly with my patch. Clearly more work must
be done. But hopefully it points to a possible way forward, and hopefully a 
close
reading of the patch will make it clear where the problem is.

> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: MessageUtilsBryan.diff, MessageUtils_2.diff, 
> MessageUtils_3.diff, MessageUtils_StandardEx_1.diff, 
> MessageUtils_StandardExcep.diff, build.out, messageUtils.diff, 
> messageUtils_1.diff, subPart2.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6801:
---
Attachment: MessageUtilsBryan.diff

Hi Abhinav, thank you for posting your patch. Once I looked at
your code more closely, I realized that the refactoring of code
in DRDAConnThread was a bit more subtle.

The important part is that we want to get the code which
encodes the messageId and the message arguments into
the network message buffer into a shared location, because
it is that encoding that needs to be shared by the client and
server logic.

But that is not ALL of buildTokenizedSqlerrmc(), it is JUST the
part that endoes that data using the SQLERRMC_TOKEN_DELIMITER.

Attached is "MessageUtilsBryan.diff", which takes a slightly
different re-factoring of DRDAConnThread and now builds
cleanly on my system.

You'll notice that my patch is significantly smaller than your last patch.

This is because I made my patch against the head of the Subversion trunk,
and so I didn't include all the changes related to ShutdownException.

Please either (a) restore your current sandbox to a clean form before
taking my patch, or (b) create a brand new sandbox from the head of
the trunk, and then my patch should apply and build cleanly for you.

I think this is exciting progress, and we're getting quite close with
this patch, so I hope you can give it a try and let me know what you think!

> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: MessageUtilsBryan.diff, MessageUtils_2.diff, 
> MessageUtils_3.diff, MessageUtils_StandardEx_1.diff, 
> MessageUtils_StandardExcep.diff, build.out, messageUtils.diff, 
> messageUtils_1.diff, subPart2.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6741) User code can get the ContextManager from an EmbedConnection

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14627498#comment-14627498
 ] 

Bryan Pendleton commented on DERBY-6741:


Indeed, that makes the test reliable for me.

> User code can get the ContextManager from an EmbedConnection
> 
>
> Key: DERBY-6741
> URL: https://issues.apache.org/jira/browse/DERBY-6741
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC, Services
>Reporter: Rick Hillegas
>Assignee: Rick Hillegas
> Fix For: 10.12.0.0
>
> Attachments: derby-6741-01-aa-usederbyinternals.diff, derby.log, 
> error-stacktrace.out
>
>
> EmbedConnection.getContextManager() is a public method. Exposing internals 
> like the ContextManager is a security risk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6807:
---
Attachment: sqlxmlutil.diff

Attached sqlxmlutil.diff is a proposed fix for this issue for
SqlXmlUtil which disables external entity expansion and
enables FEATURE_SECURE_PROCESSING, but only
if there is no security manager in effect.

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, externalGeneralEntities.diff, 
> secureXmlVTI.diff, sqlxmlutil.diff, xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (DERBY-6741) User code can get the ContextManager from an EmbedConnection

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625693#comment-14625693
 ] 

Bryan Pendleton edited comment on DERBY-6741 at 7/14/15 2:13 AM:
-

It looks like the critical line is line 138 of NoDBInternalsPermissionTest.java,
which is not inside the try {} block. 

That is, perhaps:

public  voidtest_002_EmbedConnection()
throws Exception
{
Connection  conn = getConnection();
try {
((EmbedConnection) conn).getContextManager();
fail( "Should have raised an AccessControlException" );
}
catch (AccessControlException e) { println( "Caught an 
AccessControlException" ); }
}

should be:

public  voidtest_002_EmbedConnection()
throws Exception
{
try {
Connection  conn = getConnection();
((EmbedConnection) conn).getContextManager();
fail( "Should have raised an AccessControlException" );
}
catch (AccessControlException e) { println( "Caught an 
AccessControlException" ); }
}



was (Author: bryanpendleton):
Perhaps:

public  voidtest_002_EmbedConnection()
throws Exception
{
Connection  conn = getConnection();
try {
((EmbedConnection) conn).getContextManager();
fail( "Should have raised an AccessControlException" );
}
catch (AccessControlException e) { println( "Caught an 
AccessControlException" ); }
}

should be:

public  voidtest_002_EmbedConnection()
throws Exception
{
try {
Connection  conn = getConnection();
((EmbedConnection) conn).getContextManager();
fail( "Should have raised an AccessControlException" );
}
catch (AccessControlException e) { println( "Caught an 
AccessControlException" ); }
}


> User code can get the ContextManager from an EmbedConnection
> 
>
> Key: DERBY-6741
> URL: https://issues.apache.org/jira/browse/DERBY-6741
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC, Services
>Reporter: Rick Hillegas
>Assignee: Rick Hillegas
> Fix For: 10.12.0.0
>
> Attachments: derby-6741-01-aa-usederbyinternals.diff, derby.log, 
> error-stacktrace.out
>
>
> EmbedConnection.getContextManager() is a public method. Exposing internals 
> like the ContextManager is a security risk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6741) User code can get the ContextManager from an EmbedConnection

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14625693#comment-14625693
 ] 

Bryan Pendleton commented on DERBY-6741:


Perhaps:

public  voidtest_002_EmbedConnection()
throws Exception
{
Connection  conn = getConnection();
try {
((EmbedConnection) conn).getContextManager();
fail( "Should have raised an AccessControlException" );
}
catch (AccessControlException e) { println( "Caught an 
AccessControlException" ); }
}

should be:

public  voidtest_002_EmbedConnection()
throws Exception
{
try {
Connection  conn = getConnection();
((EmbedConnection) conn).getContextManager();
fail( "Should have raised an AccessControlException" );
}
catch (AccessControlException e) { println( "Caught an 
AccessControlException" ); }
}


> User code can get the ContextManager from an EmbedConnection
> 
>
> Key: DERBY-6741
> URL: https://issues.apache.org/jira/browse/DERBY-6741
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC, Services
>Reporter: Rick Hillegas
>Assignee: Rick Hillegas
> Fix For: 10.12.0.0
>
> Attachments: derby-6741-01-aa-usederbyinternals.diff, derby.log, 
> error-stacktrace.out
>
>
> EmbedConnection.getContextManager() is a public method. Exposing internals 
> like the ContextManager is a security risk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6741) User code can get the ContextManager from an EmbedConnection

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6741:
---
Attachment: derby.log
error-stacktrace.out

I saw another failure in my test run. Attached are the derby.log and
error-stacktrace.out from 
fail/Embedded_40/NoDBInternalsPermissionTest/test_002_EmbedConnection

> User code can get the ContextManager from an EmbedConnection
> 
>
> Key: DERBY-6741
> URL: https://issues.apache.org/jira/browse/DERBY-6741
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC, Services
>Reporter: Rick Hillegas
>Assignee: Rick Hillegas
> Fix For: 10.12.0.0
>
> Attachments: derby-6741-01-aa-usederbyinternals.diff, derby.log, 
> error-stacktrace.out
>
>
> EmbedConnection.getContextManager() is a public method. Exposing internals 
> like the ContextManager is a security risk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: DERBY-6801: Feedback on MessageUtils_2.diff

[Bryan Pendleton]

Hi Abhinav,

Can you attach your most recent version of MessageUtils patch to DERBY-6801?

I should have some time over the next few days to have a closer look.

bryan

On 7/13/2015 2:00 PM, Abhinav Gupta wrote:

Hi Bryan,

If I try this alternative way, I get this error

[javac] 
/home/abhinav/Documents/Derby/Derby-6801/java/shared/org/apache/derby/shared/common/error/MessageUtils.java:88:
 error: ')' expected
 [javac] return (cause instanceof carrier.getClass())
 [javac]
  ^
 [javac] 
/home/abhinav/Documents/Derby/Derby-6801/java/shared/org/apache/derby/shared/common/error/MessageUtils.java:88:
 error: illegal start of expression
 [javac] return (cause instanceof carrier.getClass())
 [javac]
   ^
 [javac] 
/home/abhinav/Documents/Derby/Derby-6801/java/shared/org/apache/derby/shared/common/error/MessageUtils.java:88:
 error: ';' expected
 [javac] return (cause instanceof carrier.getClass())
 [javac]
^
 [javac] 
/home/abhinav/Documents/Derby/Derby-6801/java/shared/org/apache/derby/shared/common/error/MessageUtils.java:89:
 error: not a statement
 [javac] ? cause : null;
 [javac] ^
 [javac] 4 errors


I understand from this, that java is unable to parse "carrier.getClass()", am I 
right ? Also I think I don't understand what would be passed as an argument for the 
Exception carrier, when getArgumentFerry(SQLException se, Exception carrier) is called ?



On Mon, Jul 13, 2015 at 8:23 AM, Bryan Pendleton mailto:bpendleton.de...@gmail.com>> wrote:

Now when I included buildSqlerrmc(),  It needs 
StandardException.getArgumentFerry() to be precise.

How should we solve this ?


Hi Abhinav,

Another possibility would be to move StandardException.getArgumentFerry
out of StandardException and into MessageUtils.

It's not a very complicated function, although it does need to
be able to run 'instanceof'.

Perhaps a MessageUtils implementation of this function could be
something like:


 /**
  * Unpack the exception, looking for a StandardException, which carries
  * the Derby messageID and arguments.
  * @see org.apache.derby.impl.jdbc.SQLExceptionFactory
  * @see org.apache.derby.impl.jdbc.Util
  */
 public static Exception getArgumentFerry(SQLException se, Exception 
carrier)
 {
 Throwable cause = se.getCause();
 return (cause instanceof carrier.getClass()) ? cause : null;
 }

Something like this might be an alternative approach.

bryan

Re: DERBY-6801: Feedback on MessageUtils_2.diff

[Bryan Pendleton]

Now when I included buildSqlerrmc(),  It needs 
StandardException.getArgumentFerry() to be precise.

How should we solve this ?


Hi Abhinav,

Another possibility would be to move StandardException.getArgumentFerry
out of StandardException and into MessageUtils.

It's not a very complicated function, although it does need to
be able to run 'instanceof'.

Perhaps a MessageUtils implementation of this function could be
something like:


/**
 * Unpack the exception, looking for a StandardException, which carries
 * the Derby messageID and arguments.
 * @see org.apache.derby.impl.jdbc.SQLExceptionFactory
 * @see org.apache.derby.impl.jdbc.Util
 */
public static Exception getArgumentFerry(SQLException se, Exception carrier)
{
Throwable cause = se.getCause();
return (cause instanceof carrier.getClass()) ? cause : null;
}

Something like this might be an alternative approach.

bryan

[jira] [Commented] (DERBY-6741) User code can get the ContextManager from an EmbedConnection

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14624127#comment-14624127
 ] 

Bryan Pendleton commented on DERBY-6741:


Thanks Rick. If I can provoke it again, I'll capture more details.

> User code can get the ContextManager from an EmbedConnection
> 
>
> Key: DERBY-6741
> URL: https://issues.apache.org/jira/browse/DERBY-6741
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC, Services
>Reporter: Rick Hillegas
>Assignee: Rick Hillegas
> Fix For: 10.12.0.0
>
> Attachments: derby-6741-01-aa-usederbyinternals.diff
>
>
> EmbedConnection.getContextManager() is a public method. Exposing internals 
> like the ContextManager is a security risk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Looks like my ShutdownException change is causing sealing violations

[Bryan Pendleton]

I'll have to study this:

[junit] Exception in thread "Thread-117" java.lang.SecurityException: 
sealing violation: package org.apache.derby.shared.common.error is sealed
[junit] at 
java.net.URLClassLoader.getAndVerifyPackage(URLClassLoader.java:394)

bryan

Re: DERBY-6801: Feedback on MessageUtils_2.diff

[Bryan Pendleton]

MessageUtils needs StandardException.getArgumentFerry() to be precise.

How should we solve this ?


Hi Abhinav,

I wonder if we can handle this by introducing a new interface class.

Something like:

interface ExceptionCarrier
{
public static Exception getArgumentFerry( SQLException se );
}

And then:

public class StandardException extends Exception implements ExceptionCarrier
{ ... }

The ExceptionCarrier interface class would be in 
o.a.d.shared.common.error.ExceptionCarrier.

That way MessageUtils would use the ExceptionCarrier interface, but we would
not need to move the entire Standard Exception class, which is large and
complex and closely tied to the engine internals, to the shared area.

What do you think about this idea?

bryan

[jira] [Commented] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623476#comment-14623476
 ] 

Bryan Pendleton commented on DERBY-6807:


Is XmlVTI documented anywhere in our public documentation? I'm not sure if any 
documentation changes are needed here. Perhaps there is Javadoc that I should 
update...?

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, externalGeneralEntities.diff, 
> secureXmlVTI.diff, xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623474#comment-14623474
 ] 

Bryan Pendleton commented on DERBY-6801:


Hi Abhinav,

Now that DERBY-6824 is in the trunk, can you please update your sandbox
and post a revised patch containing only the new MessageUtils work with
your latest version of the code? Thanks!

> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: MessageUtils_2.diff, MessageUtils_3.diff, build.out, 
> messageUtils.diff, messageUtils_1.diff, subPart2.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DERBY-6824) Move ShutdownException into shared code area

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton resolved DERBY-6824.

   Resolution: Fixed
 Assignee: Bryan Pendleton
Fix Version/s: 10.12.0.0

> Move ShutdownException into shared code area
> 
>
> Key: DERBY-6824
> URL: https://issues.apache.org/jira/browse/DERBY-6824
> Project: Derby
>  Issue Type: Sub-task
>  Components: Miscellaneous
>    Reporter: Bryan Pendleton
>    Assignee: Bryan Pendleton
>Priority: Minor
> Fix For: 10.12.0.0
>
> Attachments: refactor.diff
>
>
> We would like to refactor MessageService.getLocalizedMessage()
> so that it resides as part of the new MessageUtils class in the
> shared library.
> However, MessageService.getLocalizedMessage() currently has
> a hard dependency on o.a.d.iapi.error.ShutdownException.
> This subtask proposes to move ShutdownException to
> o.a.d.shared.common.error.ShutdownException, so that it
> can be used by new error-handling code in the shared library
> in addition to being used in the engine code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6824) Move ShutdownException into shared code area

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623473#comment-14623473
 ] 

Bryan Pendleton commented on DERBY-6824:


My further testing has revealed no problems with this change, and since it's
blocking our work on DERBY-6801 I've decided to move ahead with it.

Committed revision 1690384.

> Move ShutdownException into shared code area
> 
>
> Key: DERBY-6824
> URL: https://issues.apache.org/jira/browse/DERBY-6824
> Project: Derby
>  Issue Type: Sub-task
>  Components: Miscellaneous
>Reporter: Bryan Pendleton
>Priority: Minor
> Attachments: refactor.diff
>
>
> We would like to refactor MessageService.getLocalizedMessage()
> so that it resides as part of the new MessageUtils class in the
> shared library.
> However, MessageService.getLocalizedMessage() currently has
> a hard dependency on o.a.d.iapi.error.ShutdownException.
> This subtask proposes to move ShutdownException to
> o.a.d.shared.common.error.ShutdownException, so that it
> can be used by new error-handling code in the shared library
> in addition to being used in the engine code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6741) User code can get the ContextManager from an EmbedConnection

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623467#comment-14623467
 ] 

Bryan Pendleton commented on DERBY-6741:


With OpenJDK 1.8.0_45 running on Fedora22, I see highly intermittent
failures in 

NoDBInternalsPermissionTest.test_002_EmbedConnection

My error is:

java.security.AccessControlException: access denied 
org.apache.derby.security.SystemPermission( "engine", "usederbyinternals" )
at 
java.security.AccessControlContext.checkPermission(AccessControlContext.java:457)
 
at 
java.security.AccessController.checkPermission(AccessController.java:884) 
at 
org.apache.derby.iapi.security.SecurityUtil.checkDerbyInternalsPrivilege(SecurityUtil.java:221)
at 
org.apache.derby.iapi.services.monitor.Monitor.getMonitorLite(Monitor.java:339)
at 
org.apache.derby.iapi.services.property.PropertyUtil$2.run(PropertyUtil.java:719)
at 
org.apache.derby.iapi.services.property.PropertyUtil$2.run(PropertyUtil.java:716)
at java.security.AccessController.doPrivileged(Native Method)

The error is quite hard to reproduce, and seems only to happen in large suite 
runs.

Any thoughts on what might be happening?

> User code can get the ContextManager from an EmbedConnection
> 
>
> Key: DERBY-6741
> URL: https://issues.apache.org/jira/browse/DERBY-6741
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC, Services
>Reporter: Rick Hillegas
>Assignee: Rick Hillegas
> Fix For: 10.12.0.0
>
> Attachments: derby-6741-01-aa-usederbyinternals.diff
>
>
> EmbedConnection.getContextManager() is a public method. Exposing internals 
> like the ContextManager is a security risk.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6807:
---
Issue & fix info: Patch Available

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, externalGeneralEntities.diff, 
> secureXmlVTI.diff, xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Can the engine detect if it is running under an explicit SecurityManager policy?

[Bryan Pendleton]

I'm working on a final fix for DERBY-6807 for SqlXmlUtils, but I'd
prefer to make that change conditional, based on whether there is
a SecurityManager policy in place for the engine.

The reason for that is that if there is a SecurityManager in place,
SqlXmlUtils is already fully secure, because it delegates the
access decisions properly to the SecurityManager policy.

But if there is NOT a SecurityManager policy in effect, then it would
be more secure for SqlXmlUtils to disable certain XML features.

Is there example code that somebody can point me at which SqlXmlUtils
could use to detect whether or not a SecurityManager policy is in effect?

thanks,

bryan

Review requested for DERBY-6807 secureXmlVTI.diff

[Bryan Pendleton]

Hi Rick (and any others who are interested in the XmlVTI),

Can you have a quick peek at DERBY-6807 and the secureXmlVTI.diff patch?

thanks,

bryan

[jira] [Updated] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6807:
---
Attachment: secureXmlVTI.diff

Attached secureXmlVTI.diff is my proposed change to
make the XmlVTI secure against external entity attacks.

It does so by disabling external entity expansion entirely,
which is a loss in functionality, but a gain in security.

Since the Derby VTI interface is public and fully extensible
(see https://db.apache.org/derby/docs/10.11/devguide/cdevspecialtfbasic.html ),
anyone who wishes to have an XmlVTI which supports external
entity expansion can simply define their own.

But the built-in Derby XmlVTI will not, for security reasons.

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, externalGeneralEntities.diff, 
> secureXmlVTI.diff, xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6825) Add basic JSON support to Derby.

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14619699#comment-14619699
 ] 

Bryan Pendleton commented on DERBY-6825:


They both sound like interesting bits of functionality. I think I would be
more interested in the feature of being able to express query results in JSON,
as that seems like a good way for Derby to provide some level of
interoperability with other systems that want to consume data in JSON format.


> Add basic JSON support to Derby.
> 
>
> Key: DERBY-6825
> URL: https://issues.apache.org/jira/browse/DERBY-6825
> Project: Derby
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 10.12.0.0
>Reporter: Rick Hillegas
> Attachments: SimpleJsonVTI.java, derby-6825-01-aa-optionalTool.diff
>
>
> Other database are adding support for the popular JSON data interchange 
> format defined by https://www.ietf.org/rfc/rfc4627.txt. We could add a VTI to 
> Derby which would unpack a JSON document into a set of rows. We could also 
> add a reverse transformation, which turns a query result into a JSON document.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6826) SOA PROD - The state of datasource DifJavaDbDataSource, application domain-level is Suspended.

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14619694#comment-14619694
 ] 

Bryan Pendleton commented on DERBY-6826:


In your derby.log is this exception:

Caused by: ERROR XJ040: Failed to start database 'cachedb_paa' with class 
loader sun.misc.Launcher$AppClassLoader@7d38847d, see the next exception for 
details.
at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
at 
org.apache.derby.impl.jdbc.SQLExceptionFactory.wrapArgsForTransportAcrossDRDA(Unknown
 Source)
... 16 more
Caused by: ERROR XSTB0: An exception was thrown during transaction abort.
at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
at org.apache.derby.impl.store.raw.xact.Xact.abort(Unknown Source)
at 
org.apache.derby.impl.store.raw.xact.XactFactory.rollbackAllTransactions(Unknown
 Source)
at org.apache.derby.impl.store.raw.log.LogToFile.recover(Unknown Source)
at org.apache.derby.impl.store.raw.RawStore.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown 
Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown 
Source)
at 
org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at 
org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source)
at 
org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.store.access.RAMAccessManager.boot(Unknown 
Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown 
Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown 
Source)
at 
org.apache.derby.impl.services.monitor.BaseMonitor.startModule(Unknown Source)
at 
org.apache.derby.impl.services.monitor.FileMonitor.startModule(Unknown Source)
at 
org.apache.derby.iapi.services.monitor.Monitor.bootServiceModule(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.bootStore(Unknown Source)
at org.apache.derby.impl.db.BasicDatabase.boot(Unknown Source)
at org.apache.derby.impl.services.monitor.BaseMonitor.boot(Unknown 
Source)
at org.apache.derby.impl.services.monitor.TopService.bootModule(Unknown 
Source)
at 
org.apache.derby.impl.services.monitor.BaseMonitor.bootService(Unknown Source)
at 
org.apache.derby.impl.services.monitor.BaseMonitor.startProviderService(Unknown 
Source)
at 
org.apache.derby.impl.services.monitor.BaseMonitor.findProviderAndStartService(Unknown
 Source)
at 
org.apache.derby.impl.services.monitor.BaseMonitor.startPersistentService(Unknown
 Source)
at 
org.apache.derby.iapi.services.monitor.Monitor.startPersistentService(Unknown 
Source)
... 13 more
Caused by: ERROR XSLA8: Cannot rollback transaction 543496278, trying to 
compensate null operation with null
at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
at org.apache.derby.impl.store.raw.log.FileLogger.undo(Unknown Source)
... 37 more
Caused by: ERROR XSCB5: During logical undo of a btree insert or delete the row 
could not be found in the tree.
at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown 
Source)
at 
org.apache.derby.impl.store.access.btree.index.B2IUndo.findUndo(Unknown Source)
at 
org.apache.derby.impl.store.raw.data.LogicalPageOperation.findLogicalPage(Unknown
 Source)
at 
org.apache.derby.impl.store.raw.data.LogicalPageOperation.generateUndo(Unknown 
Source)
at 
org.apache.derby.impl.store.raw.data.UpdateFieldOperation.generateUndo(Unknown 
Source)

This indicates that the database cannot be recovered.

Is this the oldest derby.log that you have? Do you have any records that 
indicate
what happened prior to Thu Jun 18 00:20:26 EDT 2015?

It would appear that the database is sufficiently damaged that it cannot be 
recovered, and
will have to be restored from backups.

But it would be very valuable to understand what caused this damage (e.g., hard 
disk failure,
cpu failure, power failure, disk became full, accidental system administration 
mistake, etc.)

Any additional information you can provide about what happened leading up
to the damaged database could provide useful clues.


> SOA PROD - The state of datasource DifJavaDbDataSource, application 
> domain-level is Suspended.
> --
>
> Key: DERBY-6826
> URL: https://issues.apache.org/jira/browse/DERBY-6826
> Project: Derby
>  Issue

[jira] [Commented] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614373#comment-14614373
 ] 

Bryan Pendleton commented on DERBY-6801:


I filed DERBY-6824 to track the idea that we might refactor ShutdownException
to move it from the engine code to the shared code for use by MessageUtils,
and attached a proposed patch to accomplish that result. Abhinav, can you
review my work on that?

> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: MessageUtils_2.diff, build.out, messageUtils.diff, 
> messageUtils_1.diff, subPart2.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6824) Move ShutdownException into shared code area

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6824:
---
Attachment: refactor.diff

Attached 'refactor.diff' is the proposed modification to the package of 
ShutdownException.

> Move ShutdownException into shared code area
> 
>
> Key: DERBY-6824
> URL: https://issues.apache.org/jira/browse/DERBY-6824
> Project: Derby
>  Issue Type: Sub-task
>  Components: Miscellaneous
>Reporter: Bryan Pendleton
>Priority: Minor
> Attachments: refactor.diff
>
>
> We would like to refactor MessageService.getLocalizedMessage()
> so that it resides as part of the new MessageUtils class in the
> shared library.
> However, MessageService.getLocalizedMessage() currently has
> a hard dependency on o.a.d.iapi.error.ShutdownException.
> This subtask proposes to move ShutdownException to
> o.a.d.shared.common.error.ShutdownException, so that it
> can be used by new error-handling code in the shared library
> in addition to being used in the engine code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DERBY-6824) Move ShutdownException into shared code area

[Bryan Pendleton (JIRA)]

Bryan Pendleton created DERBY-6824:
--

 Summary: Move ShutdownException into shared code area
 Key: DERBY-6824
 URL: https://issues.apache.org/jira/browse/DERBY-6824
 Project: Derby
  Issue Type: Sub-task
  Components: Miscellaneous
Reporter: Bryan Pendleton
Priority: Minor


We would like to refactor MessageService.getLocalizedMessage()
so that it resides as part of the new MessageUtils class in the
shared library.

However, MessageService.getLocalizedMessage() currently has
a hard dependency on o.a.d.iapi.error.ShutdownException.

This subtask proposes to move ShutdownException to
o.a.d.shared.common.error.ShutdownException, so that it
can be used by new error-handling code in the shared library
in addition to being used in the engine code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Ancient history: question about some mysterious code in MessageService.java

[Bryan Pendleton]

Thanks Rick, for the pointers and ideas.


In the code block which you are studying, the message formatting logic responds 
to the ShutdownException by falling through to its last chance handler. Do you 
think it should do something else?


No, I don't propose to change its behavior, just to understand it.

In the context of DERBY-6773, Abhinav and I have been studying whether
we can refactor MessageService.getLocalizedMessage() so that it
resides in the shared area of the code and can be bound into both
the client and server jars.

The intent of the refactoring is to gather all the code which knows
about packing exceptions into network messages (SQLERRMC blocks) and
then de-packing them on the other side into a single class, rather
than the current situation where the code is rather spread about.

And the code just above the code I quoted yesterday is directly
dependent on other code in the networking classes that we're also
proposing to refactor; it encodes message arguments with special delimiters:

char [] sqlerrmc_chars = sqlerrmc.toCharArray();
int numArgs = 0, lastSepIdx = -1; // last separator 
index
for (int i = 0; i < sqlerrmc_chars.length; i++)
{
if (sqlerrmc_chars[i] == 20)// separator
{
numArgs++;
lastSepIdx = i;
}
}

That's the code I'd like to be able to call directly from the client libraries.

The dependence on ShutdownException is the only thing (that I can tell)
which is requiring these two methods to be in the engine jar; otherwise
they could clearly be in the shared code.

But to accomplish that, we'll also need to refactor ShutdownException
so that it is part of the shared code, too. Which is a tiny change, but
is still a widening of our proposal.

I was searching for ways to avoid it, e.g., by having getLocalizedMessage()
throw the ShutdownException and have its caller catch and handle it, but
that's not a 1-line change to the code, either.

I guess I was sort of hoping somebody might say:

"Oh! That's dead code! It was in there as early scaffolding, and
can simply be removed."

But I don't believe that to be the case, and will stop hoping for that.

 :)

bryan

[jira] [Created] (DERBY-6823) Refactor uses of DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH

[Bryan Pendleton (JIRA)]

Bryan Pendleton created DERBY-6823:
--

 Summary: Refactor uses of DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH
 Key: DERBY-6823
 URL: https://issues.apache.org/jira/browse/DERBY-6823
 Project: Derby
  Issue Type: Sub-task
  Components: Network Server, SQL
Reporter: Bryan Pendleton
Priority: Minor


There is some duplication of code, probably historic, which
leads to multiple definitions of the named constant
DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH:

$ find . -name '*.java' -exec grep DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH {} \; 
-print
return Limits.DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH;
./java/drda/org/apache/derby/impl/drda/AppRequester.java
public static final int DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH = 2400;
./java/engine/org/apache/derby/iapi/reference/Limits.java
Types.VARCHAR, 
Limits.DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH),
./java/engine/org/apache/derby/impl/sql/catalog/DataDictionaryImpl.java

I suspect this code pre-dates the work to create the "shared" library
where code shared among all Derby jar packages can reside.

It seems like we should be able to refactor this code so that
there is only one instance of DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH,
and it seems like the new MessageUtils class proposed by
DERBY-6801 would be a good place for that constant to live.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Ancient history: question about some mysterious code in MessageService.java

[Bryan Pendleton]

I've been digging into o.a.d.iapi.services.i18n.MessageService.java.

Going all the way back to the original code import (revision 37083),
MessageService.getLocalizedMessage() contains this bit of code:

try {
msg[0] = formatMessage(getBundleForLocale(locale, 
messageId), messageId, arguments, true);
rc[0] = 0;
return;
} catch (MissingResourceException mre) {
// message does not exist in the requested locale
// most likely it does exist in our fake base class 
_en, so try that.
} catch (ShutdownException se) {
}
msg[0] = formatMessage(getBundleForLocale(EN, messageId), 
messageId, arguments, false);
rc[0] = 0;

I'm wondering specifically about the bit:

} catch (ShutdownException se) {
}

Why is this code here? Under what circumstances would we want to
discard the ShutdownException and continue processing like this?

Anybody have any memories of why this code was written this way?

I did spend some time reading DERBY-4920, DERBY-4911, and particularly
the epic DERBY-4741, all of which suggest that the presence of ShutdownException
handling in the code, in general, is due to the fact that when the engine
is being shutdown, there may be multiple threads which are active, and
the engine's shutdown logic uses JDK "interrupt" features to cause those
threads to receive ShutdownException exceptions being thrown.

But those old JIRAs, marvelous as they are, didn't really clarify why
MessageService has any business discarding the exception in this manner.

thanks,

bryan

DERBY-6801: Feedback on MessageUtils_2.diff

[Bryan Pendleton]

Hi Abhinav,

I spent some time thinking about MessageUtils_2.diff.

I'd like to see us refine this still further, because
I think we want to avoid moving many of these other
classes into the shared area wholesale.

In most of these cases, I think we can refine things
and make a smaller and more precise patch, even though
that will require some more modifications to the MessageUtils class.

For example, let's consider AppRequester.

The only reason that we needed AppRequester in
MessageUtils is for this line:

int maxlen = (sqlerrmc == null) ? -1 : Math.min(sqlerrmc.length(),
appRequester.supportedMessageParamLength());

The supportedMessageParamLength() method is pretty trivial:

public static final int DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH = 2400;

protected int supportedMessageParamLength() {
return DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH;
}

So let's not bring all of AppRequester over to the shared
library just for this one bit.

Instead, FOR NOW, let's just duplicate these 4 lines of code
into MessageUtils.java.

I realize that seems bad, but that particular named
constant is already a bit messed up in the code:

$ find . -name '*.java' -exec grep DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH {} \; 
-print
return Limits.DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH;
./java/drda/org/apache/derby/impl/drda/AppRequester.java
public static final int DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH = 2400;
./java/engine/org/apache/derby/iapi/reference/Limits.java
Types.VARCHAR, 
Limits.DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH),
./java/engine/org/apache/derby/impl/sql/catalog/DataDictionaryImpl.java
public static final int DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH = 2400;
return DB2_JCC_MAX_EXCEPTION_PARAM_LENGTH;
./java/shared/org/apache/derby/shared/common/error/AppRequester.java


So I'll file a new subtask of DERBY-6773 to clean up this
little duplication of code, which we can do as a follow-up
after this step is complete.

Regarding the other classes, it seems that nearly all of them
are present in order to provide support for the method

MessageUtils.getSQLException(...)

Can we simply remove getSQLException() from MessageUtils?

It doesn't seem to me that we should need it, and if we remove
getSQLException() from MessageUtils, and also remove the
helper method wrapArgsForTransportAcrossDRDA,
then I believe we can remove all the classes:

 - MessageService
 - ExceptionFactory
 - ArrayUtil
 - BundleFinder
 - StandardException

Next, for SQLState.java, I think that maybe is just a typo. The
correct SQLState is already in the shared library, in

org.apache.derby.shared.common.reference.SQLState

So I think you can remove the SQLState.java from your patch,
and simply add to MessageUtils.java:

import org.apache.derby.shared.common.reference.SQLState;

That leaves us with ShutdownException.

For ShutdownException, the only reason we had to bring it
into this patch is because the getLocalizedMessage()
method wants to catch and ignore it.

That is interesting; let me think about that one for a while.
It's possible that moving ShutdownException into the shared
library is the best way to deal with this, but there may be
other ways.

So, to summarize, this is my proposal:

1) Duplicate 4 lines of code from AppRequester into MessageUtils,
   and file a follow-on sub-task of DERBY-2773 to remove this
   duplication (by refactoring AppRequester, Limits, and
   DataDictionaryImpl to use the new MessageUtils method),
   after MessageUtils has been committed.

2) Remove the getSQLException() methods and change
   MessageUtils so it doesn't subclass ExceptionFactory.

3) Remove the classes: AppRequester, ExceptionFactory,
   ArrayUtil, BundleFinder, MessageService, and
   StandardException

4) We still have to do more study on ShutdownException

Does that make sense?

thanks,

bryan

[jira] [Commented] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613967#comment-14613967
 ] 

Bryan Pendleton commented on DERBY-6801:


The MessageUtils_2.diff patch compiles on my system now, too!

I have some thoughts on the patch, which are a bit detailed, so I'm going to
send them to you on derby-dev. Please have a look, and let me know
what you think about these ideas.


> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: MessageUtils_2.diff, build.out, messageUtils.diff, 
> messageUtils_1.diff, subPart2.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14613526#comment-14613526
 ] 

Bryan Pendleton commented on DERBY-6801:


Yes, that part of our process is a bit awkward: 
http://wiki.apache.org/db-derby/AddedOrDeletedFilesInDiff

I'm looking forward to working with your new patch, thanks!

> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: MessageUtils_2.diff, build.out, messageUtils.diff, 
> messageUtils_1.diff, subPart2.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6801:
---
Attachment: build.out

Hi Abhinav, thanks for the latest patch.

The code looks much cleaner now, this is great progress!

I am still seeing some compile errors on my machine; I attached
the file 'build.out' which shows what happens when I run 'ant all'
on a clean sandbox with the patch applied.

I hope to have some time this weekend to dig into your patch
in more detail, and then I can give you more substantial
feedback, but hopefully this is enough to give you something
to look at.

> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: build.out, messageUtils.diff, messageUtils_1.diff, 
> subPart2.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6801) Implement MessageUtils class so client and server can share message argument encoding/decoding

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609417#comment-14609417
 ] 

Bryan Pendleton commented on DERBY-6801:


I see what you mean; this is a little bit tricky.

I think we have to deal with these one by one.

Some of them I think we can simply remove. For example, I don't think you need:

import org.apache.derby.iapi.reference.Limits;
import org.apache.derby.iapi.reference.DRDAConstants;
import org.apache.derby.impl.jdbc.Util;

since those don't seem to be referenced at all elsewhere in Messageutils.

Some of them indicate places where we still need to do more refactoring. For
example, I think that SystemProcedures.SQLERRMC_MESSAGE_DELIMITER
should be moved to MessageUtils, and then we don't have to import
SystemProcedures anymore.

This part of this project may be pretty tricky. For this whole design to work,
it is crucial that MessageUtils be self-contained and minimal, because it
has to be a class that can be packaged either with the client jar, or with the
server jar, and so it can't have a lot of external dependencies.

Maybe you can keep studying the various dependencies and class-not-found errors,
and we can see which ones are straightforward to deal with and which are hard?

> Implement MessageUtils class so client and server can share message argument 
> encoding/decoding
> --
>
> Key: DERBY-6801
> URL: https://issues.apache.org/jira/browse/DERBY-6801
> Project: Derby
>  Issue Type: Sub-task
>  Components: JDBC, Network Client, Network Server
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
>Priority: Minor
> Attachments: messageUtils.diff, subPart2.diff
>
>
> Currently, the Network Server contains code which encodes message arguments 
> to be passed over DRDA to the client, and other code which
> decodes the encoded message arguments when the client passes
> them back in a call to SQLCAMESSAGE (see DERBY-1178 for additional
> discussion of this behavior).
> We would like to extract that logic and place it into a new MessageUtils
> class in the common shared library so that both the client and server
> can manipulate the message arguments without duplicating the code.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6810:
---
Attachment: vtiTests2.diff

vtiTests2.diff seems to pass on the platforms I have access to,
and corrects the svn properties for the new trace files (I think).

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, error-stacktrace.out, 
> readPasswordFile.diff, vtiTests.diff, vtiTests2.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DERBY-6821) Include XMLOptimizerTraceTest in XMLSuite

[Bryan Pendleton (JIRA)]

Bryan Pendleton created DERBY-6821:
--

 Summary: Include XMLOptimizerTraceTest in XMLSuite
 Key: DERBY-6821
 URL: https://issues.apache.org/jira/browse/DERBY-6821
 Project: Derby
  Issue Type: Sub-task
  Components: Test
Reporter: Bryan Pendleton
Priority: Minor


There is a sub-suite called XMLSuite that runs the XML tests,
but it does not currently run the XMLOptimizerTraceTest.

Since that test suite is a significant quite for exercising
Derby's XML support, it seems worth including that suite in
the XMLSuite.

This might require a bit of suite re-arrangement, to make sure
that we don't run XMLOptimizerTraceTest multiple times during larger suites.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (DERBY-6820) Improve error handling in XmlVTI

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton resolved DERBY-6820.

   Resolution: Fixed
Fix Version/s: 10.12.0.0

Committed revision 1688297.

> Improve error handling in XmlVTI
> 
>
> Key: DERBY-6820
> URL: https://issues.apache.org/jira/browse/DERBY-6820
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 10.11.1.1
>    Reporter: Bryan Pendleton
>Assignee: Bryan Pendleton
> Fix For: 10.12.0.0
>
> Attachments: errorHandling.diff
>
>
> While writing XXE vulnerability test suites for XmlVTI, I found that
> some of the error handling logic in XmlVTI is a bit rough and could
> be made a bit smoother.
> At least the following improvements would be useful:
> 1) Remove the unconditional printStackTrace call in XmlVTI.next()
> when it catches a Throwable.
> 2) Chain the original exception to the SQLException using the
>SQLException constructor which takes a "cause" argument.
> 3) Include an XMLErrorHandler class so that fatal exceptions that
>arise within the XML parser don't directly print their own messages
>to System.err, to avoid printing
> [junit] [Fatal Error] :1:1: The parser has encountered more than "64,000" 
> entity expansions in this document; this is the limit imposed by the 
> application.
> 4) Close the _xmlResource input stream, if it exists, from the
>XMLErrorHandler methods, so that the parser's files are closed
>sooner and we are (I think) less likely to encounter
> [junit]  attempt 1 left 2 files/dirs behind: 
> 0=extin\xmlOptimizerXXE2.trace 1=extin
> [junit]  attempt 2 left 2 files/dirs behind: 
> 0=extin\xmlOptimizerXXE2.trace 1=extin
> [junit]  attempt 3 left 2 files/dirs behind: 
> 0=extin\xmlOptimizerXXE2.trace 1=extin
> [junit]  attempt 4 left 2 files/dirs behind: 
> 0=extin\xmlOptimizerXXE2.trace 1=extin



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604930#comment-14604930
 ] 

Bryan Pendleton commented on DERBY-6807:


While continuing to research this issue I came across the very-closely-related 
DERBY-2131 and the Subversion revision 481117.

https://svn.apache.org/viewvc?view=revision&revision=481117

It's 10-year-old history, but still quite relevant to this discussion.

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, externalGeneralEntities.diff, 
> xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: XmlVTI.next() invocation of printStackTrace()

[Bryan Pendleton]

This method in XmlVTI is causing me a bit of a problem:


Thanks Kristian, Mike, and Rick!

This seems worth improving, so I filed DERBY-6820 and
will follow-up.

bryan

[jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604893#comment-14604893
 ] 

Bryan Pendleton commented on DERBY-6783:


This patch seems to be working well for me. Thanks for all the hard
work on this, Abhinav! This was a challenging first project to take
on within Derby, but it represents substantial improvement to the
TRIGGER support, I believe. 

Committed revision 1688062.

> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: 6783.diff, 6783_allTestsPass.diff, 6783_moreTests.diff, 
> 6783_moreTests_bryan.diff, 6783_newTest.diff, 6783_newTest_bryan.diff, 
> 6783_newTests.diff, cleanedUpDiff.patch, diagnostics.diff, 
> error-stacktrace.out, sortFunction.diff, testTriggerWhenClause.diff, 
> workingPatch.patch
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6820) Improve error handling in XmlVTI

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6820:
---
Attachment: errorHandling.diff

errorHandling.diff contains the proposed error handling improvements.

> Improve error handling in XmlVTI
> 
>
> Key: DERBY-6820
> URL: https://issues.apache.org/jira/browse/DERBY-6820
> Project: Derby
>  Issue Type: Sub-task
>  Components: SQL
>Affects Versions: 10.11.1.1
>    Reporter: Bryan Pendleton
>Assignee: Bryan Pendleton
> Attachments: errorHandling.diff
>
>
> While writing XXE vulnerability test suites for XmlVTI, I found that
> some of the error handling logic in XmlVTI is a bit rough and could
> be made a bit smoother.
> At least the following improvements would be useful:
> 1) Remove the unconditional printStackTrace call in XmlVTI.next()
> when it catches a Throwable.
> 2) Chain the original exception to the SQLException using the
>SQLException constructor which takes a "cause" argument.
> 3) Include an XMLErrorHandler class so that fatal exceptions that
>arise within the XML parser don't directly print their own messages
>to System.err, to avoid printing
> [junit] [Fatal Error] :1:1: The parser has encountered more than "64,000" 
> entity expansions in this document; this is the limit imposed by the 
> application.
> 4) Close the _xmlResource input stream, if it exists, from the
>XMLErrorHandler methods, so that the parser's files are closed
>sooner and we are (I think) less likely to encounter
> [junit]  attempt 1 left 2 files/dirs behind: 
> 0=extin\xmlOptimizerXXE2.trace 1=extin
> [junit]  attempt 2 left 2 files/dirs behind: 
> 0=extin\xmlOptimizerXXE2.trace 1=extin
> [junit]  attempt 3 left 2 files/dirs behind: 
> 0=extin\xmlOptimizerXXE2.trace 1=extin
> [junit]  attempt 4 left 2 files/dirs behind: 
> 0=extin\xmlOptimizerXXE2.trace 1=extin



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (DERBY-6820) Improve error handling in XmlVTI

[Bryan Pendleton (JIRA)]

Bryan Pendleton created DERBY-6820:
--

 Summary: Improve error handling in XmlVTI
 Key: DERBY-6820
 URL: https://issues.apache.org/jira/browse/DERBY-6820
 Project: Derby
  Issue Type: Sub-task
  Components: SQL
Affects Versions: 10.11.1.1
Reporter: Bryan Pendleton
Assignee: Bryan Pendleton


While writing XXE vulnerability test suites for XmlVTI, I found that
some of the error handling logic in XmlVTI is a bit rough and could
be made a bit smoother.

At least the following improvements would be useful:

1) Remove the unconditional printStackTrace call in XmlVTI.next()
when it catches a Throwable.

2) Chain the original exception to the SQLException using the
   SQLException constructor which takes a "cause" argument.

3) Include an XMLErrorHandler class so that fatal exceptions that
   arise within the XML parser don't directly print their own messages
   to System.err, to avoid printing

[junit] [Fatal Error] :1:1: The parser has encountered more than "64,000" 
entity expansions in this document; this is the limit imposed by the 
application.

4) Close the _xmlResource input stream, if it exists, from the
   XMLErrorHandler methods, so that the parser's files are closed
   sooner and we are (I think) less likely to encounter

[junit]  attempt 1 left 2 files/dirs behind: 
0=extin\xmlOptimizerXXE2.trace 1=extin
[junit]  attempt 2 left 2 files/dirs behind: 
0=extin\xmlOptimizerXXE2.trace 1=extin
[junit]  attempt 3 left 2 files/dirs behind: 
0=extin\xmlOptimizerXXE2.trace 1=extin
[junit]  attempt 4 left 2 files/dirs behind: 
0=extin\xmlOptimizerXXE2.trace 1=extin



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

XmlVTI.next() invocation of printStackTrace()

[Bryan Pendleton]

This method in XmlVTI is causing me a bit of a problem:

public  boolean next() throws SQLException
{
try {
if ( _rowIdx < 0 ) { readRows(); }

if ( ++_rowIdx < _rowCount )
{
parseRow( _rowIdx );
return true;
}
else { return false; }
} catch (Throwable t)
{
t.printStackTrace( System.out );
throw new SQLException( t.getMessage() );
}
}

Is there a compelling reason why that call to printStackTrace needs to be there?

I am exercising this code in a regression test, and provoking
an exception in it, and the unconditional printStackTrace
makes the test results rather ugly...

What would be the downside if I removed that printStackTrace() call?

thanks,

bryan

[jira] [Commented] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14604330#comment-14604330
 ] 

Bryan Pendleton commented on DERBY-6807:


I propose to do the following to resolve this issue:

1) Modify XmlVTI and SqlXmlUtils to perform

dBF.setFeature( "http://xml.org/sax/features/external-general-entities";, 
false );

2) Modify the XXE regression tests to demonstrate the new, not vulnerable, 
behavior.

3) Write a release note alerting the community to the fact that Derby's XML 
parsing
   logic will no longer allow any expansion of external general entities, and 
noting
   that any applications which rely on this behavior will need to be modified.

Note that I do NOT propose to make the first change configurable, it will be 
unconditional.

We may need to revisit this decision if it proves undesirable in the broader 
community,
but I'd rather start by solidly closing the security hole and then seeing what 
response we get.


> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, externalGeneralEntities.diff, 
> xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6810:
---
Attachment: vtiTests.diff

Attached "vtiTests.diff" contains XmlVTI variants on the
file access and billion laughs tests. I believe the tests
successfully demonstrate the vulnerability via the XmlVTI.

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, error-stacktrace.out, 
> readPasswordFile.diff, vtiTests.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

> The changes you suggested fixes the Derby5578 test. I've changed the
> return type of examineTriggerNode to int[] and I've run quite a
> few test suites without encountering any bugs.

Wonderful!

> The problem that I was facing while running the tests suites was
> because of the "junit-html" in the command for executing tests suites,
> it was causing the tests to fail, so I couldn't find a solution
> to fix this and I ran the tests without it.

Interesting. I haven't seen this problem, but I confess I don't
use the 'junit-html' target all that often.

I'm glad you were able to determine the cause of the probelm
and avoid it for now.

> Is there anything else that needs to be done before this patch is accepted ?

I will spend some time with your final patch this weekend. Perhaps
others are reviewing it too, I don't know. It sounds like we are
ready to move forward with it, though, which is great news.

Perhaps while I am studying the patch, you can return to
DERBY-6773, which we haven't had time to work on recently,
and see if you can pick up from where we left off?

Thanks again for all the hard work on DERBY-6783. I think this
is very nice work and it represents a substantial improvement
to the trigger functionality in DERBY.

bryan

[jira] [Issue Comment Deleted] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6810:
---
Comment: was deleted

(was: An example of how to exercise XmlVTI can be found in 
o.a.dT.fT.tests.lang.XMLOptimizerTraceTest.java)

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, error-stacktrace.out, 
> readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600650#comment-14600650
 ] 

Bryan Pendleton commented on DERBY-6810:


An example of how to exercise XmlVTI can be found in 
o.a.dT.fT.tests.lang.XMLOptimizerTraceTest.java

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, error-stacktrace.out, 
> readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600651#comment-14600651
 ] 

Bryan Pendleton commented on DERBY-6810:


An example of how to exercise XmlVTI can be found in 
o.a.dT.fT.tests.lang.XMLOptimizerTraceTest.java

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, error-stacktrace.out, 
> readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-2245) DatabaseMetaData.getSQLKeywords() contains words that are not keywords in Derby.

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600636#comment-14600636
 ] 

Bryan Pendleton commented on DERBY-2245:


Those 5 keywords are all part of the fundamental SQL 92 reserved keywords list, 
though.

The documentation for the getSQLKeywords() API specifically says that it should 
only
return database-specific additional keywords, not the basic SQL Standard 
reserved words.

http://docs.oracle.com/javase/6/docs/api/java/sql/DatabaseMetaData.html#getSQLKeywords()

Have you tried asking the Liquibase community whether they might include the 
basic SQL
standard keywords automatically in their tool?


> DatabaseMetaData.getSQLKeywords() contains words that are not keywords in 
> Derby.
> 
>
> Key: DERBY-2245
> URL: https://issues.apache.org/jira/browse/DERBY-2245
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.2.1, 10.1.3.1, 
> 10.2.1.6, 10.2.2.0
>Reporter: Daniel John Debrunner
>Priority: Minor
>  Labels: derby_triage10_5_2
>
> Such as REFRESH and PUBLICATION.
> Not sure what the exact contents should be.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-2245) DatabaseMetaData.getSQLKeywords() contains words that are not keywords in Derby.

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-2245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14599708#comment-14599708
 ] 

Bryan Pendleton commented on DERBY-2245:


Can you be as specific as possible? Which keywords, precisely, did 
getSQLKeywords fail to return, which caused trouble for your tool?

> DatabaseMetaData.getSQLKeywords() contains words that are not keywords in 
> Derby.
> 
>
> Key: DERBY-2245
> URL: https://issues.apache.org/jira/browse/DERBY-2245
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 10.0.2.0, 10.0.2.1, 10.1.1.0, 10.1.2.1, 10.1.3.1, 
> 10.2.1.6, 10.2.2.0
>Reporter: Daniel John Debrunner
>Priority: Minor
>  Labels: derby_triage10_5_2
>
> Such as REFRESH and PUBLICATION.
> Not sure what the exact contents should be.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14599453#comment-14599453
 ] 

Bryan Pendleton commented on DERBY-6807:


I believe that using a properly-configured Java Security Manager policy prevents
all of the XXE attacks. However, it is possible to configure Derby to run 
without
a Java Security Manager (though that is a non-default configuration).

We could, potentially, change our invocation of the XML parser by adding either:

dBF.setFeature( "http://xml.org/sax/features/external-general-entities";, 
false );

or

dBF.setAttribute( "http://apache.org/xml/features/disallow-doctype-decl";, 
true );

However, either of these changes would potentially break legitimate use of
those XML parser features (external general entities and DOCTYPE declarations)
in some existing Derby application which has a properly configured Security
Manager and hence is not currently vulnerable to XXE attacks.

I don't know of any such applications, and such applications are at indirect 
risk
because if they should at some point be changed to no longer have the
Security Manager protections, they would become vulnerable in a subtle way.

tl;dr: I think that adding dBF.setFeature( 
"http://xml.org/sax/features/external-general-entities";, false );
to the Derby code would increase its security at the risk of potentially 
breaking
some applications.

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, externalGeneralEntities.diff, 
> xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Class not found error while creating Derby Context Aware Function with ClientDataSource driver

[Bryan Pendleton]

Caused by: java.sql.SQLSyntaxErrorException: The class 'test.vti.VTITest' does 
not exist or is inaccessible.


Is this a class that you wrote?

What is the "context aware function" that you are trying to create?

Can you use the 'derby.log' on your Derby server to isolate the
specific SQL statement that is failing?

bryan

Re: [jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

Hi Abhinav,

I played around with TriggerTest.testDerby5578InvalidateAllStatementsProc()
for a while.

It seems like this test forces us through a slightly different code path
with respect to how we call DataDictionaryImpl.getTriggerActionString().

The trigger we are working with in that test case is:

s.executeUpdate("create trigger atdc_16_trigger_1 "+
"after update of b1 on atdc_16_tab1 " +
"REFERENCING NEW AS newt "+
"for each row "+
"update atdc_16_tab2 set c2 = newt.c1");

and the table we are working with is:

s.executeUpdate("create table atdc_16_tab1 (a1 integer, b1 integer, c1 
integer)");

As we can see, this trigger references "b1" ("after update of b1") and
"c1" ("c2 = newt.c1"), but does not reference column "a1" at all.

We seem to call getTriggerActionString multiple times when running the test.

The first time we call it, we have a column set that specifies that we will be
using columns 2 and 3, which is correct because our trigger references
columns b1 and c1, and we end up building the action string:

update atdc_16_tab2 set c2 = CAST 
(org.apache.derby.iapi.db.Factory::getTriggerExecutionContext().getNewRow().getObject(2)
 A
S INTEGER)

which correctly references "c1" as column 2 in the new row.

But the SECOND time we call getTriggerActionString, the 'cols' value is null,
so we go into the "else" clause in getTriggerActionString() and that code
thinks we will be retrieving the entire table as the new row, and so we
end up building the action string:

update "APP"."ATDC_16_TAB2" set c2 = CAST 
(org.apache.derby.iapi.db.Factory::getTriggerExecutionContext().getNewRow().getObj
ect(3) AS INTEGER)

which is incorrect as now we are referencing column "c1" as the third
column, but we are only retrieving columns b1 and c1 from the table,
not column a1, so c1 is really the second column.

I think that the problem is that when we call getTriggerActionString
at runtime, with cols = null, we need to instead use our
examineTriggerNodeAndCols() method to figure out the correct set of columns,
instead of assuming that all the columns will be retrieved.

I think the code that does this is in TriggerDescriptor.getSPS, at around
line 407 of TriggerDescriptor.java.

Perhaps you could experiment with trying to have that code call
examineTriggerNodeAndCols before it calls getTriggerActionString,
and see if that gives us the right column set and affects the
behavior of TriggerTest.testDerby5578InvalidateAllStatementsProc()

thanks,

bryan

[jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14597992#comment-14597992
 ] 

Bryan Pendleton commented on DERBY-6783:


Yes, this particular test is exercising some behaviors about statement query 
plan caching
that may be hard to exactly reproduce in ij.

Have you tried running TriggerTest itself directly? You should be able to do:

java -cp '$DERBY_TRUNK/tools/java/junit.jar:$DERBY_TRUNK/classes' 
junit.textui.TestRunner
org.apache.derbyTesting.functionTests.tests.lang.TriggerTest

where you might have to fix up the "$DERBY_TRUNK" syntax in your case to match 
the exactly
location of your derby trunk directory.

> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: 6783.diff, 6783_allTestsPass.diff, 6783_moreTests.diff, 
> 6783_moreTests_bryan.diff, 6783_newTest.diff, 6783_newTest_bryan.diff, 
> 6783_newTests.diff, cleanedUpDiff.patch, diagnostics.diff, 
> error-stacktrace.out, sortFunction.diff, testTriggerWhenClause.diff
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6783:
---
Attachment: error-stacktrace.out

With the latest patch I get an error in 
org.apache.derbyTesting.functionTests.tests.lang.TriggerTest.testDerby5578InvalidateAllStatementsProc.
 

I attached the 'error-stacktrace.out' from the failed test.

> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: 6783.diff, 6783_allTestsPass.diff, 6783_moreTests.diff, 
> 6783_moreTests_bryan.diff, 6783_newTest.diff, 6783_newTest_bryan.diff, 
> 6783_newTests.diff, cleanedUpDiff.patch, diagnostics.diff, 
> error-stacktrace.out, sortFunction.diff, testTriggerWhenClause.diff
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6783:
---
Attachment: cleanedUpDiff.patch

I realize that I suggested on the derby-dev list that we take this opportunity
to clean up the comments in DataDictionaryImpl, but upon further reflection
I decided that I had made a not-so-good suggestion when I said that.

Although your new comments were in fact easier to read, they made the
overall diff substantially larger and harder to read, and I thought this
made it harder to read the diff and understand precisely what we were
changing.

So I restored the original comments.

I also removed the parts of the overall patch which were simply
diagnostic traces that we had added during our study of the behavior.

So the latest "cleanedUpDiff.patch" is just the code changes and test
changes that I think we really want to keep, with the other changes removed.

I still feel like the return code of examineTriggerNodeAndCols
should not be "String", but should instead be "int []", but I think
we could address that separately.

Please have a very close look at "cleanedUpDiff.patch" to make sure
that I didn't break anything with my changes.

I think what remains, then, is to run as many of the existing tests
as we can, to verify that we haven't broken anything that was
known to work prior to our changes, and to consider whether there
are any additional tests that we wish to write.

I think we are getting very close!

> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: 6783.diff, 6783_allTestsPass.diff, 6783_moreTests.diff, 
> 6783_moreTests_bryan.diff, 6783_newTest.diff, 6783_newTest_bryan.diff, 
> 6783_newTests.diff, cleanedUpDiff.patch, diagnostics.diff, sortFunction.diff, 
> testTriggerWhenClause.diff
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14595189#comment-14595189
 ] 

Bryan Pendleton commented on DERBY-6810:


It seems that, if I remove the "+ File.separator" from the construction of the 
URL,
then I get a file: scheme URL which works on both Windows and Linux.

I'm going to try committing that variant, and see how it behaves on the Jenkins 
builds.

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, error-stacktrace.out, 
> readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6810:
---
Attachment: error-stacktrace.out

Attached is the error I get when running XMLXXETest on Linux. I see:

Caused by: java.net.UnknownHostException: home
at 
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at sun.net.ftp.impl.FtpClient.doConnect(FtpClient.java:957)
at sun.net.ftp.impl.FtpClient.tryConnect(FtpClient.java:917)
at sun.net.ftp.impl.FtpClient.connect(FtpClient.java:1012)
at sun.net.ftp.impl.FtpClient.connect(FtpClient.java:998)
at 
sun.net.www.protocol.ftp.FtpURLConnection.connect(FtpURLConnection.java:294)
at 
sun.net.www.protocol.ftp.FtpURLConnection.getInputStream(FtpURLConnection.java:393)
at 
com.sun.org.apache.xerces.internal.impl.XMLEntityManager.setupCurrentEntity(XMLEntityManager.java:622)

Which suggests that the format of the URL in the XML document isn't quite what 
we intend it to be.

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, error-stacktrace.out, 
> readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6783:
---
Attachment: 6783_moreTests_bryan.diff

Slightly-tweaked copy of 6783_moreTests.diff; see the discussion
on derby-dev for some background about the change related to
 sortTriggerColsAndTriggerActionCols()


> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: 6783.diff, 6783_moreTests.diff, 
> 6783_moreTests_bryan.diff, 6783_newTest.diff, 6783_newTest_bryan.diff, 
> 6783_newTests.diff, diagnostics.diff, sortFunction.diff, 
> testTriggerWhenClause.diff
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

Hi Abhinav,

I found a bit of time to experiment some more with your patch.

The additional tests you have added are very interesting, thank you
very much for constructing them!

I am a bit uncertain about the new method you added:

DataDictionaryImpl.sortTriggerColsAndTriggerActionCols()

Perhaps you could provide some additional background about the problem
you are trying to address with this method?

As an experiment, I tried changing DataDictionaryImpl as follows:

/*
triggerColsAndTriggerActionCols = 
sortTriggerColsAndTriggerActionCols(
triggerColsAndTriggerActionCols, triggerActionColsOnly);
*/
Arrays.sort( triggerColsAndTriggerActionCols );

This substantially alters the behavior of TriggerWhenClauseTest.

Maybe you could try making this modification in your sandbox and
let me know what sort of results you see?

thanks,

bryan

[jira] [Commented] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14592904#comment-14592904
 ] 

Bryan Pendleton commented on DERBY-6810:


It looks like the XMLXXETest test suite is reliable on Windows, but not on Unix,
as the Jenkins builds are seeing success on all the Windows platforms, but
failures on all the Unix platforms.

I'm not quite sure what's wrong; it will take me some time to get access to
a Unix platform to work on this issue. So for the time being I'll back out
revision 1686138 so that we can get back to clean Jenkins builds.

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

I've edited the large inline comments in the latest patch. And added a few more 
tests.


Hi Abhinav,

Thanks for the updated patch! It seems you've made quite a bit of progress.

I'll try to dig into your patch in more detail soon, but it may
be a couple days until I can get to it, because I have some other
issues that are stealing my attention.

bryan

[jira] [Reopened] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton reopened DERBY-6810:


After further study, I realize that the new tests in XMLXXETest
exercise the built-in Derby XML datatype, but do not exercise
the XmlVTI.

Since the XmlVTI also creates a XML Parser and calls it, it
may also be vulnerable to XXE attacks, and so we need to
test it, as well.

Re-opening the issue because we have more tests to write.

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6807:
---
Attachment: externalGeneralEntities.diff

I ran a set of tests (tests.lang._Suite) with 'externalGeneralEntities.diff'
and the tests passed.

So this change appears to pass our existing tests, and fix the
file disclosure vulnerability described by XMLXXETest.testDerby6807FileAccess().

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, externalGeneralEntities.diff, 
> xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Need help understanding how ContextManager.cleanupOnError could create a 835MB errorStringBuilder

[Bryan Pendleton]

I am also a little concerned by the “forever” loop within the 
ContextManager.cleanupOnError.


I wish I could give you more help.

About all I can say is: that is some old, old code, and I don't
recall it being a problem.

That doesn't mean it has no bugs, but I definitely don't know of any.

That being said, going to the Derby JIRA

https://issues.apache.org/jira/browse/DERBY

and doing a search for "derby contextmanager" shows some interesting
old issues, such as DERBY-1947:

https://issues.apache.org/jira/browse/DERBY-1947

So you might want to spend some time looking through a few of those
old jobs for some possible clues.

bryan

[jira] [Commented] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590905#comment-14590905
 ] 

Bryan Pendleton commented on DERBY-6810:


Oh! In number (3) above, I had a double-negative problem. The correct line of 
code is:

dBF.setAttribute( "http://apache.org/xml/features/disallow-doctype-decl";, true 
);

(That is, set the "disallow" feature to "true" :) )

With that line in place, the behavior DOES change, and we get an error due to 
the
presence of the DOCTYPE.

ALSO, I had success with this:

dBF.setFeature( 
"http://xml.org/sax/features/external-general-entities";, false );

with this change, the xml document does not provoke an error, it just quietly
doesn't expand the &xxe entity, and the result of the entity references is an 
empty string.

Good news! This may be progress.

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590877#comment-14590877
 ] 

Bryan Pendleton commented on DERBY-6810:


I tried several experimental changes to SqlXmlUtil:

1   dBF.setExpandEntityReferences(false);
2   dBF.setFeature( XMLConstants.ACCESS_EXTERNAL_DTD, false );
3   dBF.setAttribute( 
"http://apache.org/xml/features/disallow-doctype-decl";, false );

Number (1) and number (3) had no effect on the file access behavior of 
XMLXXETest.testDerby6807FileAccess()

Number (2) didn't even compile in my environment.


> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

I was trying to understand the truncation error, I changed the values of the 
field marks1 and marks2 in the derbyTest-6783_1 and that '20' is basically the 
value of the integer field marks1 that I think is being cast as a char value. 
But I haven't been
able to track where this error is occurring.

ERROR 22001: A truncation error was encountered trying to shrink CHAR '20' 
to length 1.



At first I thought "CHAR '20'" meant a character string of length 20, but
then I looked again at the funny way that '20' was written, and perhaps
it is the hex representation of an ASCII space character which has the
character code 0x20.

If the 20 is the length of the char field, then perhaps this is because
that's how the CHAR function works:

http://db.apache.org/derby/docs/10.11/ref/rrefbuiltchar.html

If the first argument is a BIGINT: The length of the result is 20.
If the number of characters in the result is less than 20, then the
result is padded on the right with blanks to length 20.

But I'm not sure where the CHAR function is coming into play in your trigger.

I agree with you, it's not obvious where this error is arising.

I'll try to find a bit of time to dig into this myself over the next day or two.

bryan

[jira] [Resolved] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton resolved DERBY-6810.

Resolution: Fixed

I believe we've added the regression tests we planned to add.

If more ideas for tests come up, we can add more tests. So, resolving.

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: build failure because of jardriftcheck

[Bryan Pendleton]

I noticed a recent commit caused a build failure in the jardriftcheck.


Hi Myrna,

Thanks for noticing this.

I think the problem is that I updated sane.derbyTesting.jar.lastcontents,
but didn't update insane.derbyTesting.jar.lastcontents (oops).


But if this check is too much work to run on the jars everytime


No, I think it's a good tool, it's not too much work.

I just always build sane, and so I didn't think about the insane build.

I'll get it fixed straightaway.

bryan

Re: DERBY-6873

[Bryan Pendleton]

I was hoping if you could give a feedback on the progress


Hi Abhinav,

In addition to the notes I posted on JIRA, here's a few other things:

1) I'd like to look for an alternate way to code the handling of
   the "cols" variable. I think it's not the best technique to
   convert the array of column numbers into a string, and then convert
   the string back into an array of column numbers; it seems like
   we should be able to just keep the data in an array of column
   numbers and pass that around

2) Since we're doing some significant re-arranging of the
   getTriggerActionString() method, I'd like to re-work the
   large inline comments in that code; I find them very hard to read.

3) I'd like us to start thinking about how we can break this big
   patch down into several smaller patches, so that we can
   commit the overall work in several smaller changes, since this
   change has become fairly large.

4) I think we're going to need to add several new tests. I'd
   like to see different combinations of column references in
   the different parts of the trigger definition, and I'd like
   to see different data types for the columns so we get some
   feeling that we've added some good coverage here.

Overall, I think the work is going great. This is a pretty big
project, but I hope that you've learned a bunch about how Derby
handles dynamic query execution via triggers, and also about
how the Derby parser/compiler and query execution system are
implemented.

Fun stuff!

bryan

[jira] [Updated] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6783:
---
Attachment: 6783_newTest_bryan.diff

Hi Abhinav, I spent some time today studying your latest patch.

I think we are making progress!

I made a few small changes to your patch, and have attached mine.

1) In CreateTriggerNode, I removed the "whenCols" variable, and just
   used the "cols" variable in both places, because I think we want to
   combine the two sets of variables into one list.

2) In DataDictionaryImpl.examineTriggerNodeAndCols(), I added a
   couple lines of code to initialize triggerColsAndTriggerActionCols
   from referencedColsInTriggerAction, if it is passed, so that the
   columns we computed in the first invocation get carried through
   into the second invocation.

3) In DataDictionaryImpl.getTriggerActionString(), I changed the
   handling of "cols = null". We still want to execute most of the code
   in this method even if cols = null, but in that case we need to
   set triggerColsAndTriggerActionCols to contain the entire set of
   columns in the table.

The overall set of changes from your patch to mine was very small;
the easiest way to see this is to "diff the diffs" -- that is, download
your patch and my patch and put them together in a directory
and diff them.

With the patch that I've attached, we now pass all of the test
cases in TriggerWhenClauseTest except for the new "testDerby6783_1"
test, which I think is a brand new test that you have just added. That
test gets the error:

ERROR 22001: A truncation error was encountered trying to shrink CHAR '20' to 
length 1.

So clearly there are still some issues with our patch, but I felt like
this was good progress and wanted to share it with you.

Do try to download my latest patch and let me know what you think!

> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: 6783.diff, 6783_newTest.diff, 6783_newTest_bryan.diff, 
> diagnostics.diff, sortFunction.diff, testTriggerWhenClause.diff
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6818) java.lang.OutOfMemoryError

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581990#comment-14581990
 ] 

Bryan Pendleton commented on DERBY-6818:


For this particular query, what if you wrote it as:

SELECT F.ID, F.NEXT_ID, F.PREV_ID, XMLSERIALIZE(F.TEST_XML AS CLOB(1M)) 
AS TEST_XML FROM TEST F
WHERE F.ID = ( SELECT MAX( ID ) FROM TEST )

since finding the row with the max ID is the same as sorting the id descending 
and
then fetching the first row.


> java.lang.OutOfMemoryError
> --
>
> Key: DERBY-6818
> URL: https://issues.apache.org/jira/browse/DERBY-6818
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 10.11.1.1
> Environment: Using Derby disk database with JBoss eap 6.4.
>Reporter: Mattias
>
> We got a 'java.lang.OutOfMemoryError: Java heap space' when using a Derby 
> disk database with the following schema.
> {code}
> CREATE TABLE TEST (   ID BIGINT NOT NULL
> ,   NEXT _ID BIGINT
> ,   PREV_ ID BIGINT
> ,   TEST_XML XML);
> ALTER TABLE TABLE
> ADD PRIMARY KEY
> (ID);
> {code}
> I opened the heap dump with Eclipse Memory Analyzer and found that tree 
> instances of org.apache.derby.impl.store.access.sort.MergeInserter occupies 
> 290 MB of memory (that is 66 % of the total heap)  
> The problem seems to be that MergeInserter holds a sort buffer that contains 
> complete rows and the TEST_XML is about 160 kB each. MergeInserter has an 
> advanced algorithm to calculate the allowed size of the buffer, but is it 
> necessary store the complete row in the buffer, i.e. the non-indexed values 
> (TEST_XML is this case)?
> Is there some property I can set to limit the buffer size or is it possible 
> to rewrite the SQL schema someway to exclude the XML type from the buffer?
> Following is the dominator tree from Eclipse Memory Analyzer:
> {code}
> org.apache.derby.impl.store.access.sort.MergeInserter @ 0xe3f72e20
>   
>   
>|  
>  72 |   127 533 080 | 30,76%
> |- org.apache.derby.impl.store.access.sort.SortBuffer @ 0xe3f73018
>   
>   
>|  
>  48 |   127 532 792 | 30,76%
> |  |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed5e0   
>   
>   
>|  
>  40 |   164 384 |  0,04%
> |  |  '- org.apache.derby.iapi.types.DataValueDescriptor[4] @ 0xee3da280  
>   
>   
>|  
>  32 |   164 344 |  0,04%
> |  | |- org.apache.derby.iapi.types.SQLClob @ 0xee3da2a0  
>   
>   
>|  
>  56 |   164 240 |  0,04%
> |  | |  |- java.lang.String @ 0xee3da2d8  < ?xml version="1.0"?...|   
> 24 |   164 160 |  0,04%
> |  | |  |- char[1][] @ 0xee3da2f0 
>   
>   
>|  
>  24 |24 |  0,00%
> |  | |  '- Total: 2 entries   
>   
> 

[jira] [Comment Edited] (DERBY-6818) java.lang.OutOfMemoryError

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581947#comment-14581947
 ] 

Bryan Pendleton edited comment on DERBY-6818 at 6/11/15 2:03 PM:
-

Thanks for the additional data. If "ID" is the primary key, 
then "ORDER BY F.ID" should not result in a sort, I believe.

Derby is supposed to recognize when you ORDER BY a
column which is already sorted, and avoid the sort in that case.

Perhaps that algorithm is not working properly in this case?

EDIT: Oh, wait, I just read more closely: you are doing "ORDER DESCENDING". 
Yes, that will require a sort.


was (Author: bryanpendleton):
Thanks for the additional data. If "ID" is the primary key, 
then "ORDER BY F.ID" should not result in a sort, I believe.

Derby is supposed to recognize when you ORDER BY a
column which is already sorted, and avoid the sort in that case.

Perhaps that algorithm is not working properly in this case?

> java.lang.OutOfMemoryError
> --
>
> Key: DERBY-6818
> URL: https://issues.apache.org/jira/browse/DERBY-6818
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 10.11.1.1
> Environment: Using Derby disk database with JBoss eap 6.4.
>Reporter: Mattias
>
> We got a 'java.lang.OutOfMemoryError: Java heap space' when using a Derby 
> disk database with the following schema.
> {code}
> CREATE TABLE TEST (   ID BIGINT NOT NULL
> ,   NEXT _ID BIGINT
> ,   PREV_ ID BIGINT
> ,   TEST_XML XML);
> ALTER TABLE TABLE
> ADD PRIMARY KEY
> (ID);
> {code}
> I opened the heap dump with Eclipse Memory Analyzer and found that tree 
> instances of org.apache.derby.impl.store.access.sort.MergeInserter occupies 
> 290 MB of memory (that is 66 % of the total heap)  
> The problem seems to be that MergeInserter holds a sort buffer that contains 
> complete rows and the TEST_XML is about 160 kB each. MergeInserter has an 
> advanced algorithm to calculate the allowed size of the buffer, but is it 
> necessary store the complete row in the buffer, i.e. the non-indexed values 
> (TEST_XML is this case)?
> Is there some property I can set to limit the buffer size or is it possible 
> to rewrite the SQL schema someway to exclude the XML type from the buffer?
> Following is the dominator tree from Eclipse Memory Analyzer:
> {code}
> org.apache.derby.impl.store.access.sort.MergeInserter @ 0xe3f72e20
>   
>   
>|  
>  72 |   127 533 080 | 30,76%
> |- org.apache.derby.impl.store.access.sort.SortBuffer @ 0xe3f73018
>   
>   
>|  
>  48 |   127 532 792 | 30,76%
> |  |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed5e0   
>   
>   
>|  
>  40 |   164 384 |  0,04%
> |  |  '- org.apache.derby.iapi.types.DataValueDescriptor[4] @ 0xee3da280  
>   
>   
>|  
>  32 |   164 344 |  0,04%
> |  | |- org.apache.derby.iapi.types.SQLClob @ 0xee3da2a0  
>   
>   
>|  
>  56 |   164 240 |  0,04%
> |  | |  |- java.lang.String @ 0xee3da2d8  < ?xml version="1.0"?...|   
> 24 |   164 160 |  0,04%
> |  | |  |- char[1][] @ 0xee3da2f0 
>

[jira] [Commented] (DERBY-6818) java.lang.OutOfMemoryError

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581947#comment-14581947
 ] 

Bryan Pendleton commented on DERBY-6818:


Thanks for the additional data. If "ID" is the primary key, 
then "ORDER BY F.ID" should not result in a sort, I believe.

Derby is supposed to recognize when you ORDER BY a
column which is already sorted, and avoid the sort in that case.

Perhaps that algorithm is not working properly in this case?

> java.lang.OutOfMemoryError
> --
>
> Key: DERBY-6818
> URL: https://issues.apache.org/jira/browse/DERBY-6818
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 10.11.1.1
> Environment: Using Derby disk database with JBoss eap 6.4.
>Reporter: Mattias
>
> We got a 'java.lang.OutOfMemoryError: Java heap space' when using a Derby 
> disk database with the following schema.
> {code}
> CREATE TABLE TEST (   ID BIGINT NOT NULL
> ,   NEXT _ID BIGINT
> ,   PREV_ ID BIGINT
> ,   TEST_XML XML);
> ALTER TABLE TABLE
> ADD PRIMARY KEY
> (ID);
> {code}
> I opened the heap dump with Eclipse Memory Analyzer and found that tree 
> instances of org.apache.derby.impl.store.access.sort.MergeInserter occupies 
> 290 MB of memory (that is 66 % of the total heap)  
> The problem seems to be that MergeInserter holds a sort buffer that contains 
> complete rows and the TEST_XML is about 160 kB each. MergeInserter has an 
> advanced algorithm to calculate the allowed size of the buffer, but is it 
> necessary store the complete row in the buffer, i.e. the non-indexed values 
> (TEST_XML is this case)?
> Is there some property I can set to limit the buffer size or is it possible 
> to rewrite the SQL schema someway to exclude the XML type from the buffer?
> Following is the dominator tree from Eclipse Memory Analyzer:
> {code}
> org.apache.derby.impl.store.access.sort.MergeInserter @ 0xe3f72e20
>   
>   
>|  
>  72 |   127 533 080 | 30,76%
> |- org.apache.derby.impl.store.access.sort.SortBuffer @ 0xe3f73018
>   
>   
>|  
>  48 |   127 532 792 | 30,76%
> |  |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed5e0   
>   
>   
>|  
>  40 |   164 384 |  0,04%
> |  |  '- org.apache.derby.iapi.types.DataValueDescriptor[4] @ 0xee3da280  
>   
>   
>|  
>  32 |   164 344 |  0,04%
> |  | |- org.apache.derby.iapi.types.SQLClob @ 0xee3da2a0  
>   
>   
>|  
>  56 |   164 240 |  0,04%
> |  | |  |- java.lang.String @ 0xee3da2d8  < ?xml version="1.0"?...|   
> 24 |   164 160 |  0,04%
> |  | |  |- char[1][] @ 0xee3da2f0 
>   
>   
>|  
>  24 |24 |  0,00%
> |  | |  '- Total: 2 entries   
>   
&

[jira] [Commented] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581364#comment-14581364
 ] 

Bryan Pendleton commented on DERBY-6810:


This page, which discusses non-Java XML processing:

https://msdn.microsoft.com/en-us/magazine/ee335713.aspx

suggests:

The easiest way to defend against all types of XML entity attacks 
is to simply disable altogether the use of inline DTD schemas in 
your XML parsing objects. This is a straightforward application 
of the principle of attack surface reduction: if you’re not using a 
feature, turn it off so that attackers won’t be able to abuse it.

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581363#comment-14581363
 ] 

Bryan Pendleton commented on DERBY-6810:


Here is a bit of documentation about the 64,000 entity expansion limit
demonstrated by the "billion laughs" test:

http://docs.oracle.com/javase/1.5.0/docs/guide/xml/jaxp/JAXP-Compatibility_150.html#JAXP_security

I'm still having trouble finding much information about the file disclosure
vulnerability and whether there are mechanisms other than the
Java security manager which control it. This page:

http://blog.gdssecurity.com/labs/2014/6/13/castor-library-xml-external-entity-xxe-vulnerability.html

suggests that we investigate the external-general-entities and the 
external-parameter-entities
features of the XML parser, while this page:

http://www.slideshare.net/OWASP_Ottawa/pierre-ernst-xml-attack-surface-owasp-ottawa

suggests that we investigate the disallow-doctype-decl feature of the XML 
parser, and this page:

https://jaxp.java.net/1.4/JAXP-Compatibility.html#JAXP_security

suggests that all these features are automatically enabled whenever a Java 
security manager is present.

Still researching... 

> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6818) java.lang.OutOfMemoryError

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14580563#comment-14580563
 ] 

Bryan Pendleton commented on DERBY-6818:


To clarify: It was the ALTER TABLE statement that ran out of memory? If so, it
certainly seems like we only need to sort the key column(s), not the non-key 
column(s).

> java.lang.OutOfMemoryError
> --
>
> Key: DERBY-6818
> URL: https://issues.apache.org/jira/browse/DERBY-6818
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC
>Affects Versions: 10.11.1.1
> Environment: Using Derby disk database with JBoss eap 6.4.
>Reporter: Mattias
>
> We got a 'java.lang.OutOfMemoryError: Java heap space' when using a Derby 
> disk database with the following schema.
> {code}
> CREATE TABLE TEST (   ID BIGINT NOT NULL
> ,   NEXT _ID BIGINT
> ,   PREV_ ID BIGINT
> ,   TEST_XML XML);
> ALTER TABLE TABLE
> ADD PRIMARY KEY
> (ID);
> {code}
> I opened the heap dump with Eclipse Memory Analyzer and found that tree 
> instances of org.apache.derby.impl.store.access.sort.MergeInserter occupies 
> 290 MB of memory (that is 66 % of the total heap)  
> The problem seems to be that MergeInserter holds a sort buffer that contains 
> complete rows and the TEST_XML is about 160 kB each. MergeInserter has an 
> advanced algorithm to calculate the allowed size of the buffer, but is it 
> necessary store the complete row in the buffer, i.e. the non-indexed values 
> (TEST_XML is this case)?
> Is there some property I can set to limit the buffer size or is it possible 
> to rewrite the SQL schema someway to exclude the XML type from the buffer?
> Following is the dominator tree from Eclipse Memory Analyzer:
> {code}
> org.apache.derby.impl.store.access.sort.MergeInserter @ 0xe3f72e20
>   
>   
>|  
>  72 |   127 533 080 | 30,76%
> |- org.apache.derby.impl.store.access.sort.SortBuffer @ 0xe3f73018
>   
>   
>|  
>  48 |   127 532 792 | 30,76%
> |  |- org.apache.derby.impl.store.access.sort.Node @ 0xe3fed5e0   
>   
>   
>|  
>  40 |   164 384 |  0,04%
> |  |  '- org.apache.derby.iapi.types.DataValueDescriptor[4] @ 0xee3da280  
>   
>   
>|  
>  32 |   164 344 |  0,04%
> |  | |- org.apache.derby.iapi.types.SQLClob @ 0xee3da2a0  
>   
>   
>|  
>  56 |   164 240 |  0,04%
> |  | |  |- java.lang.String @ 0xee3da2d8  < ?xml version="1.0"?...|   
> 24 |   164 160 |  0,04%
> |  | |  |- char[1][] @ 0xee3da2f0 
>   
>   
>|  
>  24 |24 |  0,00%
> |  | |  '- Total: 2 entries   
>   
>   
>|  
> |   |  
> |  | |- org.apache.derby.iapi.types.SQLLongint @ 0xee3da308   
>   
>   

[jira] [Resolved] (DERBY-6817) Cannot stop Query

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton resolved DERBY-6817.

Resolution: Duplicate

Duplicate of DERBY-3908

> Cannot stop Query
> -
>
> Key: DERBY-6817
> URL: https://issues.apache.org/jira/browse/DERBY-6817
> Project: Derby
>  Issue Type: Bug
>  Components: JDBC, Network Server
>Affects Versions: 10.11.1.1
>Reporter: sagar
>
> We cannot stop a SQL query from being executed or cancel it.
> If we realize the SQL Query is wrong, and gone recursive or undeterminate 
> with very high continuous CPU usage, the Query cannot be stopped. 
> Even if we shut down the Network server the server does not shutdown and the 
> java process has to be killed ...
> There must be a mechanism to do so.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

I've been looking at the errors in derby.log of TriggerWhenClauseTest.  Some of 
them seem to be syntactical errors


I think that's probably expected.

A lot of the Derby test suites intentionally submit SQL statements
with syntax errors in them (in order to test that the syntax
errors are properly detected and a reasonable error message is
returned).

So the derby.log after running a test may be FULL of messages, many
of them completely normal.

That does make it a challenge if you are trying to distinguish the
expected error messages in derby.log from the unexpected ones.

bryan

[jira] [Commented] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14576345#comment-14576345
 ] 

Bryan Pendleton commented on DERBY-6807:


Abhinav, I took your test case and put it into a separate test suite by itself 
and
attached the new test suite to DERBY-6810. Can you have a look at your 
convenience?

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6810) Add regression tests for XXE vulnerability

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6810:
---
Attachment: readPasswordFile.diff

I took Abhinav's file access test from DERBY-6807 and re-located
it to a new test suite (XMLXXETest.java), which runs with the
special no-security-manager configuration.

This test, I think, successfully demonstrates that a malicious
XML query can use an external entity to coerce Derby into
attempting to read an arbitrary flat file on the user's filesystem.


> Add regression tests for XXE vulnerability
> --
>
> Key: DERBY-6810
> URL: https://issues.apache.org/jira/browse/DERBY-6810
> Project: Derby
>  Issue Type: Sub-task
>    Reporter: Bryan Pendleton
>Assignee: Abhinav Gupta
> Attachments: billionLaughs.diff, readPasswordFile.diff
>
>
> We should add some regression tests demonstrating that
> Derby is no longer vulnerable to an XXE assault.
> One possibility would be to have a example using a local
> file disclosure.
> Another possibility would be to have example based on the
> well-known "Billion Laughs" denial of service attack.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

Though I noticed when I clean and apply the correct patch, the build fails 
while running build_toursdb.sql
I'm trying to fix it and I'm attaching this patch as well.


Hi Abhinav,

No need to be sorry!

Thanks for the updated patch. I am able to apply it, and I see the
same problem you do; the build fails while building toursdb.

The place to look for more information about this is:

derby\trunk\generated\toursdb\toursdb.out

When I look in that file, I see that it is trying to create triggers,
and failing:

CREATE TRIGGER TRIG1 AFTER UPDATE ON FLIGHTS REFERENCING OLD AS UPDATEDROW FOR E
ACH ROW INSERT INTO FLIGHTS_HISTORY VALUES (UPDATEDROW.FLIGHT_ID, UPDATEDROW.SEG
MENT_NUMBER, UPDATEDROW.ORIG_AIRPORT,UPDATEDROW.DEPART_TIME, UPDATEDROW.DEST_AIR
PORT,UPDATEDROW.ARRIVE_TIME,UPDATEDROW.MEAL, UPDATEDROW.FLYING_TIME, UPDATEDROW.
MILES, UPDATEDROW.AIRCRAFT,'INSERTED FROM TRIG1');
ERROR XJ001: Java exception: '10: java.lang.ArrayIndexOutOfBoundsException'.

Hopefully the information in toursdb.out is enough for you to set up a
simple standalone test case in 'ij' that can reproduce the 
ArrayIndexOutOfBoundsException
and then you can learn where your code is having a problem.

thanks,

bryan

Re: [jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

Hi Abhinav,

I'd like to study your latest 6783.diff patch. But when I download it and
apply it and do a clean build, I get these errors:

compile_types:
[javac] Compiling 64 source files to C:\Users\Bryan\derby\trunk\classes
[javac] 
C:\Users\Bryan\derby\trunk\java\engine\org\apache\derby\iapi\sql\dictionary\TriggerDescriptor.java:407:
 error: method getTriggerActionString in interface DataDictionary cannot be 
applied to given types;
[javac] String newText = dd.getTriggerActionString(stmtnode,
[javac]^
[javac]   required: 
Visitable,String,String,String,int[],int[],int,TableDescriptor,int,boolean,List,String
[javac]   found: 
Visitable,String,String,String,int[],int[],int,TableDescriptor,int,boolean,
[javac]   reason: actual and formal argument lists differ in length
[javac] 
C:\Users\Bryan\derby\trunk\java\engine\org\apache\derby\impl\sql\execute\AlterTableConstantAction.java:1917:
 error: method getTriggerActionString in interface DataDictionary cannot be 
applied to given types;
[javac] String newText = dd.getTriggerActionString(node,
[javac]^
[javac]   required: 
Visitable,String,String,String,int[],int[],int,TableDescriptor,int,boolean,List,String
[javac]   found: 
Visitable,String,String,String,int[],int[],int,TableDescriptor,int,boolean,
[javac]   reason: actual and formal argument lists differ in length
[javac] 2 errors

Can you have a look and see if maybe some files are missing from the diff?

thanks,

bryan

Re: [jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

I tried to refactor getTriggerActionString, the way you suggested. It still 
doesn't solve the problem.


Hi Abhinav,

Thanks for attaching your work. I will try to spend some time looking at
it this weekend.

Please keep us updated as you continue to dig into the code and
explore it in more detail.

Possibly, it might be useful to write some additional test cases, varying
some of the columns in the different parts in the trigger definition;
sometimes that helps bring out the behaviors more clearly.

thanks,

bryan

Re: [jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

Another thing that I noticed today was, in this loop that is a part of 
getTriggerActionString

 for (ColumnReference ref : refs)
{
TableName tableName = ref.getQualifiedTableName();
  int tableBeginOffset = tableName.getBeginOffset() 
- actionOffset;

I suppose this loop should work for Status and ID when it is called by the 
action node and only for status when it is called by the when node. But when I 
print out the value of columnReference, I notice that for action node, I only 
get ID in my output
though I was expecting both ID and Status. And I get Status for when node, 
which I is what I expect.


Yes, this code is quite complex to understand.

The "refs" collection comes from the parse tree node, which is called
"actionStmt" in this subroutine, but which is really an arbitrary
node in the parse tree which is constructed by the parser.

SortedSet refs = getTransitionVariables(
actionStmt, oldReferencingName, newReferencingName);

So the first time we call this, actionStmt is indeed the action
statement for our trigger, which is:

UPDATE table SET result='completed' WHERE id=newrow.id

In this parse tree, the only reference to a column in "newrow" is
"newrow.id", which is why ID is the only column that comes back
in the "refs" collection.

The second time we call this, actionStmt is pointing to the parse
tree for the WHEN clause, which is:

WHEN (newrow.status='d')

In this parse tree, the only reference to a column in "newrow" is
"newrow.status", which is why STATUS is the only column that comes
back in the "refs" collection the second time.

The argument

int[] referencedCols,

that is passed to DataDictionaryImpl.getTriggerActionString() also contains
the STATUS column; this is because elsewhere we have analyzed:

... AFTER UPDATE OF status ON ...

and determined that STATUS is the column that "fires" the trigger.

bryan

Re: [jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

Hi Abhinav,

I was able to build and run your sort code, and it seemed to be behaving
as you expected it to, but the test case still seemed to be generating
the wrong code, such that the "when" clause was accessing the value of
the ID column instead of accessing the value of the STATUS column.)

I feel like part of the problem is that the method

DataDictionaryImpl.getTriggerActionString()

is trying to do two different things:

1) It is computing the set of columns that are referenced by the parse node

2) And it is using that set of columns to generate the trigger action SQL

I think that the source of the bug is that we FIRST need to do task (1)
for both the trigger action node and the trigger when node, and THEN
do task (2) for both the trigger action node and the trigger when node.

But since the two tasks are bundled into a single routine, we end up
doing them in this order:

(1) (for the action node)
(2) (for the action node)
(1) (for the WHEN clause)
(2) (for the WHEN clause)

Instead of doing them in this order:

(1) (for the action node)
(1) (for the WHEN clause)
(2) (for the action node)
(2) (for the WHEN clause)

I think that if we re-factored getTriggerActionString() into
two different routines, for these two different tasks, then
we could have CreateTriggerNode call the code in the order (1,1,2,2)
instead of the order (1,2,1,2), and then that might fix the problem.

So in CreateTriggerNode, instead of code that looks like:

transformedActionText =
getDataDictionary().getTriggerActionString(actionNode, ...);

// If there is a WHEN clause, we need to transform its text too.
if (whenClause != null)
transformedWhenText =
 getDataDictionary().getTriggerActionString(whenClause, ...);

We'd have code more like:

getDataDictionary().examineTriggerNodeAndCols(actionNode,...);
if (whenClause != null)
getDataDictionary().examineTriggerNodeAndCols(whenClause,...);

transformedActionText =
getDataDictionary().getTriggerActionString(actionNode, ...);

// If there is a WHEN clause, we need to transform its text too.
if (whenClause != null)
transformedWhenText =
 getDataDictionary().getTriggerActionString(whenClause, ...);

What do you think? Is this idea worth trying?

thanks,

bryan

Re: [jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton]

I spent quite some time on this bug today and there was a slight problem with 
my sort function which I corrected,


Wonderful! Thank you for the updates.

I see that you have attached the new version to DERBY-6783, and
I will try to download that to my sandbox and see what behavior I see.

thanks,

bryan

[jira] [Commented] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568473#comment-14568473
 ] 

Bryan Pendleton commented on DERBY-6807:


I changed line 96 of XMLTypeAndOpsTest.java so it read:

return SecurityManagerSetup.noSecurityManager(new XMLTestSetup(suite));

which causes this entire test suite to run without the standard Derby Security 
manager
in place. After I did this, the new test case now results in:

Expected: >NULL<
Found:>HelloWorld^M
<

I think this means that the test case therefore demonstrates the bug.

However, enabling FEATURE_SECURE_PROCESSING in SqlXmlUtils.java had
no effect on running the test.

Moreover, I don't think that we want *all* of XMLTypeAndOpsTest to run with
no security manager in place, so I think this means that we want to take our
two new test cases (the test case attached to this issue, and the test case
attached to DERBY-6810), and put them in a new test suite by themselves,
with no security manager installed, and see if that standalone test suite
demonstrates the vulnerability.

Then, we still have to figure out where to put the FEATURE_SECURE_PROCESSING
to address the vulnerability.



> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryan Pendleton updated DERBY-6807:
---
Attachment: error-stacktrace.out

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: error-stacktrace.out, xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6807) XXE attack possible by using XmlVTI and the XML datatype

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566855#comment-14566855
 ] 

Bryan Pendleton commented on DERBY-6807:


The 'database not found' message in the derby.log file is normal. It is tricky 
to read the
Derby test results. The Derby test harness always uses a test database called 
'Wombat',
and the harness starts by attempting to delete any old copy of the test 
database before
it begins the new tests. In your case, since you are running the tests on a 
clean working
directory (junit-single always does this), there is no old copy of the test 
database so
the 'Wombat' not found is totally normal.

The more interesting output is in the 'error-stacktrace.out'. When I run your 
test, I get
an error that I am attaching as 'error-stacktrace.out'. Note that the test's 
attempt to
get the XML parser to access a system file has been detected and prevented by 
the
system security manager, which has returned a error exception:
   AccessControlException: access denied ("java.util.PropertyPermission" 
"user.dir" "read")

Again, as I did in DERBY-6810, I made an experimental modification to 
SqlXmlUtils.java
to enable FEATURE_SECURE_PROCESSING, and I detected no change in the
behavior of the test.

I think this means, that, for the test to demonstrate the XXE vulnerability, we 
will need
to disable the Java Security Manager that normally runs with the test.

Can you have a look in your 'error-stacktrace.out' and see if your results 
match mine?

> XXE attack possible by using XmlVTI and the XML datatype
> 
>
> Key: DERBY-6807
> URL: https://issues.apache.org/jira/browse/DERBY-6807
> Project: Derby
>  Issue Type: Bug
>Affects Versions: 10.11.1.1
>Reporter: Rick Hillegas
> Attachments: xmltest.diff
>
>
> The Derby XML datatype and XmlVTI can be exploited, via XXE-based attacks, to 
> expose sensitive information or launch denial-of-service assaults. This issue 
> has CVE id CVE-2015-1832. This issue was brought to our attention by Philippe 
> Arteau.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6813) Undocumented ON DELETE SET DEFAULT is parsed, accepted, but not enforced

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1452#comment-1452
 ] 

Bryan Pendleton commented on DERBY-6813:


Often, but not always, situations like this where Derby accepts a certain syntax
but doesn't implement the behavior turn out to be cases where the original
Cloudscape product actually *did* implement the behavior, but the implementation
was disabled during a time when that behavior was felt to be undesired.

If that is true in this case, it may be a relatively small project to re-enable 
it.

If you wanted, you could do some investigation into the source tree to see
if you can gather any clues about this theory; I'd be glad to assist with that.

The first step is to identify the part of the SQL grammar where that 
construction
is recognized, and then look through the Subversion history to see what you can
learn about that part of the parser's history.

> Undocumented ON DELETE SET DEFAULT is parsed, accepted, but not enforced
> 
>
> Key: DERBY-6813
> URL: https://issues.apache.org/jira/browse/DERBY-6813
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Lukas Eder
>Priority: Minor
>
> The REFERENCES clause implements almost all SQL standard actions for ON 
> DELETE and ON UPDATE:
> https://db.apache.org/derby/docs/10.11/ref/rrefsqlj16357.html#rrefsqlj16357
> ON DELETE SET DEFAULT is not explicitly supported by Derby (although, it 
> would be nice). But the Derby parser accepts it, but doesn't enforce it as 
> can be seen in this script
> {code}
> create table "t1"("v" int not null)
> create table "t2"("w" int)
> alter table "t1" add constraint "pk" primary key ("v")
> insert into "t1" ("v") values (1), (2)
> insert into "t2" ("w") values (1)
> alter table "t2" alter "w" set default 2
> alter table "t2" add constraint "t2_fk" foreign key ("w") references "t1" 
> ("v") on delete set default
> delete from "t1" where "v" = 1 -- this fails, but it shouldn't fail
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565277#comment-14565277
 ] 

Bryan Pendleton commented on DERBY-6783:


My theory is that the root cause of this bug is the following code from 
CreateTriggerNode.bindReferencesClause():

//Now that we have verified that are no invalid column references
//for trigger columns, let's go ahead and transform the OLD/NEW
//transient table references in the trigger action sql.
transformedActionText = getDataDictionary().getTriggerActionString(a
ctionNode,
oldTableName,
newTableName,
originalActionText,
referencedColInts,
referencedColsInTriggerAction,
actionNode.getBeginOffset(),
triggerTableDescriptor,
triggerEventMask,
true,
actionTransformations);

// If there is a WHEN clause, we need to transform its text too.
if (whenClause != null) {
transformedWhenText =
getDataDictionary().getTriggerActionString(
whenClause, oldTableName, newTableName,
originalWhenText, referencedColInts,
referencedColsInTriggerAction,
whenClause.getBeginOffset(),
triggerTableDescriptor, triggerEventMask, true,
whenClauseTransformations);
}

Note that this code calls getTriggerActionString() twice.

The first time, we are working with the Trigger "action node", which references 
the column "id":

UPDATE t1 SET done_date=current_date WHERE id=newrow.id;

and also references the column "status":

AFTER UPDATE OF status ON t1

The second time, we are working with the Trigger "when node", which references 
the column "status":

WHEN (newrow.status='d')

and also references the column "status":

AFTER UPDATE OF status ON t1

The net result is that, the first time through, we determine that we will have 
two columns (id, status),
but the second time through, we incorrectly determine that the 'newrow' will 
have only one column (status),
and therefore we generate the reference to column 1, thinking we will be 
referencing the column "status",
but instead we end up referencing the column "id".

To fix this bug, we have to somehow change the compilation of the trigger so 
that it
computes the set of columns that we will need in 'newrow' exactly once, and use 
that
set of columns for compiling both the action node and the when node, instead of
computing their column sets independently.


> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: diagnostics.diff, testTriggerWhenClause.diff
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565214#comment-14565214
 ] 

Bryan Pendleton commented on DERBY-6783:


Here's a workaround: If the trigger definition is changed to:

   CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 
 REFERENCING NEW AS newrow FOR EACH ROW 
WHEN (newrow.id > 0 and newrow.status='d') 
UPDATE t1 SET done_date=current_date WHERE id=newrow.id;

then everything works fine.

The reason this workaround works is that it causes the set of columns
in the trigger's WHEN clause (id, status) to be the same as the set
of columns in the trigger's REFERENCING clause (id, status).

> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: diagnostics.diff, testTriggerWhenClause.diff
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-4003) Allow user-defined functions in GROUP BY expressions

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-4003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14565065#comment-14565065
 ] 

Bryan Pendleton commented on DERBY-4003:


Does this help at all? 
http://blog.jooq.org/2012/05/11/simulation-of-trunc-in-derby/


> Allow user-defined functions in GROUP BY expressions
> 
>
> Key: DERBY-4003
> URL: https://issues.apache.org/jira/browse/DERBY-4003
> Project: Derby
>  Issue Type: Improvement
>  Components: SQL
>Affects Versions: 10.5.1.1
>Reporter: Rick Hillegas
>  Labels: derby_triage10_10
>
> Derby does not let you GROUP BY an expression which involves a user-defined 
> function. Technically, I think that the SQL standard forbids grouping by 
> anything other than a plain column reference. See chapter 2 of the 2003 SQL 
> standard, section 7.9 . However, I think that many other 
> databases allow this useful extension. Derby already lets you GROUP BY 
> expressions involving system functions--this extension was added as part of 
> DERBY-883. The discussion around that issue raised the concern that you 
> should only be able to use DETERMINISTIC user-defined functions in GROUP BY 
> expressions. See 
> http://www.nabble.com/Functions-in-GROUP-BY-expressions--(related-to-DERBY-883)-td7021186.html
> A follow-on email thread pointed out that you can work around this limitation 
> by putting your expressions inside a subquery and then grouping by a select 
> from the subquery results: 
> http://www.nabble.com/User-Defined-Functions-in-a-Group-By-Clause-td21326165.html#a21326165
> Release 10.5 will add the DETERMINISTIC keyword to function declarations, so 
> now we should be able to allow DETERMINISTIC user-defined functions in GROUP 
> BY expressions.
> We could further relax the current limitation by also allowing 
> non-DETERMINISTIC functions in GROUP BY expressions. The distinction between 
> DETERMINISTIC and non-DETERMINISTIC functions does not seem to me to be 
> rooted in the SQL standard since the standard only allows plain column 
> references. Using the subquery-workaround mentioned above, you can already 
> ask for non-deterministic grouped results. However, there may be some 
> implementation reasons for limiting this extension to DETERMINISTIC functions.
> Here is a script showing the issue:
> drop table t;
> drop function f;
> create table t( a int, b int );
> insert into t(a, b) values ( 1, 0 ), ( -1, 1 ), ( -2, 2 );
> create function f
> (
> raw int
> )
> returns int
> language java
> parameter style java
> deterministic
> no sql
> external name 'java.lang.Math.abs'
> ;
> select abs( a ), count(*)
> from t
> group by abs( a );
> select f( a ), count(*)
> from t
> group by f( a );



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DERBY-6783) WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script below

[Bryan Pendleton (JIRA)]

[ 
https://issues.apache.org/jira/browse/DERBY-6783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14564136#comment-14564136
 ] 

Bryan Pendleton commented on DERBY-6783:


In DataDictionaryImpl.getTriggerActionString, there is some very complex logic
which computes the "table position" of each column in the trigger versus the
"trigger position" of that column (the terms "table position" and "trigger 
position"
are mine; they do not arise directly in the code).

In this code, in our test case, the data structures correctly represent that the
STATUS column is column 3 in the base table:

[junit] Trigger column is STATUS, so columnDescriptor is columnName: STATUS
[junit] columnPosition: 3
[junit] columnType: CHAR(1)

But then, at the point where that method builds the 
"triggerColsAndTriggerActionCols"
variable, it thinks that "STATUS" is the only column that will be retrieved, 
and so
it sets the column position to 1.

But, in fact, the execution of the query retrieves *two* columns: ID and 
STATUS, to
the correct column position should have been 2.

So I think that the bug arises in DataDictionaryImpl.getTriggerActionString.

Interestingly, we seem to call this routine *twice*: once with the set of 
columns (ID,STATUS),
and once with only (STATUS) in the column list.

So possibly the flaw is that either (a) we should only be calling this routine 
once, or
(b) we should be calling it twice, but in each case the column list should be
(ID,STATUS), but in the second case the column list is wrong, it is only 
(STATUS).

Abhinav, perhaps you can focus your attention on 
DataDictionaryImpl.getTriggerActionString
for a while and tell me what you think about how this method is interpreting 
the trigger
definition in this particular test case.

> WHEN clause in CREATE TRIGGER for UPDATE is not working for the sql script 
> below
> 
>
> Key: DERBY-6783
> URL: https://issues.apache.org/jira/browse/DERBY-6783
> Project: Derby
>  Issue Type: Bug
>  Components: SQL
>Affects Versions: 10.11.1.1
>Reporter: Mamta A. Satoor
>Assignee: Abhinav Gupta
> Attachments: diagnostics.diff, testTriggerWhenClause.diff
>
>
> Following sql script was shared on 
> derby-user(http://mail-archives.apache.org/mod_mbox/db-derby-user/201412.mbox/%3c548aba6d.8000...@zoho.com%3e).
> The UPDATE TRIGGER  with the WHEN clause below does not fire as expected. 
> Same script works fine on DB2.
> ij version 10.11 
>  ij> connect 'jdbc:derby:MyDbTest;create=true'; 
>  ij> CREATE TABLE t1 (id INTEGER, done_date DATE, status CHAR(1)); 
>  0 rows inserted/updated/deleted 
>  ij> CREATE TRIGGER tr1 AFTER UPDATE OF status ON t1 REFERENCING NEW AS 
> newrow FOR EACH ROW WHEN (newrow.status='d') UPDATE t1 SET 
> done_date=current_date WHERE id=newrow.id; 
>  0 rows inserted/updated/deleted 
>  ij> insert into t1 values (1, null, 'a'); 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |a
>   
>  1 row selected 
>  ij> UPDATE t1 SET status='d'; 
>  1 row inserted/updated/deleted 
>  ij> SELECT * FROM t1; 
>  ID |DONE_DATE |STA& 
>  --- 
>  1  |NULL  |d
>   
>  1 row selected 
>  ij> exit; 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)