Re: drill tests not passing

[Mike Beckerle]

Thank you for your help.

To review,  this command works and succeeds:

mvn clean install -DskipTests=true

mvn test ... with args per your prior email...

fails with the "drill-java-exec: Artifact has not been packaged yet..."
error.

On Fri, Aug 25, 2023 at 3:54 AM James Turton  wrote:

> Okay, that attempt couldn't even build java-exec. What does the
> following do when run from the root of the source tree?"
>
> mvn package -f exec/vector/pom.xml
>

The above builds successfully.

The below builds, but when executing tests, huge number of errors. File
attached.


> mvn package -f exec/java-exec/pom.xml
>
> On 2023/08/24 17:09, Mike Beckerle wrote:
> >
> > Still no luck.
> >
> > Output from your mvn-test-drill command is attached.
> >
> > drill-java-exec: Artifact has not been packaged yet. 
> >
> >
> > On Mon, Aug 21, 2023 at 10:36 AM James Turton  > > wrote:
> >
> > __
> > Hi Mike
> >
> > I took a look at the build log that you shared with me recently and
> > what happened in that run was OOM (first report on line 9348).
> >
> > |   9355 2198813 ERROR [UserServer-1]
> > [org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC
> > communication.9355  Connection: /127.0.0.1:31046
> >  <--> /127.0.0.1:39558
> >  (user server).  Closing connection.
> > 9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
> > |
> > My command to run Drill tests set memory limits as follows.
> >
> > |alias mvn-test-drill="mvn test \
> >  -Djunit.args=\"-Duser.timezone=UTC -Duser.language=en
> > -Duser.region=US\" \
> >  -DmemoryMb=2560 -DdirectMemoryMb=2560 \
> >  -DforkCount=2"
> > |
> > Hope this helps...
> > James
> >
> > On 2023/07/25 00:00, Mike Beckerle wrote:
> >> Hi drill devs,
> >>
> >> I'm still stuck on this problem. Can anyone suggest a way past this?
> >>
> >> Mike Beckerle
> >> Apache Daffodil PMC |daffodil.apache.org  <
> http://daffodil.apache.org>
> >> OGF DFDL Workgroup Co-Chair |
> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl  <
> http://www.ogf.org/ogf/doku.php/standards/dfdl/dfdl>
> >> Owl Cyber Defense |www.owlcyberdefense.com  <
> http://www.owlcyberdefense.com>
> >>
> >>
> >>
> >> On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle
>   wrote:
> >>
> >>> Looks like I attached the wrong file. Doing too many things at
> once.
> >>>
> >>> The correct file is attached here.
> >>>
> >>>
> >>>
> >>> On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle
> 
> >>> wrote:
> >>>
>  Update: I did a clean and install -DskipTests=true.
> 
>  Then I tried the mvn test using the non-UTC timezone stuff, as
> suggested.
> 
>  But alas, it still fails, this time the failure unique and is
> only in
>  "Java Execution Engine"
> 
>  [ERROR] Failed to execute goal
>  org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
>  (unpack-vector-types) on project drill-java-exec: Artifact has
> not been
>  packaged yet. When used on reactor artifact, unpack should be
> executed
>  after packaging: see MDEP-98. -> [Help 1]
> 
>  The command and complete trace output are below.
> 
>  I need assistance on how to proceed.
> 
>  Complete trace from the mvn test is attached.
> 
> 
>  On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle<
> mbecke...@apache.org>  
>  wrote:
> 
> > To answer questions:
> >
> > 1. Paul: This is a 100% stock build. All I have done is clone
> the repo
> > (master branch). Make a new git branch (in case I make future
> changes). Try
> > to build (success) and test (failed so far).
> >
> > 2. James: The /opt/drill directory I created is owned by my
> userid and
> > has full read/write access for all the development activities. I
> just put
> > it there so it would have a shorter path to fix the first
> Hive-related
> > glitch I encountered with the Linux 255 limit on file pathname
> length.
> >
> > I will try the suggested maven command line for non-UTC and see
> if
> > things improve.
> >
> > The challenge for me as a newby is how do I know if I have
> everything
> > properly configured?
> >
> > Can I just turn off building and testing of the Hive-related
> stuff in
> > some supported/well-known way?
> >
> > If so, I would suggest I'd like to turn off not just Hive, but
> *as much
> > as possible*. I really just need the embedded drill to work.
> >
> > I would agree with @Charles Givre   cgi...@gmail.com>   that a contrib
> > package addition is the ideal approach and that's 

Re: drill tests not passing

[James Turton]

Okay, that attempt couldn't even build java-exec. What does the 
following do when run from the root of the source tree?"


mvn package -f exec/vector/pom.xml
mvn package -f exec/java-exec/pom.xml

On 2023/08/24 17:09, Mike Beckerle wrote:


Still no luck.

Output from your mvn-test-drill command is attached.

drill-java-exec: Artifact has not been packaged yet. 


On Mon, Aug 21, 2023 at 10:36 AM James Turton > wrote:


__
Hi Mike

I took a look at the build log that you shared with me recently and
what happened in that run was OOM (first report on line 9348).

|   9355 2198813 ERROR [UserServer-1]
[org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC
communication.    9355  Connection: /127.0.0.1:31046
 <--> /127.0.0.1:39558
 (user server).  Closing connection.
    9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
|
My command to run Drill tests set memory limits as follows.

|alias mvn-test-drill="mvn test \
     -Djunit.args=\"-Duser.timezone=UTC -Duser.language=en
-Duser.region=US\" \
     -DmemoryMb=2560 -DdirectMemoryMb=2560 \
     -DforkCount=2"
|
Hope this helps...
James

On 2023/07/25 00:00, Mike Beckerle wrote:

Hi drill devs,

I'm still stuck on this problem. Can anyone suggest a way past this?

Mike Beckerle
Apache Daffodil PMC |daffodil.apache.org  
OGF DFDL Workgroup Co-Chair |www.ogf.org/ogf/doku.php/standards/dfdl/dfdl  

Owl Cyber Defense |www.owlcyberdefense.com  



On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle  
  wrote:


Looks like I attached the wrong file. Doing too many things at once.

The correct file is attached here.



On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle  

wrote:


Update: I did a clean and install -DskipTests=true.

Then I tried the mvn test using the non-UTC timezone stuff, as suggested.

But alas, it still fails, this time the failure unique and is only in
"Java Execution Engine"

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
(unpack-vector-types) on project drill-java-exec: Artifact has not been
packaged yet. When used on reactor artifact, unpack should be executed
after packaging: see MDEP-98. -> [Help 1]

The command and complete trace output are below.

I need assistance on how to proceed.

Complete trace from the mvn test is attached.


On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle  

wrote:


To answer questions:

1. Paul: This is a 100% stock build. All I have done is clone the repo
(master branch). Make a new git branch (in case I make future changes). Try
to build (success) and test (failed so far).

2. James: The /opt/drill directory I created is owned by my userid and
has full read/write access for all the development activities. I just put
it there so it would have a shorter path to fix the first Hive-related
glitch I encountered with the Linux 255 limit on file pathname length.

I will try the suggested maven command line for non-UTC and see if
things improve.

The challenge for me as a newby is how do I know if I have everything
properly configured?

Can I just turn off building and testing of the Hive-related stuff in
some supported/well-known way?

If so, I would suggest I'd like to turn off not just Hive, but *as much
as possible*. I really just need the embedded drill to work.

I would agree with @Charles Givre  
   that a contrib
package addition is the ideal approach and that's what I'll be attempting.

-mikeb

On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  
  wrote:


I'll add some heresy here... IMHO, for the purposes of developing a
DFDL extension, you probably don't need all the Drill tests to run.  For
your project, my suggestion would be to add a module to the contrib package
and that way your changes are relatively self contained.
Best,
-- C




On Jul 13, 2023, at 10:27 AM, James Turton  
  wrote:

Hi Mike

Here's the command line I use to run tests on a machine that's not in

the UTC time zone (plus some unrelated memory size arguments).

mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en

-Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560

I have one other question to add to Paul's comments - does the OS

user that you're running Maven under have write access to all of the source
tree that you put at /opt/drill?

On 2023/07/11 22:12, Paul Rogers wrote:

Hi Mike,

 

Re: drill tests not passing

[Mike Beckerle]

Still no luck.

Output from your mvn-test-drill command is attached.

drill-java-exec: Artifact has not been packaged yet. 


On Mon, Aug 21, 2023 at 10:36 AM James Turton  wrote:

> Hi Mike
>
> I took a look at the build log that you shared with me recently and what
> happened in that run was OOM (first report on line 9348).
>
>9355 2198813 ERROR [UserServer-1]
> [org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC
> communication.9355  Connection: /127.0.0.1:31046 <--> /127.0.0.1:39558
> (user server).  Closing connection.
>9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
>
> My command to run Drill tests set memory limits as follows.
>
> alias mvn-test-drill="mvn test \
> -Djunit.args=\"-Duser.timezone=UTC -Duser.language=en
> -Duser.region=US\" \
> -DmemoryMb=2560 -DdirectMemoryMb=2560 \
> -DforkCount=2"
>
> Hope this helps...
> James
>
> On 2023/07/25 00:00, Mike Beckerle wrote:
>
> Hi drill devs,
>
> I'm still stuck on this problem. Can anyone suggest a way past this?
>
> Mike Beckerle
> Apache Daffodil PMC | daffodil.apache.org
> OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> Owl Cyber Defense | www.owlcyberdefense.com
>
>
>
> On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle  
>  wrote:
>
>
> Looks like I attached the wrong file. Doing too many things at once.
>
> The correct file is attached here.
>
>
>
> On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle  
> 
> wrote:
>
>
> Update: I did a clean and install -DskipTests=true.
>
> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
>
> But alas, it still fails, this time the failure unique and is only in
> "Java Execution Engine"
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
> (unpack-vector-types) on project drill-java-exec: Artifact has not been
> packaged yet. When used on reactor artifact, unpack should be executed
> after packaging: see MDEP-98. -> [Help 1]
>
> The command and complete trace output are below.
>
> I need assistance on how to proceed.
>
> Complete trace from the mvn test is attached.
>
>
> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle  
> 
> wrote:
>
>
> To answer questions:
>
> 1. Paul: This is a 100% stock build. All I have done is clone the repo
> (master branch). Make a new git branch (in case I make future changes). Try
> to build (success) and test (failed so far).
>
> 2. James: The /opt/drill directory I created is owned by my userid and
> has full read/write access for all the development activities. I just put
> it there so it would have a shorter path to fix the first Hive-related
> glitch I encountered with the Linux 255 limit on file pathname length.
>
> I will try the suggested maven command line for non-UTC and see if
> things improve.
>
> The challenge for me as a newby is how do I know if I have everything
> properly configured?
>
> Can I just turn off building and testing of the Hive-related stuff in
> some supported/well-known way?
>
> If so, I would suggest I'd like to turn off not just Hive, but *as much
> as possible*. I really just need the embedded drill to work.
>
> I would agree with @Charles Givrethat 
> a contrib
> package addition is the ideal approach and that's what I'll be attempting.
>
> -mikeb
>
> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  
>  wrote:
>
>
> I'll add some heresy here... IMHO, for the purposes of developing a
> DFDL extension, you probably don't need all the Drill tests to run.  For
> your project, my suggestion would be to add a module to the contrib package
> and that way your changes are relatively self contained.
> Best,
> -- C
>
>
>
>
> On Jul 13, 2023, at 10:27 AM, James Turton  
>  wrote:
>
> Hi Mike
>
> Here's the command line I use to run tests on a machine that's not in
>
> the UTC time zone (plus some unrelated memory size arguments).
>
> mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
>
> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
>
> I have one other question to add to Paul's comments - does the OS
>
> user that you're running Maven under have write access to all of the source
> tree that you put at /opt/drill?
>
> On 2023/07/11 22:12, Paul Rogers wrote:
>
> Hi Mike,
>
> A quick glance at the log suggests a failure in the tests for the
>
> JSON
>
> reader, in the Mongo extended types. Drill's date/time support has
> historically been fragile. Some tests only work if your machine is
>
> set to
>
> use the UTC time zone (or Java is told to pretend that the time is
>
> UTC.)
>
> The Mongo types test failure seems to be around a date/time test so
>
> maybe
>
> this is the issue?
>
> There are also failures indicating that the Drillbit (Drill server)
>
> died.
>
> Not sure how this can happen, as tests run Drill embedded (or used
>
> to.)
>
> Looking earlier in the logs, it seems that the Drillbit didn't start
>
> due to
>
> UDF (user-defined function) failures:
>
> Found 

Re: drill tests not passing

[James Turton]

Hi Mike

I took a look at the build log that you shared with me recently and what 
happened in that run was OOM (first report on line 9348).


|   9355 2198813 ERROR [UserServer-1] 
[org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC 
communication.    9355  Connection: /127.0.0.1:31046 <--> 
/127.0.0.1:39558 (user server).  Closing connection.

   9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
|
My command to run Drill tests set memory limits as follows.

|alias mvn-test-drill="mvn test \
    -Djunit.args=\"-Duser.timezone=UTC -Duser.language=en 
-Duser.region=US\" \

    -DmemoryMb=2560 -DdirectMemoryMb=2560 \
    -DforkCount=2"
|
Hope this helps...
James

On 2023/07/25 00:00, Mike Beckerle wrote:

Hi drill devs,

I'm still stuck on this problem. Can anyone suggest a way past this?

Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair |www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense |www.owlcyberdefense.com



On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle  wrote:


Looks like I attached the wrong file. Doing too many things at once.

The correct file is attached here.



On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle
wrote:


Update: I did a clean and install -DskipTests=true.

Then I tried the mvn test using the non-UTC timezone stuff, as suggested.

But alas, it still fails, this time the failure unique and is only in
"Java Execution Engine"

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
(unpack-vector-types) on project drill-java-exec: Artifact has not been
packaged yet. When used on reactor artifact, unpack should be executed
after packaging: see MDEP-98. -> [Help 1]

The command and complete trace output are below.

I need assistance on how to proceed.

Complete trace from the mvn test is attached.


On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle
wrote:


To answer questions:

1. Paul: This is a 100% stock build. All I have done is clone the repo
(master branch). Make a new git branch (in case I make future changes). Try
to build (success) and test (failed so far).

2. James: The /opt/drill directory I created is owned by my userid and
has full read/write access for all the development activities. I just put
it there so it would have a shorter path to fix the first Hive-related
glitch I encountered with the Linux 255 limit on file pathname length.

I will try the suggested maven command line for non-UTC and see if
things improve.

The challenge for me as a newby is how do I know if I have everything
properly configured?

Can I just turn off building and testing of the Hive-related stuff in
some supported/well-known way?

If so, I would suggest I'd like to turn off not just Hive, but *as much
as possible*. I really just need the embedded drill to work.

I would agree with @Charles Givre   that a contrib
package addition is the ideal approach and that's what I'll be attempting.

-mikeb

On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  wrote:


I'll add some heresy here... IMHO, for the purposes of developing a
DFDL extension, you probably don't need all the Drill tests to run.  For
your project, my suggestion would be to add a module to the contrib package
and that way your changes are relatively self contained.
Best,
-- C




On Jul 13, 2023, at 10:27 AM, James Turton  wrote:

Hi Mike

Here's the command line I use to run tests on a machine that's not in

the UTC time zone (plus some unrelated memory size arguments).

mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en

-Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560

I have one other question to add to Paul's comments - does the OS

user that you're running Maven under have write access to all of the source
tree that you put at /opt/drill?

On 2023/07/11 22:12, Paul Rogers wrote:

Hi Mike,

A quick glance at the log suggests a failure in the tests for the

JSON

reader, in the Mongo extended types. Drill's date/time support has
historically been fragile. Some tests only work if your machine is

set to

use the UTC time zone (or Java is told to pretend that the time is

UTC.)

The Mongo types test failure seems to be around a date/time test so

maybe

this is the issue?

There are also failures indicating that the Drillbit (Drill server)

died.

Not sure how this can happen, as tests run Drill embedded (or used

to.)

Looking earlier in the logs, it seems that the Drillbit didn't start

due to

UDF (user-defined function) failures:

Found duplicated function in drill-custom-lower.jar:
custom_lower(VARCHAR-REQUIRED)
Found duplicated function in built-in: lower(VARCHAR-REQUIRED)

Not sure how this could occur: it should have failed in all builds.

Also:

File


/opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar

does not exist on file systemfile:///

This is complaining that Drill needs the source code (not 

Re: drill tests not passing

[Mike Beckerle]

Charles,

fixed your testComplexXSD in your xsd_reader branch.

https://github.com/cgivre/drill/pull/5

Need to add attribute support, but this is quite close.




On Fri, Jul 14, 2023 at 5:59 PM Charles Givre  wrote:

> Hi Mike,
> One more thing... I've been working on an XSD Reader for Drill for some
> time.  (This is still very buggy)
> https://github.com/cgivre/drill/tree/xsd_reader
>
>  What this does is attempt to convert a XML XSD file into a Drill Schema.
> Best,
> -- C
>
>
>
> On Jul 14, 2023, at 2:20 PM, Charles Givre  wrote:
>
> Mike,
> Are you able to build Drill w/o the tests?  If so, my suggestion is really
> just to start working on the DFDL extensions.  I've been doing Drill stuff
> for far too long and really haven't needed to run the full battery of unit
> tests locally.  As long as you can build it and can execute individual unit
> tests, you should be ok.  Others may disagree, but for what you're doing,
> I'd think it would be fine.
> Best,
> -- C
>
>
>
> On Jul 14, 2023, at 2:04 PM, Mike Beckerle  wrote:
>
> Update: I did a clean and install -DskipTests=true.
>
> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
>
> But alas, it still fails, this time the failure unique and is only in
> "Java Execution Engine"
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
> (unpack-vector-types) on project drill-java-exec: Artifact has not been
> packaged yet. When used on reactor artifact, unpack should be executed
> after packaging: see MDEP-98. -> [Help 1]
>
> The command and complete trace output are below.
>
> I need assistance on how to proceed.
>
> Complete trace from the mvn test is attached.
>
>
> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle 
> wrote:
>
>> To answer questions:
>>
>> 1. Paul: This is a 100% stock build. All I have done is clone the repo
>> (master branch). Make a new git branch (in case I make future changes). Try
>> to build (success) and test (failed so far).
>>
>> 2. James: The /opt/drill directory I created is owned by my userid and
>> has full read/write access for all the development activities. I just put
>> it there so it would have a shorter path to fix the first Hive-related
>> glitch I encountered with the Linux 255 limit on file pathname length.
>>
>> I will try the suggested maven command line for non-UTC and see if things
>> improve.
>>
>> The challenge for me as a newby is how do I know if I have everything
>> properly configured?
>>
>> Can I just turn off building and testing of the Hive-related stuff in
>> some supported/well-known way?
>>
>> If so, I would suggest I'd like to turn off not just Hive, but *as much
>> as possible*. I really just need the embedded drill to work.
>>
>> I would agree with @Charles Givre   that a contrib
>> package addition is the ideal approach and that's what I'll be attempting.
>>
>> -mikeb
>>
>> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  wrote:
>>
>>> I'll add some heresy here... IMHO, for the purposes of developing a DFDL
>>> extension, you probably don't need all the Drill tests to run.  For your
>>> project, my suggestion would be to add a module to the contrib package and
>>> that way your changes are relatively self contained.
>>> Best,
>>> -- C
>>>
>>>
>>>
>>> > On Jul 13, 2023, at 10:27 AM, James Turton  wrote:
>>> >
>>> > Hi Mike
>>> >
>>> > Here's the command line I use to run tests on a machine that's not in
>>> the UTC time zone (plus some unrelated memory size arguments).
>>> >
>>> > mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
>>> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
>>> >
>>> > I have one other question to add to Paul's comments - does the OS user
>>> that you're running Maven under have write access to all of the source tree
>>> that you put at /opt/drill?
>>> >
>>> > On 2023/07/11 22:12, Paul Rogers wrote:
>>> >> Hi Mike,
>>> >>
>>> >> A quick glance at the log suggests a failure in the tests for the JSON
>>> >> reader, in the Mongo extended types. Drill's date/time support has
>>> >> historically been fragile. Some tests only work if your machine is
>>> set to
>>> >> use the UTC time zone (or Java is told to pretend that the time is
>>> UTC.)
>>> >> The Mongo types test failure seems to be around a date/time test so
>>> maybe
>>> >> this is the issue?
>>> >>
>>> >> There are also failures indicating that the Drillbit (Drill server)
>>> died.
>>> >> Not sure how this can happen, as tests run Drill embedded (or used
>>> to.)
>>> >> Looking earlier in the logs, it seems that the Drillbit didn't start
>>> due to
>>> >> UDF (user-defined function) failures:
>>> >>
>>> >> Found duplicated function in drill-custom-lower.jar:
>>> >> custom_lower(VARCHAR-REQUIRED)
>>> >> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
>>> >>
>>> >> Not sure how this could occur: it should have failed in all builds.
>>> >>
>>> >> Also:
>>> >>
>>> >> File
>>> >>
>>> 

Re: drill tests not passing

[Mike Beckerle]

Figured out my own issue. This --add-opens is a java 9+ thing. I'll switch
from Java 8 to Java 11.

On Mon, Jul 24, 2023 at 7:07 PM Mike Beckerle  wrote:

> Charles,
>
> When you say this is close to working what is the expected behavior of
> the code currently.
> I could debug into this, but I'm frankly unable to get anything to run.
>
> Currently when I try to run just one test: TestXSDSchema:testSimpleXSD(),
> I get the below giant message, to cut to the chase ends in:
>
> Unrecognized option: --add-opens
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
>
> Process finished with exit code 1
>
>
> ---
>
> /home/mbeckerle/installed-software/jdk1.8.0_361/bin/java -ea
> -Djava.io.tmpdir=/opt/drill/contrib/format-xml/target -Xms512m -Xmx2500m
> -Ddrill.exec.http.enabled=false
> -Ddrill.exec.memory.enable_unsafe_bounds_check=true
> -Ddrill.exec.sys.store.provider.local.write=false
> -Dorg.apache.drill.exec.server.Drillbit.system_options=org.apache.drill.exec.compile.ClassTransformer.scalar_replacement=on
> -Ddrill.catastrophic_to_standard_out=true -XX:MaxDirectMemorySize=4500M
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -ea --add-opens
> java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED
> --add-opens java.base/java.nio=ALL-UNNAMED --add-opens
> java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED
> --add-opens java.security.jgss/sun.security.krb5=ALL-UNNAMED
> -Djdk.attach.allowAttachSelf=true
> -javaagent:/home/mbeckerle/.m2/repository/org/jmockit/jmockit/1.47/jmockit-1.47.jar
> -Didea.test.cyclic.buffer.size=1048576
> -javaagent:/home/mbeckerle/installed-software/idea-IU-231.8770.65/lib/idea_rt.jar=46065:/home/mbeckerle/installed-software/idea-IU-231.8770.65/bin
> -Dfile.encoding=UTF-8 -classpath
> 

Re: drill tests not passing

[Charles Givre]

Hmm... Ok.. that's not what's happening on my machine.  On my box, the simple 
unit tests are passing.  The complex ones are still missing some levels of 
nesting.  But mine will build w/o errors.
What Java version are you using?
-- C


> On Jul 24, 2023, at 7:07 PM, Mike Beckerle  wrote:
> 
> Charles,
> 
> When you say this is close to working what is the expected behavior of
> the code currently.
> I could debug into this, but I'm frankly unable to get anything to run.
> 
> Currently when I try to run just one test: TestXSDSchema:testSimpleXSD(), I
> get the below giant message, to cut to the chase ends in:
> 
> Unrecognized option: --add-opens
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.
> 
> Process finished with exit code 1
> 
> ---
> 
> /home/mbeckerle/installed-software/jdk1.8.0_361/bin/java -ea
> -Djava.io.tmpdir=/opt/drill/contrib/format-xml/target -Xms512m -Xmx2500m
> -Ddrill.exec.http.enabled=false
> -Ddrill.exec.memory.enable_unsafe_bounds_check=true
> -Ddrill.exec.sys.store.provider.local.write=false
> -Dorg.apache.drill.exec.server.Drillbit.system_options=org.apache.drill.exec.compile.ClassTransformer.scalar_replacement=on
> -Ddrill.catastrophic_to_standard_out=true -XX:MaxDirectMemorySize=4500M
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -ea --add-opens
> java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED
> --add-opens java.base/java.nio=ALL-UNNAMED --add-opens
> java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED
> --add-opens java.security.jgss/sun.security.krb5=ALL-UNNAMED
> -Djdk.attach.allowAttachSelf=true
> -javaagent:/home/mbeckerle/.m2/repository/org/jmockit/jmockit/1.47/jmockit-1.47.jar
> -Didea.test.cyclic.buffer.size=1048576
> -javaagent:/home/mbeckerle/installed-software/idea-IU-231.8770.65/lib/idea_rt.jar=46065:/home/mbeckerle/installed-software/idea-IU-231.8770.65/bin
> -Dfile.encoding=UTF-8 -classpath
> 

Re: drill tests not passing

[Mike Beckerle]

Charles,

When you say this is close to working what is the expected behavior of
the code currently.
I could debug into this, but I'm frankly unable to get anything to run.

Currently when I try to run just one test: TestXSDSchema:testSimpleXSD(), I
get the below giant message, to cut to the chase ends in:

Unrecognized option: --add-opens
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Process finished with exit code 1

---

/home/mbeckerle/installed-software/jdk1.8.0_361/bin/java -ea
-Djava.io.tmpdir=/opt/drill/contrib/format-xml/target -Xms512m -Xmx2500m
-Ddrill.exec.http.enabled=false
-Ddrill.exec.memory.enable_unsafe_bounds_check=true
-Ddrill.exec.sys.store.provider.local.write=false
-Dorg.apache.drill.exec.server.Drillbit.system_options=org.apache.drill.exec.compile.ClassTransformer.scalar_replacement=on
-Ddrill.catastrophic_to_standard_out=true -XX:MaxDirectMemorySize=4500M
-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -ea --add-opens
java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED
--add-opens java.base/java.nio=ALL-UNNAMED --add-opens
java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED
--add-opens java.security.jgss/sun.security.krb5=ALL-UNNAMED
-Djdk.attach.allowAttachSelf=true
-javaagent:/home/mbeckerle/.m2/repository/org/jmockit/jmockit/1.47/jmockit-1.47.jar
-Didea.test.cyclic.buffer.size=1048576
-javaagent:/home/mbeckerle/installed-software/idea-IU-231.8770.65/lib/idea_rt.jar=46065:/home/mbeckerle/installed-software/idea-IU-231.8770.65/bin
-Dfile.encoding=UTF-8 -classpath

Re: drill tests not passing

[Mike Beckerle]

Hi drill devs,

I'm still stuck on this problem. Can anyone suggest a way past this?

Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense | www.owlcyberdefense.com



On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle  wrote:

> Looks like I attached the wrong file. Doing too many things at once.
>
> The correct file is attached here.
>
>
>
> On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle 
> wrote:
>
>> Update: I did a clean and install -DskipTests=true.
>>
>> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
>>
>> But alas, it still fails, this time the failure unique and is only in
>> "Java Execution Engine"
>>
>> [ERROR] Failed to execute goal
>> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
>> (unpack-vector-types) on project drill-java-exec: Artifact has not been
>> packaged yet. When used on reactor artifact, unpack should be executed
>> after packaging: see MDEP-98. -> [Help 1]
>>
>> The command and complete trace output are below.
>>
>> I need assistance on how to proceed.
>>
>> Complete trace from the mvn test is attached.
>>
>>
>> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle 
>> wrote:
>>
>>> To answer questions:
>>>
>>> 1. Paul: This is a 100% stock build. All I have done is clone the repo
>>> (master branch). Make a new git branch (in case I make future changes). Try
>>> to build (success) and test (failed so far).
>>>
>>> 2. James: The /opt/drill directory I created is owned by my userid and
>>> has full read/write access for all the development activities. I just put
>>> it there so it would have a shorter path to fix the first Hive-related
>>> glitch I encountered with the Linux 255 limit on file pathname length.
>>>
>>> I will try the suggested maven command line for non-UTC and see if
>>> things improve.
>>>
>>> The challenge for me as a newby is how do I know if I have everything
>>> properly configured?
>>>
>>> Can I just turn off building and testing of the Hive-related stuff in
>>> some supported/well-known way?
>>>
>>> If so, I would suggest I'd like to turn off not just Hive, but *as much
>>> as possible*. I really just need the embedded drill to work.
>>>
>>> I would agree with @Charles Givre   that a contrib
>>> package addition is the ideal approach and that's what I'll be attempting.
>>>
>>> -mikeb
>>>
>>> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  wrote:
>>>
 I'll add some heresy here... IMHO, for the purposes of developing a
 DFDL extension, you probably don't need all the Drill tests to run.  For
 your project, my suggestion would be to add a module to the contrib package
 and that way your changes are relatively self contained.
 Best,
 -- C



 > On Jul 13, 2023, at 10:27 AM, James Turton  wrote:
 >
 > Hi Mike
 >
 > Here's the command line I use to run tests on a machine that's not in
 the UTC time zone (plus some unrelated memory size arguments).
 >
 > mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
 -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
 >
 > I have one other question to add to Paul's comments - does the OS
 user that you're running Maven under have write access to all of the source
 tree that you put at /opt/drill?
 >
 > On 2023/07/11 22:12, Paul Rogers wrote:
 >> Hi Mike,
 >>
 >> A quick glance at the log suggests a failure in the tests for the
 JSON
 >> reader, in the Mongo extended types. Drill's date/time support has
 >> historically been fragile. Some tests only work if your machine is
 set to
 >> use the UTC time zone (or Java is told to pretend that the time is
 UTC.)
 >> The Mongo types test failure seems to be around a date/time test so
 maybe
 >> this is the issue?
 >>
 >> There are also failures indicating that the Drillbit (Drill server)
 died.
 >> Not sure how this can happen, as tests run Drill embedded (or used
 to.)
 >> Looking earlier in the logs, it seems that the Drillbit didn't start
 due to
 >> UDF (user-defined function) failures:
 >>
 >> Found duplicated function in drill-custom-lower.jar:
 >> custom_lower(VARCHAR-REQUIRED)
 >> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
 >>
 >> Not sure how this could occur: it should have failed in all builds.
 >>
 >> Also:
 >>
 >> File
 >>
 /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
 >> does not exist on file system file:///
 >>
 >> This is complaining that Drill needs the source code (not just class
 file)
 >> for its built-in functions. Again, this should not fail in a standard
 >> build, because if it did, it would fail in all builds.
 >>
 >> There are other odd errors 

Re: drill tests not passing

[Mike Beckerle]

That looks like a great start at what we will need for DFDL.

I will study what you've done carefully and get back to you with questions.


On Fri, Jul 14, 2023 at 5:59 PM Charles Givre  wrote:

> Hi Mike,
> One more thing... I've been working on an XSD Reader for Drill for some
> time.  (This is still very buggy)
> https://github.com/cgivre/drill/tree/xsd_reader
>
>  What this does is attempt to convert a XML XSD file into a Drill Schema.
> Best,
> -- C
>
>
>
> On Jul 14, 2023, at 2:20 PM, Charles Givre  wrote:
>
> Mike,
> Are you able to build Drill w/o the tests?  If so, my suggestion is really
> just to start working on the DFDL extensions.  I've been doing Drill stuff
> for far too long and really haven't needed to run the full battery of unit
> tests locally.  As long as you can build it and can execute individual unit
> tests, you should be ok.  Others may disagree, but for what you're doing,
> I'd think it would be fine.
> Best,
> -- C
>
>
>
> On Jul 14, 2023, at 2:04 PM, Mike Beckerle  wrote:
>
> Update: I did a clean and install -DskipTests=true.
>
> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
>
> But alas, it still fails, this time the failure unique and is only in
> "Java Execution Engine"
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
> (unpack-vector-types) on project drill-java-exec: Artifact has not been
> packaged yet. When used on reactor artifact, unpack should be executed
> after packaging: see MDEP-98. -> [Help 1]
>
> The command and complete trace output are below.
>
> I need assistance on how to proceed.
>
> Complete trace from the mvn test is attached.
>
>
> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle 
> wrote:
>
>> To answer questions:
>>
>> 1. Paul: This is a 100% stock build. All I have done is clone the repo
>> (master branch). Make a new git branch (in case I make future changes). Try
>> to build (success) and test (failed so far).
>>
>> 2. James: The /opt/drill directory I created is owned by my userid and
>> has full read/write access for all the development activities. I just put
>> it there so it would have a shorter path to fix the first Hive-related
>> glitch I encountered with the Linux 255 limit on file pathname length.
>>
>> I will try the suggested maven command line for non-UTC and see if things
>> improve.
>>
>> The challenge for me as a newby is how do I know if I have everything
>> properly configured?
>>
>> Can I just turn off building and testing of the Hive-related stuff in
>> some supported/well-known way?
>>
>> If so, I would suggest I'd like to turn off not just Hive, but *as much
>> as possible*. I really just need the embedded drill to work.
>>
>> I would agree with @Charles Givre   that a contrib
>> package addition is the ideal approach and that's what I'll be attempting.
>>
>> -mikeb
>>
>> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  wrote:
>>
>>> I'll add some heresy here... IMHO, for the purposes of developing a DFDL
>>> extension, you probably don't need all the Drill tests to run.  For your
>>> project, my suggestion would be to add a module to the contrib package and
>>> that way your changes are relatively self contained.
>>> Best,
>>> -- C
>>>
>>>
>>>
>>> > On Jul 13, 2023, at 10:27 AM, James Turton  wrote:
>>> >
>>> > Hi Mike
>>> >
>>> > Here's the command line I use to run tests on a machine that's not in
>>> the UTC time zone (plus some unrelated memory size arguments).
>>> >
>>> > mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
>>> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
>>> >
>>> > I have one other question to add to Paul's comments - does the OS user
>>> that you're running Maven under have write access to all of the source tree
>>> that you put at /opt/drill?
>>> >
>>> > On 2023/07/11 22:12, Paul Rogers wrote:
>>> >> Hi Mike,
>>> >>
>>> >> A quick glance at the log suggests a failure in the tests for the JSON
>>> >> reader, in the Mongo extended types. Drill's date/time support has
>>> >> historically been fragile. Some tests only work if your machine is
>>> set to
>>> >> use the UTC time zone (or Java is told to pretend that the time is
>>> UTC.)
>>> >> The Mongo types test failure seems to be around a date/time test so
>>> maybe
>>> >> this is the issue?
>>> >>
>>> >> There are also failures indicating that the Drillbit (Drill server)
>>> died.
>>> >> Not sure how this can happen, as tests run Drill embedded (or used
>>> to.)
>>> >> Looking earlier in the logs, it seems that the Drillbit didn't start
>>> due to
>>> >> UDF (user-defined function) failures:
>>> >>
>>> >> Found duplicated function in drill-custom-lower.jar:
>>> >> custom_lower(VARCHAR-REQUIRED)
>>> >> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
>>> >>
>>> >> Not sure how this could occur: it should have failed in all builds.
>>> >>
>>> >> Also:
>>> >>
>>> >> File
>>> >>
>>> 

Re: drill tests not passing

[Charles Givre]

Hi Mike, 
One more thing... I've been working on an XSD Reader for Drill for some time.  
(This is still very buggy)  https://github.com/cgivre/drill/tree/xsd_reader

 What this does is attempt to convert a XML XSD file into a Drill Schema.  
Best,
-- C



> On Jul 14, 2023, at 2:20 PM, Charles Givre  wrote:
> 
> Mike, 
> Are you able to build Drill w/o the tests?  If so, my suggestion is really 
> just to start working on the DFDL extensions.  I've been doing Drill stuff 
> for far too long and really haven't needed to run the full battery of unit 
> tests locally.  As long as you can build it and can execute individual unit 
> tests, you should be ok.  Others may disagree, but for what you're doing, I'd 
> think it would be fine.
> Best,
> -- C
> 
> 
> 
>> On Jul 14, 2023, at 2:04 PM, Mike Beckerle  wrote:
>> 
>> Update: I did a clean and install -DskipTests=true.
>> 
>> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
>> 
>> But alas, it still fails, this time the failure unique and is only in "Java 
>> Execution Engine"
>> 
>> [ERROR] Failed to execute goal 
>> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack 
>> (unpack-vector-types) on project drill-java-exec: Artifact has not been 
>> packaged yet. When used on reactor artifact, unpack should be executed after 
>> packaging: see MDEP-98. -> [Help 1]
>> 
>> The command and complete trace output are below. 
>> 
>> I need assistance on how to proceed. 
>> 
>> Complete trace from the mvn test is attached. 
>>  
>> 
>> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle > > wrote:
>>> To answer questions:
>>> 
>>> 1. Paul: This is a 100% stock build. All I have done is clone the repo 
>>> (master branch). Make a new git branch (in case I make future changes). Try 
>>> to build (success) and test (failed so far). 
>>> 
>>> 2. James: The /opt/drill directory I created is owned by my userid and has 
>>> full read/write access for all the development activities. I just put it 
>>> there so it would have a shorter path to fix the first Hive-related glitch 
>>> I encountered with the Linux 255 limit on file pathname length. 
>>> 
>>> I will try the suggested maven command line for non-UTC and see if things 
>>> improve. 
>>> 
>>> The challenge for me as a newby is how do I know if I have everything 
>>> properly configured? 
>>> 
>>> Can I just turn off building and testing of the Hive-related stuff in some 
>>> supported/well-known way? 
>>> 
>>> If so, I would suggest I'd like to turn off not just Hive, but as much as 
>>> possible. I really just need the embedded drill to work. 
>>> 
>>> I would agree with @Charles Givre   that a contrib 
>>> package addition is the ideal approach and that's what I'll be attempting. 
>>> 
>>> -mikeb
>>> 
>>> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre >> > wrote:
 I'll add some heresy here... IMHO, for the purposes of developing a DFDL 
 extension, you probably don't need all the Drill tests to run.  For your 
 project, my suggestion would be to add a module to the contrib package and 
 that way your changes are relatively self contained.  
 Best,
 -- C
 
 
 
 > On Jul 13, 2023, at 10:27 AM, James Turton >>> > > wrote:
 > 
 > Hi Mike
 > 
 > Here's the command line I use to run tests on a machine that's not in 
 > the UTC time zone (plus some unrelated memory size arguments).
 > 
 > mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en 
 > -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
 > 
 > I have one other question to add to Paul's comments - does the OS user 
 > that you're running Maven under have write access to all of the source 
 > tree that you put at /opt/drill?
 > 
 > On 2023/07/11 22:12, Paul Rogers wrote:
 >> Hi Mike,
 >> 
 >> A quick glance at the log suggests a failure in the tests for the JSON
 >> reader, in the Mongo extended types. Drill's date/time support has
 >> historically been fragile. Some tests only work if your machine is set 
 >> to
 >> use the UTC time zone (or Java is told to pretend that the time is UTC.)
 >> The Mongo types test failure seems to be around a date/time test so 
 >> maybe
 >> this is the issue?
 >> 
 >> There are also failures indicating that the Drillbit (Drill server) 
 >> died.
 >> Not sure how this can happen, as tests run Drill embedded (or used to.)
 >> Looking earlier in the logs, it seems that the Drillbit didn't start 
 >> due to
 >> UDF (user-defined function) failures:
 >> 
 >> Found duplicated function in drill-custom-lower.jar:
 >> custom_lower(VARCHAR-REQUIRED)
 >> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
 >> 
 >> Not sure how this could occur: it should have failed in all builds.
 >> 
 >> Also:

Re: drill tests not passing

[Charles Givre]

Mike, 
Are you able to build Drill w/o the tests?  If so, my suggestion is really just 
to start working on the DFDL extensions.  I've been doing Drill stuff for far 
too long and really haven't needed to run the full battery of unit tests 
locally.  As long as you can build it and can execute individual unit tests, 
you should be ok.  Others may disagree, but for what you're doing, I'd think it 
would be fine.
Best,
-- C



> On Jul 14, 2023, at 2:04 PM, Mike Beckerle  wrote:
> 
> Update: I did a clean and install -DskipTests=true.
> 
> Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
> 
> But alas, it still fails, this time the failure unique and is only in "Java 
> Execution Engine"
> 
> [ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack 
> (unpack-vector-types) on project drill-java-exec: Artifact has not been 
> packaged yet. When used on reactor artifact, unpack should be executed after 
> packaging: see MDEP-98. -> [Help 1]
> 
> The command and complete trace output are below. 
> 
> I need assistance on how to proceed. 
> 
> Complete trace from the mvn test is attached. 
>  
> 
> On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle  > wrote:
>> To answer questions:
>> 
>> 1. Paul: This is a 100% stock build. All I have done is clone the repo 
>> (master branch). Make a new git branch (in case I make future changes). Try 
>> to build (success) and test (failed so far). 
>> 
>> 2. James: The /opt/drill directory I created is owned by my userid and has 
>> full read/write access for all the development activities. I just put it 
>> there so it would have a shorter path to fix the first Hive-related glitch I 
>> encountered with the Linux 255 limit on file pathname length. 
>> 
>> I will try the suggested maven command line for non-UTC and see if things 
>> improve. 
>> 
>> The challenge for me as a newby is how do I know if I have everything 
>> properly configured? 
>> 
>> Can I just turn off building and testing of the Hive-related stuff in some 
>> supported/well-known way? 
>> 
>> If so, I would suggest I'd like to turn off not just Hive, but as much as 
>> possible. I really just need the embedded drill to work. 
>> 
>> I would agree with @Charles Givre   that a contrib 
>> package addition is the ideal approach and that's what I'll be attempting. 
>> 
>> -mikeb
>> 
>> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre > > wrote:
>>> I'll add some heresy here... IMHO, for the purposes of developing a DFDL 
>>> extension, you probably don't need all the Drill tests to run.  For your 
>>> project, my suggestion would be to add a module to the contrib package and 
>>> that way your changes are relatively self contained.  
>>> Best,
>>> -- C
>>> 
>>> 
>>> 
>>> > On Jul 13, 2023, at 10:27 AM, James Turton >> > > wrote:
>>> > 
>>> > Hi Mike
>>> > 
>>> > Here's the command line I use to run tests on a machine that's not in the 
>>> > UTC time zone (plus some unrelated memory size arguments).
>>> > 
>>> > mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en 
>>> > -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
>>> > 
>>> > I have one other question to add to Paul's comments - does the OS user 
>>> > that you're running Maven under have write access to all of the source 
>>> > tree that you put at /opt/drill?
>>> > 
>>> > On 2023/07/11 22:12, Paul Rogers wrote:
>>> >> Hi Mike,
>>> >> 
>>> >> A quick glance at the log suggests a failure in the tests for the JSON
>>> >> reader, in the Mongo extended types. Drill's date/time support has
>>> >> historically been fragile. Some tests only work if your machine is set to
>>> >> use the UTC time zone (or Java is told to pretend that the time is UTC.)
>>> >> The Mongo types test failure seems to be around a date/time test so maybe
>>> >> this is the issue?
>>> >> 
>>> >> There are also failures indicating that the Drillbit (Drill server) died.
>>> >> Not sure how this can happen, as tests run Drill embedded (or used to.)
>>> >> Looking earlier in the logs, it seems that the Drillbit didn't start due 
>>> >> to
>>> >> UDF (user-defined function) failures:
>>> >> 
>>> >> Found duplicated function in drill-custom-lower.jar:
>>> >> custom_lower(VARCHAR-REQUIRED)
>>> >> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
>>> >> 
>>> >> Not sure how this could occur: it should have failed in all builds.
>>> >> 
>>> >> Also:
>>> >> 
>>> >> File
>>> >> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
>>> >> does not exist on file system file:///
>>> >> 
>>> >> This is complaining that Drill needs the source code (not just class 
>>> >> file)
>>> >> for its built-in functions. Again, this should not fail in a standard
>>> >> build, because if it did, it would fail in all builds.
>>> >> 

Re: drill tests not passing

[Mike Beckerle]

Update: I did a clean and install -DskipTests=true.

Then I tried the mvn test using the non-UTC timezone stuff, as suggested.

But alas, it still fails, this time the failure unique and is only in "Java
Execution Engine"

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
(unpack-vector-types) on project drill-java-exec: Artifact has not been
packaged yet. When used on reactor artifact, unpack should be executed
after packaging: see MDEP-98. -> [Help 1]

The command and complete trace output are below.

I need assistance on how to proceed.

Complete trace from the mvn test is attached.


On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle  wrote:

> To answer questions:
>
> 1. Paul: This is a 100% stock build. All I have done is clone the repo
> (master branch). Make a new git branch (in case I make future changes). Try
> to build (success) and test (failed so far).
>
> 2. James: The /opt/drill directory I created is owned by my userid and has
> full read/write access for all the development activities. I just put it
> there so it would have a shorter path to fix the first Hive-related glitch
> I encountered with the Linux 255 limit on file pathname length.
>
> I will try the suggested maven command line for non-UTC and see if things
> improve.
>
> The challenge for me as a newby is how do I know if I have everything
> properly configured?
>
> Can I just turn off building and testing of the Hive-related stuff in some
> supported/well-known way?
>
> If so, I would suggest I'd like to turn off not just Hive, but *as much
> as possible*. I really just need the embedded drill to work.
>
> I would agree with @Charles Givre   that a contrib
> package addition is the ideal approach and that's what I'll be attempting.
>
> -mikeb
>
> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  wrote:
>
>> I'll add some heresy here... IMHO, for the purposes of developing a DFDL
>> extension, you probably don't need all the Drill tests to run.  For your
>> project, my suggestion would be to add a module to the contrib package and
>> that way your changes are relatively self contained.
>> Best,
>> -- C
>>
>>
>>
>> > On Jul 13, 2023, at 10:27 AM, James Turton  wrote:
>> >
>> > Hi Mike
>> >
>> > Here's the command line I use to run tests on a machine that's not in
>> the UTC time zone (plus some unrelated memory size arguments).
>> >
>> > mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
>> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
>> >
>> > I have one other question to add to Paul's comments - does the OS user
>> that you're running Maven under have write access to all of the source tree
>> that you put at /opt/drill?
>> >
>> > On 2023/07/11 22:12, Paul Rogers wrote:
>> >> Hi Mike,
>> >>
>> >> A quick glance at the log suggests a failure in the tests for the JSON
>> >> reader, in the Mongo extended types. Drill's date/time support has
>> >> historically been fragile. Some tests only work if your machine is set
>> to
>> >> use the UTC time zone (or Java is told to pretend that the time is
>> UTC.)
>> >> The Mongo types test failure seems to be around a date/time test so
>> maybe
>> >> this is the issue?
>> >>
>> >> There are also failures indicating that the Drillbit (Drill server)
>> died.
>> >> Not sure how this can happen, as tests run Drill embedded (or used to.)
>> >> Looking earlier in the logs, it seems that the Drillbit didn't start
>> due to
>> >> UDF (user-defined function) failures:
>> >>
>> >> Found duplicated function in drill-custom-lower.jar:
>> >> custom_lower(VARCHAR-REQUIRED)
>> >> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
>> >>
>> >> Not sure how this could occur: it should have failed in all builds.
>> >>
>> >> Also:
>> >>
>> >> File
>> >>
>> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
>> >> does not exist on file system file:///
>> >>
>> >> This is complaining that Drill needs the source code (not just class
>> file)
>> >> for its built-in functions. Again, this should not fail in a standard
>> >> build, because if it did, it would fail in all builds.
>> >>
>> >> There are other odd errors as well.
>> >>
>> >> Perhaps we should ask: is this a "stock" build? Check out Drill and run
>> >> tests? Or, have you already started making changes for your project?
>> >>
>> >> - Paul
>> >>
>> >>
>> >> On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle 
>> wrote:
>> >>
>> >>> I have drill building and running its tests. Some tests fail: [ERROR]
>> >>> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
>> >>>
>> >>> I am wondering if there is perhaps some setup step that I missed in
>> the
>> >>> instructions.
>> >>>
>> >>> I have attached the output from the 'mvn clean install
>> -DskipTests=false'
>> >>> execution. (zipped)
>> >>> I am running on Ubuntu 20.04, definitely have Java 8 setup.
>> >>>
>> >>> I'm hoping someone can skim it and 

Re: drill tests not passing

[Charles Givre]

Hi Mike,
You can just build Drill with the -DskipTests = true option and that should
work.  I do my development in intellij, and just run the relevant unit
tests there.  Then to test Drill in its entirety, I'll use the CI in github
which works most of the time. ;-)


-- C

On Thu, Jul 13, 2023 at 1:14 PM Mike Beckerle  wrote:

> To answer questions:
>
> 1. Paul: This is a 100% stock build. All I have done is clone the repo
> (master branch). Make a new git branch (in case I make future changes). Try
> to build (success) and test (failed so far).
>
> 2. James: The /opt/drill directory I created is owned by my userid and has
> full read/write access for all the development activities. I just put it
> there so it would have a shorter path to fix the first Hive-related glitch
> I encountered with the Linux 255 limit on file pathname length.
>
> I will try the suggested maven command line for non-UTC and see if things
> improve.
>
> The challenge for me as a newby is how do I know if I have everything
> properly configured?
>
> Can I just turn off building and testing of the Hive-related stuff in some
> supported/well-known way?
>
> If so, I would suggest I'd like to turn off not just Hive, but *as much as
> possible*. I really just need the embedded drill to work.
>
> I would agree with @Charles Givre   that a contrib
> package addition is the ideal approach and that's what I'll be attempting.
>
> -mikeb
>
> On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  wrote:
>
> > I'll add some heresy here... IMHO, for the purposes of developing a DFDL
> > extension, you probably don't need all the Drill tests to run.  For your
> > project, my suggestion would be to add a module to the contrib package
> and
> > that way your changes are relatively self contained.
> > Best,
> > -- C
> >
> >
> >
> > > On Jul 13, 2023, at 10:27 AM, James Turton  wrote:
> > >
> > > Hi Mike
> > >
> > > Here's the command line I use to run tests on a machine that's not in
> > the UTC time zone (plus some unrelated memory size arguments).
> > >
> > > mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
> > -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
> > >
> > > I have one other question to add to Paul's comments - does the OS user
> > that you're running Maven under have write access to all of the source
> tree
> > that you put at /opt/drill?
> > >
> > > On 2023/07/11 22:12, Paul Rogers wrote:
> > >> Hi Mike,
> > >>
> > >> A quick glance at the log suggests a failure in the tests for the JSON
> > >> reader, in the Mongo extended types. Drill's date/time support has
> > >> historically been fragile. Some tests only work if your machine is set
> > to
> > >> use the UTC time zone (or Java is told to pretend that the time is
> UTC.)
> > >> The Mongo types test failure seems to be around a date/time test so
> > maybe
> > >> this is the issue?
> > >>
> > >> There are also failures indicating that the Drillbit (Drill server)
> > died.
> > >> Not sure how this can happen, as tests run Drill embedded (or used
> to.)
> > >> Looking earlier in the logs, it seems that the Drillbit didn't start
> > due to
> > >> UDF (user-defined function) failures:
> > >>
> > >> Found duplicated function in drill-custom-lower.jar:
> > >> custom_lower(VARCHAR-REQUIRED)
> > >> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
> > >>
> > >> Not sure how this could occur: it should have failed in all builds.
> > >>
> > >> Also:
> > >>
> > >> File
> > >>
> >
> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
> > >> does not exist on file system file:///
> > >>
> > >> This is complaining that Drill needs the source code (not just class
> > file)
> > >> for its built-in functions. Again, this should not fail in a standard
> > >> build, because if it did, it would fail in all builds.
> > >>
> > >> There are other odd errors as well.
> > >>
> > >> Perhaps we should ask: is this a "stock" build? Check out Drill and
> run
> > >> tests? Or, have you already started making changes for your project?
> > >>
> > >> - Paul
> > >>
> > >>
> > >> On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle 
> > wrote:
> > >>
> > >>> I have drill building and running its tests. Some tests fail: [ERROR]
> > >>> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
> > >>>
> > >>> I am wondering if there is perhaps some setup step that I missed in
> the
> > >>> instructions.
> > >>>
> > >>> I have attached the output from the 'mvn clean install
> > -DskipTests=false'
> > >>> execution. (zipped)
> > >>> I am running on Ubuntu 20.04, definitely have Java 8 setup.
> > >>>
> > >>> I'm hoping someone can skim it and spot the issue(s).
> > >>>
> > >>> Thanks for any help
> > >>>
> > >>> Mike Beckerle
> > >>> Apache Daffodil PMC | daffodil.apache.org
> > >>> OGF DFDL Workgroup Co-Chair |
> > www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> > >>> Owl Cyber Defense | www.owlcyberdefense.com

Re: drill tests not passing

[Mike Beckerle]

To answer questions:

1. Paul: This is a 100% stock build. All I have done is clone the repo
(master branch). Make a new git branch (in case I make future changes). Try
to build (success) and test (failed so far).

2. James: The /opt/drill directory I created is owned by my userid and has
full read/write access for all the development activities. I just put it
there so it would have a shorter path to fix the first Hive-related glitch
I encountered with the Linux 255 limit on file pathname length.

I will try the suggested maven command line for non-UTC and see if things
improve.

The challenge for me as a newby is how do I know if I have everything
properly configured?

Can I just turn off building and testing of the Hive-related stuff in some
supported/well-known way?

If so, I would suggest I'd like to turn off not just Hive, but *as much as
possible*. I really just need the embedded drill to work.

I would agree with @Charles Givre   that a contrib
package addition is the ideal approach and that's what I'll be attempting.

-mikeb

On Thu, Jul 13, 2023 at 10:59 AM Charles Givre  wrote:

> I'll add some heresy here... IMHO, for the purposes of developing a DFDL
> extension, you probably don't need all the Drill tests to run.  For your
> project, my suggestion would be to add a module to the contrib package and
> that way your changes are relatively self contained.
> Best,
> -- C
>
>
>
> > On Jul 13, 2023, at 10:27 AM, James Turton  wrote:
> >
> > Hi Mike
> >
> > Here's the command line I use to run tests on a machine that's not in
> the UTC time zone (plus some unrelated memory size arguments).
> >
> > mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
> >
> > I have one other question to add to Paul's comments - does the OS user
> that you're running Maven under have write access to all of the source tree
> that you put at /opt/drill?
> >
> > On 2023/07/11 22:12, Paul Rogers wrote:
> >> Hi Mike,
> >>
> >> A quick glance at the log suggests a failure in the tests for the JSON
> >> reader, in the Mongo extended types. Drill's date/time support has
> >> historically been fragile. Some tests only work if your machine is set
> to
> >> use the UTC time zone (or Java is told to pretend that the time is UTC.)
> >> The Mongo types test failure seems to be around a date/time test so
> maybe
> >> this is the issue?
> >>
> >> There are also failures indicating that the Drillbit (Drill server)
> died.
> >> Not sure how this can happen, as tests run Drill embedded (or used to.)
> >> Looking earlier in the logs, it seems that the Drillbit didn't start
> due to
> >> UDF (user-defined function) failures:
> >>
> >> Found duplicated function in drill-custom-lower.jar:
> >> custom_lower(VARCHAR-REQUIRED)
> >> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
> >>
> >> Not sure how this could occur: it should have failed in all builds.
> >>
> >> Also:
> >>
> >> File
> >>
> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
> >> does not exist on file system file:///
> >>
> >> This is complaining that Drill needs the source code (not just class
> file)
> >> for its built-in functions. Again, this should not fail in a standard
> >> build, because if it did, it would fail in all builds.
> >>
> >> There are other odd errors as well.
> >>
> >> Perhaps we should ask: is this a "stock" build? Check out Drill and run
> >> tests? Or, have you already started making changes for your project?
> >>
> >> - Paul
> >>
> >>
> >> On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle 
> wrote:
> >>
> >>> I have drill building and running its tests. Some tests fail: [ERROR]
> >>> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
> >>>
> >>> I am wondering if there is perhaps some setup step that I missed in the
> >>> instructions.
> >>>
> >>> I have attached the output from the 'mvn clean install
> -DskipTests=false'
> >>> execution. (zipped)
> >>> I am running on Ubuntu 20.04, definitely have Java 8 setup.
> >>>
> >>> I'm hoping someone can skim it and spot the issue(s).
> >>>
> >>> Thanks for any help
> >>>
> >>> Mike Beckerle
> >>> Apache Daffodil PMC | daffodil.apache.org
> >>> OGF DFDL Workgroup Co-Chair |
> www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> >>> Owl Cyber Defense | www.owlcyberdefense.com
> >>>
> >>>
> >>>
> >
>
>

Re: drill tests not passing

[Charles Givre]

I'll add some heresy here... IMHO, for the purposes of developing a DFDL 
extension, you probably don't need all the Drill tests to run.  For your 
project, my suggestion would be to add a module to the contrib package and that 
way your changes are relatively self contained.  
Best,
-- C



> On Jul 13, 2023, at 10:27 AM, James Turton  wrote:
> 
> Hi Mike
> 
> Here's the command line I use to run tests on a machine that's not in the UTC 
> time zone (plus some unrelated memory size arguments).
> 
> mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en 
> -Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
> 
> I have one other question to add to Paul's comments - does the OS user that 
> you're running Maven under have write access to all of the source tree that 
> you put at /opt/drill?
> 
> On 2023/07/11 22:12, Paul Rogers wrote:
>> Hi Mike,
>> 
>> A quick glance at the log suggests a failure in the tests for the JSON
>> reader, in the Mongo extended types. Drill's date/time support has
>> historically been fragile. Some tests only work if your machine is set to
>> use the UTC time zone (or Java is told to pretend that the time is UTC.)
>> The Mongo types test failure seems to be around a date/time test so maybe
>> this is the issue?
>> 
>> There are also failures indicating that the Drillbit (Drill server) died.
>> Not sure how this can happen, as tests run Drill embedded (or used to.)
>> Looking earlier in the logs, it seems that the Drillbit didn't start due to
>> UDF (user-defined function) failures:
>> 
>> Found duplicated function in drill-custom-lower.jar:
>> custom_lower(VARCHAR-REQUIRED)
>> Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
>> 
>> Not sure how this could occur: it should have failed in all builds.
>> 
>> Also:
>> 
>> File
>> /opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
>> does not exist on file system file:///
>> 
>> This is complaining that Drill needs the source code (not just class file)
>> for its built-in functions. Again, this should not fail in a standard
>> build, because if it did, it would fail in all builds.
>> 
>> There are other odd errors as well.
>> 
>> Perhaps we should ask: is this a "stock" build? Check out Drill and run
>> tests? Or, have you already started making changes for your project?
>> 
>> - Paul
>> 
>> 
>> On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle  wrote:
>> 
>>> I have drill building and running its tests. Some tests fail: [ERROR]
>>> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
>>> 
>>> I am wondering if there is perhaps some setup step that I missed in the
>>> instructions.
>>> 
>>> I have attached the output from the 'mvn clean install -DskipTests=false'
>>> execution. (zipped)
>>> I am running on Ubuntu 20.04, definitely have Java 8 setup.
>>> 
>>> I'm hoping someone can skim it and spot the issue(s).
>>> 
>>> Thanks for any help
>>> 
>>> Mike Beckerle
>>> Apache Daffodil PMC | daffodil.apache.org
>>> OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
>>> Owl Cyber Defense | www.owlcyberdefense.com
>>> 
>>> 
>>> 
> 

Re: drill tests not passing

[James Turton]

Hi Mike

Here's the command line I use to run tests on a machine that's not in 
the UTC time zone (plus some unrelated memory size arguments).


mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en 
-Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560


I have one other question to add to Paul's comments - does the OS user 
that you're running Maven under have write access to all of the source 
tree that you put at /opt/drill?


On 2023/07/11 22:12, Paul Rogers wrote:

Hi Mike,

A quick glance at the log suggests a failure in the tests for the JSON
reader, in the Mongo extended types. Drill's date/time support has
historically been fragile. Some tests only work if your machine is set to
use the UTC time zone (or Java is told to pretend that the time is UTC.)
The Mongo types test failure seems to be around a date/time test so maybe
this is the issue?

There are also failures indicating that the Drillbit (Drill server) died.
Not sure how this can happen, as tests run Drill embedded (or used to.)
Looking earlier in the logs, it seems that the Drillbit didn't start due to
UDF (user-defined function) failures:

Found duplicated function in drill-custom-lower.jar:
custom_lower(VARCHAR-REQUIRED)
Found duplicated function in built-in: lower(VARCHAR-REQUIRED)

Not sure how this could occur: it should have failed in all builds.

Also:

File
/opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
does not exist on file system file:///

This is complaining that Drill needs the source code (not just class file)
for its built-in functions. Again, this should not fail in a standard
build, because if it did, it would fail in all builds.

There are other odd errors as well.

Perhaps we should ask: is this a "stock" build? Check out Drill and run
tests? Or, have you already started making changes for your project?

- Paul


On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle  wrote:


I have drill building and running its tests. Some tests fail: [ERROR]
Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133

I am wondering if there is perhaps some setup step that I missed in the
instructions.

I have attached the output from the 'mvn clean install -DskipTests=false'
execution. (zipped)
I am running on Ubuntu 20.04, definitely have Java 8 setup.

I'm hoping someone can skim it and spot the issue(s).

Thanks for any help

Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense | www.owlcyberdefense.com

Re: drill tests not passing

[Paul Rogers]

Hi Mike,

A quick glance at the log suggests a failure in the tests for the JSON
reader, in the Mongo extended types. Drill's date/time support has
historically been fragile. Some tests only work if your machine is set to
use the UTC time zone (or Java is told to pretend that the time is UTC.)
The Mongo types test failure seems to be around a date/time test so maybe
this is the issue?

There are also failures indicating that the Drillbit (Drill server) died.
Not sure how this can happen, as tests run Drill embedded (or used to.)
Looking earlier in the logs, it seems that the Drillbit didn't start due to
UDF (user-defined function) failures:

Found duplicated function in drill-custom-lower.jar:
custom_lower(VARCHAR-REQUIRED)
Found duplicated function in built-in: lower(VARCHAR-REQUIRED)

Not sure how this could occur: it should have failed in all builds.

Also:

File
/opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
does not exist on file system file:///

This is complaining that Drill needs the source code (not just class file)
for its built-in functions. Again, this should not fail in a standard
build, because if it did, it would fail in all builds.

There are other odd errors as well.

Perhaps we should ask: is this a "stock" build? Check out Drill and run
tests? Or, have you already started making changes for your project?

- Paul


On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle  wrote:

>
> I have drill building and running its tests. Some tests fail: [ERROR]
> Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
>
> I am wondering if there is perhaps some setup step that I missed in the
> instructions.
>
> I have attached the output from the 'mvn clean install -DskipTests=false'
> execution. (zipped)
> I am running on Ubuntu 20.04, definitely have Java 8 setup.
>
> I'm hoping someone can skim it and spot the issue(s).
>
> Thanks for any help
>
> Mike Beckerle
> Apache Daffodil PMC | daffodil.apache.org
> OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
> Owl Cyber Defense | www.owlcyberdefense.com
>
>
>