Hi Mike
I took a look at the build log that you shared with me recently and what
happened in that run was OOM (first report on line 9348).
| 9355 2198813 ERROR [UserServer-1]
[org.apache.drill.exec.rpc.RpcExceptionHandler] - Exception in RPC
communication. 9355 Connection: /127.0.0.1:31046 <-->
/127.0.0.1:39558 (user server). Closing connection.
9356 java.lang.OutOfMemoryError: GC overhead limit exceeded
|
My command to run Drill tests set memory limits as follows.
|alias mvn-test-drill="mvn test \
-Djunit.args=\"-Duser.timezone=UTC -Duser.language=en
-Duser.region=US\" \
-DmemoryMb=2560 -DdirectMemoryMb=2560 \
-DforkCount=2"
|
Hope this helps...
James
On 2023/07/25 00:00, Mike Beckerle wrote:
Hi drill devs,
I'm still stuck on this problem. Can anyone suggest a way past this?
Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair |www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense |www.owlcyberdefense.com
On Mon, Jul 17, 2023 at 9:53 AM Mike Beckerle<mbecke...@apache.org> wrote:
Looks like I attached the wrong file. Doing too many things at once.
The correct file is attached here.
On Fri, Jul 14, 2023 at 2:04 PM Mike Beckerle<mbecke...@apache.org>
wrote:
Update: I did a clean and install -DskipTests=true.
Then I tried the mvn test using the non-UTC timezone stuff, as suggested.
But alas, it still fails, this time the failure unique and is only in
"Java Execution Engine"
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-dependency-plugin:3.4.0:unpack
(unpack-vector-types) on project drill-java-exec: Artifact has not been
packaged yet. When used on reactor artifact, unpack should be executed
after packaging: see MDEP-98. -> [Help 1]
The command and complete trace output are below.
I need assistance on how to proceed.
Complete trace from the mvn test is attached.
On Thu, Jul 13, 2023 at 1:13 PM Mike Beckerle<mbecke...@apache.org>
wrote:
To answer questions:
1. Paul: This is a 100% stock build. All I have done is clone the repo
(master branch). Make a new git branch (in case I make future changes). Try
to build (success) and test (failed so far).
2. James: The /opt/drill directory I created is owned by my userid and
has full read/write access for all the development activities. I just put
it there so it would have a shorter path to fix the first Hive-related
glitch I encountered with the Linux 255 limit on file pathname length.
I will try the suggested maven command line for non-UTC and see if
things improve.
The challenge for me as a newby is how do I know if I have everything
properly configured?
Can I just turn off building and testing of the Hive-related stuff in
some supported/well-known way?
If so, I would suggest I'd like to turn off not just Hive, but *as much
as possible*. I really just need the embedded drill to work.
I would agree with @Charles Givre<cgi...@gmail.com> that a contrib
package addition is the ideal approach and that's what I'll be attempting.
-mikeb
On Thu, Jul 13, 2023 at 10:59 AM Charles Givre<cgi...@gmail.com> wrote:
I'll add some heresy here... IMHO, for the purposes of developing a
DFDL extension, you probably don't need all the Drill tests to run. For
your project, my suggestion would be to add a module to the contrib package
and that way your changes are relatively self contained.
Best,
-- C
On Jul 13, 2023, at 10:27 AM, James Turton<dz...@apache.org> wrote:
Hi Mike
Here's the command line I use to run tests on a machine that's not in
the UTC time zone (plus some unrelated memory size arguments).
mvn test -Djunit.args="-Duser.timezone=UTC -Duser.language=en
-Duser.region=US" -DmemoryMb=2560 -DdirectMemoryMb=2560
I have one other question to add to Paul's comments - does the OS
user that you're running Maven under have write access to all of the source
tree that you put at /opt/drill?
On 2023/07/11 22:12, Paul Rogers wrote:
Hi Mike,
A quick glance at the log suggests a failure in the tests for the
JSON
reader, in the Mongo extended types. Drill's date/time support has
historically been fragile. Some tests only work if your machine is
set to
use the UTC time zone (or Java is told to pretend that the time is
UTC.)
The Mongo types test failure seems to be around a date/time test so
maybe
this is the issue?
There are also failures indicating that the Drillbit (Drill server)
died.
Not sure how this can happen, as tests run Drill embedded (or used
to.)
Looking earlier in the logs, it seems that the Drillbit didn't start
due to
UDF (user-defined function) failures:
Found duplicated function in drill-custom-lower.jar:
custom_lower(VARCHAR-REQUIRED)
Found duplicated function in built-in: lower(VARCHAR-REQUIRED)
Not sure how this could occur: it should have failed in all builds.
Also:
File
/opt/drill/exec/java-exec/target/org.apache.drill.exec.udf.dynamic.TestDynamicUDFSupport/home/drill/happy/udf/staging/drill-custom-lower-sources.jar
does not exist on file systemfile:///
This is complaining that Drill needs the source code (not just class
file)
for its built-in functions. Again, this should not fail in a standard
build, because if it did, it would fail in all builds.
There are other odd errors as well.
Perhaps we should ask: is this a "stock" build? Check out Drill and
run
tests? Or, have you already started making changes for your project?
- Paul
On Tue, Jul 11, 2023 at 9:07 AM Mike Beckerle<mbecke...@apache.org>
wrote:
I have drill building and running its tests. Some tests fail:
[ERROR]
Tests run: 4366, Failures: 2, Errors: 1, Skipped: 133
I am wondering if there is perhaps some setup step that I missed in
the
instructions.
I have attached the output from the 'mvn clean install
-DskipTests=false'
execution. (zipped)
I am running on Ubuntu 20.04, definitely have Java 8 setup.
I'm hoping someone can skim it and spot the issue(s).
Thanks for any help
Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair |
www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense |www.owlcyberdefense.com
