[
https://issues.apache.org/jira/browse/IMPALA-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104934#comment-18104934
]
ASF subversion and git services commented on IMPALA-9153:
---------------------------------------------------------
Commit 57af1e63f5fd7f735f732a68b66673a4ab77a4c1 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=57af1e63f ]
IMPALA-9153: Generate CUP parser Java source via CMake
This moves the code generation for the CUP parser from a
Maven plugin to CMake. The Maven plugin is old and always
regenerates the sources even if they haven't changed. This
means that Maven rebuilds many source files on each run,
slowing down the iteration cycle when running frontend
tests.
Fixing the Maven CUP plugin is possible, but it involves
maintaining our own custom Maven plugin for our version
of CUP. The CMake version is less code and almost as good.
Since Maven doesn't generate the files, they never change
and Maven won't rebuild source files on each run. However,
it means that if a developer is actually changing sql-parser.cup,
then Maven won't notice itself. Instead, the developer
needs to run "make java" to regenerate the CUP sources and
rebuild the Java code. This properly specifies the inputs /
outputs, so it will only be regenerated if necessary.
As part of this, it moves the sql-parser.cup from fe/src/main/cup
to common/cup. This is not strictly necessary, but it matches
how Thrift and other code generation is structured.
Testing:
- Ran local build / local tests
- Compared generated source for both methods and only the
timestamps are different
Change-Id: I180b191b72b54e3661726e1560be844341ed92e5
Reviewed-on: http://gerrit.cloudera.org:8080/24598
Reviewed-by: Michael Smith <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Avoid unneccessary cup compilation
> ----------------------------------
>
> Key: IMPALA-9153
> URL: https://issues.apache.org/jira/browse/IMPALA-9153
> Project: IMPALA
> Issue Type: Test
> Components: Infrastructure
> Reporter: Quanlong Huang
> Priority: Major
>
> Everytime when I run a single FE test, maven always recompile the whole
> module since it detects changes. The change is made by cup-maven-plugin. It
> recompiles sql-parser.cup to SqlParser.java and SqlParserSymbols.java
> everytime even there're no changes in sql-parser.cup. The recompilation cost
> some time and slow down the reproducing of some flasky test that requires
> running it many times.
> An output example:
> {code:java}
> $ (pushd fe && mvn test -fae -Dtest=AuthorizationStmtTest#test*Enabled[1])
> ~/workspace/Impala/fe ~/workspace/Impala
> [INFO] Scanning for projects...
> [INFO]
> [INFO] -----------------< org.apache.impala:impala-frontend
> >------------------
> [INFO] Building Apache Impala Query Engine Frontend 0.1-SNAPSHOT
> [INFO] --------------------------------[ jar
> ]---------------------------------
> [WARNING] Could not transfer metadata
> com.cloudera.cdh:cdh-root:6.x-SNAPSHOT/maven-metadata.xml from/to
> ${distMgmtSnapshotsId} (${distMgmtSnapshotsUrl}): Cannot access
> ${distMgmtSnapshotsUrl} with type default using the available connector
> factories: BasicRepositoryConnectorFactory
> [INFO]
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce
> (enforce-banned-dependencies) @ impala-frontend ---
> [INFO]
> [INFO] --- jacoco-maven-plugin:0.7.6.201602180812:prepare-agent
> (prepare-jacoco-agent) @ impala-frontend ---
> [INFO] Skipping JaCoCo execution because property jacoco.skip is set.
> [INFO] surefireJacocoArg set to empty
> [INFO]
> [INFO] --- cup-maven-plugin:1.6-cdh:generate (cup) @ impala-frontend ---
> [INFO] CUP: Processing 1 cup files <------------ Always regenerate the files
> Warning : Terminal "EMPTY_IDENT" was declared but never used
> Warning : Terminal "UNEXPECTED_CHAR" was declared but never used
> ------- CUP v0.11a czt01 beta Parser Generation Summary -------
> 0 errors and 2 warnings
> 229 terminals, 228 non-terminals, and 875 productions declared,
> producing 1620 unique parse states.
> 2 terminals declared but not used.
> 0 non-terminals declared but not used.
> 0 productions never reduced.
> 0 conflicts detected (0 expected).
> Code written to "SqlParser.java", and "SqlParserSymbols.java".
> ---------------------------------------------------- (v0.11a czt01 beta)
> [WARNING] /home/quanlong/workspace/Impala/fe/src/main/cup/sql-parser.cup
> [0:0]: Terminal "EMPTY_IDENT" was declared but never used
> [WARNING] /home/quanlong/workspace/Impala/fe/src/main/cup/sql-parser.cup
> [0:0]: Terminal "UNEXPECTED_CHAR" was declared but never used
> [INFO] CUP: generated
> /home/quanlong/workspace/Impala/fe/target/generated-sources/cup/org/apache/impala/analysis/SqlParser.java
> [INFO] CUP: generated
> /home/quanlong/workspace/Impala/fe/target/generated-sources/cup/org/apache/impala/analysis/SqlParserSymbols.java
> [INFO] CUP: generated
> /home/quanlong/workspace/Impala/fe/target/generated-sources/cup/org/apache/impala/analysis/action_table.dat
> [INFO]
> [INFO] --- maven-jflex-plugin:1.4.3:generate (jflex) @ impala-frontend ---
> [INFO] SqlScanner.java is up to date.
> [INFO]
> [INFO] --- maven-dependency-plugin:3.1.1:build-classpath (write-classpath) @
> impala-frontend ---
> [INFO] Skipped writing classpath file
> '/home/quanlong/workspace/Impala/fe/target/build-classpath.txt'. No changes
> found.
> [INFO]
> [INFO] --- build-helper-maven-plugin:1.5:add-source (add-source) @
> impala-frontend ---
> [INFO] Source directory:
> /home/quanlong/workspace/Impala/fe/generated-sources/gen-java added.
> [INFO] Source directory:
> /home/quanlong/workspace/Impala/fe/target/generated-sources/cup added.
> [INFO] Source directory:
> /home/quanlong/workspace/Impala/fe/src/compat-hive-2/java added.
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
> impala-frontend ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] skip non existing resourceDirectory
> /home/quanlong/workspace/Impala/fe/src/main/resources
> [INFO] Copying 1 resource
> [INFO]
> [INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @
> impala-frontend ---
> [INFO] Changes detected - recompiling the module! <--------- Decide to
> recomple
> [INFO] Compiling 907 source files to
> /home/quanlong/workspace/Impala/fe/target/classes
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[24,16]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[24,16]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[24,16]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[33,23]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[40,15]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [WARNING]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/util/UnsafeUtil.java:[45,25]
> sun.misc.Unsafe is internal proprietary API and may be removed in a future
> release
> [INFO]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java:
> Some input files use or override a deprecated API.
> [INFO]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/hive/executor/UdfExecutor.java:
> Recompile with -Xlint:deprecation for details.
> [INFO]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/analysis/Expr.java:
> Some input files use unchecked or unsafe operations.
> [INFO]
> /home/quanlong/workspace/Impala/fe/src/main/java/org/apache/impala/analysis/Expr.java:
> Recompile with -Xlint:unchecked for details.
> [INFO]
> [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
> impala-frontend ---
> [INFO] Using 'UTF-8' encoding to copy filtered resources.
> [INFO] Copying 43 resources
> [INFO]
> [INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @
> impala-frontend ---
> [INFO] Nothing to compile - all classes are up to date
> [INFO]
> [INFO] --- maven-surefire-plugin:2.20:test (default-test) @ impala-frontend
> ---
> [INFO]
> [INFO] -------------------------------------------------------
> [INFO] T E S T S
> [INFO] -------------------------------------------------------
> [INFO] Running org.apache.impala.authorization.AuthorizationStmtTest
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 69.847
> s - in org.apache.impala.authorization.AuthorizationStmtTest
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
> [INFO] --- jacoco-maven-plugin:0.7.6.201602180812:report (jacoco-report) @
> impala-frontend ---
> [INFO] Skipping JaCoCo execution because property jacoco.skip is set.
> [INFO]
> ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO]
> ------------------------------------------------------------------------
> [INFO] Total time: 01:23 min
> [INFO] Finished at: 2019-11-13T15:50:33+08:00
> [INFO]
> ------------------------------------------------------------------------
> {code}
> We should configure cup-maven-plugin to avoid unneccessary recompilation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
