I just captured a debug log while hitting this issue, running: `./gradlew -d :beam-sdks-java-extensions-sql:compileJava`
The log is here <https://gist.githubusercontent.com/ryan-williams/1c0178b7a0f33a25c62a5d5c07d690fa/raw/05f74ca200149bd04e7aa6426bf7de9a651b0a6d/gistfile1.txt>. It's 18MB and doesn't really load in Chrome for me; I'd recommend wget'ing it. I haven't had a chance to look at it too deeply yet. On Sun, Feb 17, 2019 at 1:49 PM Ryan Williams <r...@runsascoded.com> wrote: > Good to know, Reuven! > > I just hit it again with `--console=verbose`, and have more info: > https://gist.github.com/ryan-williams/252597d9ecd6973378d5cc2e9dabceb1 > > It seems that the generated files are correctly in place when the failing > compileJava is invoked. > > Perhaps something is wrong (and nondeterministic) with the classpath of > the `:beam-sdks-java-extensions-sql:compileJava` task? > > I'll see if I can run with more verbose/DEBUG logging and get any more > info. > > On Sun, Feb 17, 2019 at 1:42 PM Reuven Lax <re...@google.com> wrote: > >> I also see this almost every other run. >> >> On Sun, Feb 17, 2019 at 8:24 AM Ryan Williams <r...@runsascoded.com> >> wrote: >> >>> I still see this about 20% of the time I run `./gradlew compileTestJava >>> spotlessApply checkstyleMain checkstyleTest` (with 8-way parallelism), >>> which I've taken to running as a kind of "pre-PreCommit" (for Java). >>> >>> In total I'd estimate I've seen this error 20 times (across multiple >>> `./gradlew clean`s and spanning several weeks), but never twice in a row: I >>> always immediately re-run, and the error goes away. >>> >>> That seems consistent with the idea that gradle is racing the codegen >>> task against the compile task that requires the codegen's outputs. >>> >>> However, `./gradlew -p sdks/java/extensions/sql -m compileJava | grep >>> compileJavacc` implies that Gradle knows that compilation depends on the >>> codegen having run. >>> >>> Here are three successive invocations of `./gradlew compileTestJava >>> spotlessApply checkstyleMain checkstyleTest` >>> <https://gist.github.com/ryan-williams/6d2735b2484e9459be656abf379533c5> >>> ; >>> >>> - The first one succeeds (and includes output showing that the >>> codegen task, `:beam-sdks-java-extensions-sql:compileJavacc`, was run). >>> - Then, I rebased on one new upstream commit and ran again, and saw >>> the failure. >>> - Immediately re-running saw the failure go away. >>> >>> I guess in that output, tasks that are run but produce no stdout/stderr >>> are elided from the output, so we can't completely tell when the codegen >>> and compile tasks in question are being run. I'll start running with >>> --console=verbose >>> <https://stackoverflow.com/questions/45883963/gradle-4-0-does-not-display-executed-tasks-in-command-line#comment92914258_45889966>, >>> which I think will give a better sense of how this is happening. >>> >>> >>> On Fri, Feb 8, 2019 at 12:41 AM Ryan Williams <r...@runsascoded.com> >>> wrote: >>> >>>> After your last message, Alex, I saw the issue on a branch with minimal >>>> unrelated changes on top of b4b5495307 >>>> <https://github.com/apache/beam/commit/b4b549530730d9e0283ad0cdb203d384a107dfbf> >>>> (January >>>> 28). >>>> >>>> I ran `./gradlew clean` for the first time in a while at that time, and >>>> worked happily for 11 days, but just saw the issue again on a branch >>>> rebased on top of 381ab55b59 >>>> <https://github.com/apache/beam/commit/381ab55b59> (today): >>>> >>>> > Task :beam-sdks-java-extensions-sql:compileJava FAILED >>>> …/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29: >>>> error: cannot find symbol >>>> import >>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; >>>> ^ >>>> symbol: class BeamSqlParserImpl >>>> location: package org.apache.beam.sdk.extensions.sql.impl.parser.impl >>>> 1 error >>>> >>>> Every time I've seen it, I immediately re-run the compile, and it >>>> succeeds. >>>> >>>> Perhaps something is still wrong with my environment, but otherwise it >>>> would seem that something is still flaky here. >>>> >>>> I'm compiling on an 8-core macOS machine, fwiw, and usually running >>>> `./gradlew compileTestJava` which compiles many projects concurrently. >>>> >>>> On Mon, Jan 28, 2019 at 4:12 PM Alex Amato <ajam...@google.com> wrote: >>>> >>>>> If it continues to occur, maybe it is an environmental issue, be sure >>>>> to try to clean as well. >>>>> ./gradlew clean >>>>> >>>>> On Mon, Jan 28, 2019 at 11:12 AM Ryan Williams <r...@runsascoded.com> >>>>> wrote: >>>>> >>>>>> Yea I was rebased on top of a more recent master than your previous >>>>>> message, when I saw it again. Perhaps I was mistaken. I'll ping here if I >>>>>> see it again, thanks. >>>>>> >>>>>> On Mon, Jan 28, 2019 at 1:52 PM Alex Amato <ajam...@google.com> >>>>>> wrote: >>>>>> >>>>>>> After I did a rebase, it went away for me. So I think that this >>>>>>> should work. Are you saying that you did rebase ontop of master and it >>>>>>> still occurred? Strange. >>>>>>> >>>>>>> On Sat, Jan 26, 2019 at 3:48 PM Ryan Williams <r...@runsascoded.com> >>>>>>> wrote: >>>>>>> >>>>>>>> Hm, I just encountered this again on a branch that based on >>>>>>>> 5b46b02b49 (top of trunk from this afternoon). Is it definitely >>>>>>>> supposed to >>>>>>>> be fixed? >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jan 24, 2019 at 9:19 PM Alex Amato <ajam...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Please try rebasing from master, I believe this issue has been >>>>>>>>> resolved. >>>>>>>>> >>>>>>>>> On Thu, Jan 24, 2019 at 3:29 PM Ryan Williams < >>>>>>>>> r...@runsascoded.com> wrote: >>>>>>>>> >>>>>>>>>> I'm seeing every ≈third `./gradlew compileJava` fail locally due >>>>>>>>>> to this; re-running the commit has always succeeded, so far. >>>>>>>>>> >>>>>>>>>> Sounds like there is not an immediate fix in the works / no one >>>>>>>>>> assigned on the JIRA? >>>>>>>>>> >>>>>>>>>> On Wed, Jan 23, 2019 at 3:17 PM Kenneth Knowles <k...@apache.org> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> This might connect to vendoring Calcite. It will be easiest, and >>>>>>>>>>> have the best incremental build, if we separate the generated code >>>>>>>>>>> into its >>>>>>>>>>> own module that has relocation to match the vendored Calcite. >>>>>>>>>>> >>>>>>>>>>> Kenn >>>>>>>>>>> >>>>>>>>>>> On Wed, Jan 23, 2019 at 11:29 AM Anton Kedin <ke...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> We don't pre-generate the code as a separate step. Code gen >>>>>>>>>>>> from the SQL parser syntax spec and its compilation happens both >>>>>>>>>>>> during the >>>>>>>>>>>> Beam SQL build task. Splitting the code generation and compilation >>>>>>>>>>>> might >>>>>>>>>>>> not be trivial. We definitely should look into fixing this though. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Anton >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jan 23, 2019 at 11:13 AM Alex Amato <ajam...@google.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Okay, make sense perhaps we can somehow make it fail when it >>>>>>>>>>>>> fails to generate the dep, rather than when compiling the java >>>>>>>>>>>>> code later on >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Jan 23, 2019 at 11:12 AM Anton Kedin <ke...@google.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> ParserImpl is autogenerated by Calcite at build time. It >>>>>>>>>>>>>> seems that there's a race condition there and it sometimes >>>>>>>>>>>>>> fails. Rerunning >>>>>>>>>>>>>> the build works for me. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>> Anton >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Wed, Jan 23, 2019, 11:06 AM Alex Amato <ajam...@google.com> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> https://jira.apache.org/jira/browse/BEAM-6495?filter=-2 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Any ideas, how this got through the precommit? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > Task :beam-sdks-java-extensions-sql:compileJava FAILED >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> /usr/local/google/home/ajamato/go/src/ >>>>>>>>>>>>>>> github.com/apache/beam/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29: >>>>>>>>>>>>>>> error: cannot find symbol >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> import >>>>>>>>>>>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ^ >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> symbol: class BeamSqlParserImpl >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> location: package >>>>>>>>>>>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1 error >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
