I still see this about 20% of the time I run `./gradlew compileTestJava spotlessApply checkstyleMain checkstyleTest` (with 8-way parallelism), which I've taken to running as a kind of "pre-PreCommit" (for Java).
In total I'd estimate I've seen this error 20 times (across multiple `./gradlew clean`s and spanning several weeks), but never twice in a row: I always immediately re-run, and the error goes away. That seems consistent with the idea that gradle is racing the codegen task against the compile task that requires the codegen's outputs. However, `./gradlew -p sdks/java/extensions/sql -m compileJava | grep compileJavacc` implies that Gradle knows that compilation depends on the codegen having run. Here are three successive invocations of `./gradlew compileTestJava spotlessApply checkstyleMain checkstyleTest` <https://gist.github.com/ryan-williams/6d2735b2484e9459be656abf379533c5>; - The first one succeeds (and includes output showing that the codegen task, `:beam-sdks-java-extensions-sql:compileJavacc`, was run). - Then, I rebased on one new upstream commit and ran again, and saw the failure. - Immediately re-running saw the failure go away. I guess in that output, tasks that are run but produce no stdout/stderr are elided from the output, so we can't completely tell when the codegen and compile tasks in question are being run. I'll start running with --console=verbose <https://stackoverflow.com/questions/45883963/gradle-4-0-does-not-display-executed-tasks-in-command-line#comment92914258_45889966>, which I think will give a better sense of how this is happening. On Fri, Feb 8, 2019 at 12:41 AM Ryan Williams <r...@runsascoded.com> wrote: > After your last message, Alex, I saw the issue on a branch with minimal > unrelated changes on top of b4b5495307 > <https://github.com/apache/beam/commit/b4b549530730d9e0283ad0cdb203d384a107dfbf> > (January > 28). > > I ran `./gradlew clean` for the first time in a while at that time, and > worked happily for 11 days, but just saw the issue again on a branch > rebased on top of 381ab55b59 > <https://github.com/apache/beam/commit/381ab55b59> (today): > > > Task :beam-sdks-java-extensions-sql:compileJava FAILED > …/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29: > error: cannot find symbol > import > org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; > ^ > symbol: class BeamSqlParserImpl > location: package org.apache.beam.sdk.extensions.sql.impl.parser.impl > 1 error > > Every time I've seen it, I immediately re-run the compile, and it succeeds. > > Perhaps something is still wrong with my environment, but otherwise it > would seem that something is still flaky here. > > I'm compiling on an 8-core macOS machine, fwiw, and usually running > `./gradlew compileTestJava` which compiles many projects concurrently. > > On Mon, Jan 28, 2019 at 4:12 PM Alex Amato <ajam...@google.com> wrote: > >> If it continues to occur, maybe it is an environmental issue, be sure to >> try to clean as well. >> ./gradlew clean >> >> On Mon, Jan 28, 2019 at 11:12 AM Ryan Williams <r...@runsascoded.com> >> wrote: >> >>> Yea I was rebased on top of a more recent master than your previous >>> message, when I saw it again. Perhaps I was mistaken. I'll ping here if I >>> see it again, thanks. >>> >>> On Mon, Jan 28, 2019 at 1:52 PM Alex Amato <ajam...@google.com> wrote: >>> >>>> After I did a rebase, it went away for me. So I think that this should >>>> work. Are you saying that you did rebase ontop of master and it still >>>> occurred? Strange. >>>> >>>> On Sat, Jan 26, 2019 at 3:48 PM Ryan Williams <r...@runsascoded.com> >>>> wrote: >>>> >>>>> Hm, I just encountered this again on a branch that based on 5b46b02b49 >>>>> (top of trunk from this afternoon). Is it definitely supposed to be fixed? >>>>> >>>>> >>>>> On Thu, Jan 24, 2019 at 9:19 PM Alex Amato <ajam...@google.com> wrote: >>>>> >>>>>> Please try rebasing from master, I believe this issue has been >>>>>> resolved. >>>>>> >>>>>> On Thu, Jan 24, 2019 at 3:29 PM Ryan Williams <r...@runsascoded.com> >>>>>> wrote: >>>>>> >>>>>>> I'm seeing every ≈third `./gradlew compileJava` fail locally due to >>>>>>> this; re-running the commit has always succeeded, so far. >>>>>>> >>>>>>> Sounds like there is not an immediate fix in the works / no one >>>>>>> assigned on the JIRA? >>>>>>> >>>>>>> On Wed, Jan 23, 2019 at 3:17 PM Kenneth Knowles <k...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>>> This might connect to vendoring Calcite. It will be easiest, and >>>>>>>> have the best incremental build, if we separate the generated code >>>>>>>> into its >>>>>>>> own module that has relocation to match the vendored Calcite. >>>>>>>> >>>>>>>> Kenn >>>>>>>> >>>>>>>> On Wed, Jan 23, 2019 at 11:29 AM Anton Kedin <ke...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> We don't pre-generate the code as a separate step. Code gen from >>>>>>>>> the SQL parser syntax spec and its compilation happens both during >>>>>>>>> the Beam >>>>>>>>> SQL build task. Splitting the code generation and compilation might >>>>>>>>> not be >>>>>>>>> trivial. We definitely should look into fixing this though. >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Anton >>>>>>>>> >>>>>>>>> On Wed, Jan 23, 2019 at 11:13 AM Alex Amato <ajam...@google.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Okay, make sense perhaps we can somehow make it fail when it >>>>>>>>>> fails to generate the dep, rather than when compiling the java code >>>>>>>>>> later on >>>>>>>>>> >>>>>>>>>> On Wed, Jan 23, 2019 at 11:12 AM Anton Kedin <ke...@google.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> ParserImpl is autogenerated by Calcite at build time. It seems >>>>>>>>>>> that there's a race condition there and it sometimes fails. >>>>>>>>>>> Rerunning the >>>>>>>>>>> build works for me. >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Anton >>>>>>>>>>> >>>>>>>>>>> On Wed, Jan 23, 2019, 11:06 AM Alex Amato <ajam...@google.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> https://jira.apache.org/jira/browse/BEAM-6495?filter=-2 >>>>>>>>>>>> >>>>>>>>>>>> Any ideas, how this got through the precommit? >>>>>>>>>>>> >>>>>>>>>>>> > Task :beam-sdks-java-extensions-sql:compileJava FAILED >>>>>>>>>>>> >>>>>>>>>>>> /usr/local/google/home/ajamato/go/src/ >>>>>>>>>>>> github.com/apache/beam/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/JdbcFactory.java:29: >>>>>>>>>>>> error: cannot find symbol >>>>>>>>>>>> >>>>>>>>>>>> import >>>>>>>>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl.BeamSqlParserImpl; >>>>>>>>>>>> >>>>>>>>>>>> ^ >>>>>>>>>>>> >>>>>>>>>>>> symbol: class BeamSqlParserImpl >>>>>>>>>>>> >>>>>>>>>>>> location: package >>>>>>>>>>>> org.apache.beam.sdk.extensions.sql.impl.parser.impl >>>>>>>>>>>> >>>>>>>>>>>> 1 error >>>>>>>>>>>> >>>>>>>>>>>>
