[jira] [Created] (CALCITE-6359) Update GitHub Actions workflows to use docker compose v2
Francis Chuang created CALCITE-6359: --- Summary: Update GitHub Actions workflows to use docker compose v2 Key: CALCITE-6359 URL: https://issues.apache.org/jira/browse/CALCITE-6359 Project: Calcite Issue Type: Task Components: avatica, avatica-go, core Reporter: Francis Chuang Assignee: Francis Chuang Fix For: 1.37.0, avatica-go-5.4.0, avatica-1.26.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5235) Run Github Actions tests using docker and upgrade Go
Francis Chuang created CALCITE-5235: --- Summary: Run Github Actions tests using docker and upgrade Go Key: CALCITE-5235 URL: https://issues.apache.org/jira/browse/CALCITE-5235 Project: Calcite Issue Type: Task Components: avatica-go Reporter: Francis Chuang Assignee: Francis Chuang Fix For: avatica-go-5.2.0 We should run Github Actions tests using docker to avoid duplication in the Github Actions workflow file. This should make it easier to update Go versions to test against when they are released. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (CALCITE-5059) Update Github Actions to latest versions
Francis Chuang created CALCITE-5059: --- Summary: Update Github Actions to latest versions Key: CALCITE-5059 URL: https://issues.apache.org/jira/browse/CALCITE-5059 Project: Calcite Issue Type: Task Components: avatica-go Reporter: Francis Chuang Assignee: Francis Chuang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (CALCITE-4829) Bump Gradle to 7.2 and test with Java 17 at GitHub Actions
Vladimir Sitnikov created CALCITE-4829: -- Summary: Bump Gradle to 7.2 and test with Java 17 at GitHub Actions Key: CALCITE-4829 URL: https://issues.apache.org/jira/browse/CALCITE-4829 Project: Calcite Issue Type: Improvement Components: build, core Reporter: Vladimir Sitnikov -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: GitHub actions disabled
The actions have been approved, so GitHub CI is back on track :) On top of that, I added potiuk/cancel-workflow-runs so it cancels workflows if the branch or PR is updated multiple times. Hopefully, it would reduce CI wait times. Please share if you happen to love or hate it. Vladimir
Re: GitHub actions disabled
I have no evidence for this whatsoever, but I have a hunch that GitHub actions could be exploited by bitcoin miners. I would be concerned that someone could create a PR and burn several cents of Apache's (or GitHub's) operational budget. On Fri, Jan 8, 2021 at 9:53 AM Michael Mior wrote: > > I suppose that is a fair concern. The restriction may at least force > folks to be aware of the potential risks. Although completely > eliminated by explicitly referencing a particular commit instead of a > tag. > -- > Michael Mior > mm...@apache.org > > Le ven. 8 janv. 2021 à 12:30, Vladimir Sitnikov > a écrit : > > > > Michael>Has there been a clear statement as to why the restrictions are in > > Michael>place? > > > > They say "for security reasons". > > > > Michael>seems like the restriction is rather pointless. > > > > My feeling exactly :-( > > > > I guess someone submitted something like > > https://julienrenaux.fr/2019/12/20/github-actions-security-risk/#the-problem > > as a security issue to the ASF, and it triggered the wave :-( > > > > I guess they mentioned that tag-based and branch-based action > > references like AdoptOpenJDK/install-jdk@v1 > > could silently change (e.g. git force push), and the action would silently > > capture > > secrets or even push something to the ASF repository. > > > > Vladimir
Re: GitHub actions disabled
I suppose that is a fair concern. The restriction may at least force folks to be aware of the potential risks. Although completely eliminated by explicitly referencing a particular commit instead of a tag. -- Michael Mior mm...@apache.org Le ven. 8 janv. 2021 à 12:30, Vladimir Sitnikov a écrit : > > Michael>Has there been a clear statement as to why the restrictions are in > Michael>place? > > They say "for security reasons". > > Michael>seems like the restriction is rather pointless. > > My feeling exactly :-( > > I guess someone submitted something like > https://julienrenaux.fr/2019/12/20/github-actions-security-risk/#the-problem > as a security issue to the ASF, and it triggered the wave :-( > > I guess they mentioned that tag-based and branch-based action > references like AdoptOpenJDK/install-jdk@v1 > could silently change (e.g. git force push), and the action would silently > capture > secrets or even push something to the ASF repository. > > Vladimir
Re: GitHub actions disabled
Has there been a clear statement as to why the restrictions are in place? Given that the workaround is so trivial, it seems like the restriction is rather pointless. -- Michael Mior mm...@apache.org Le jeu. 7 janv. 2021 à 11:53, Vladimir Sitnikov a écrit : > > I've found a workaround that unlocks arbitrary action. > See PR https://github.com/apache/calcite/pull/2318 > The idea is to clone the action via git clone, and then use the "local" > folder as the action source. > > As of now, we need only burrunan/gradle-cache-action > and AdoptOpenJDK/install-jdk, > so my suggestion would be to ask infra to allow those actions, and apply > the workaround > if the approval takes too long. > > Vladimir
Re: GitHub actions disabled
I've found a workaround that unlocks arbitrary action. See PR https://github.com/apache/calcite/pull/2318 The idea is to clone the action via git clone, and then use the "local" folder as the action source. As of now, we need only burrunan/gradle-cache-action and AdoptOpenJDK/install-jdk, so my suggestion would be to ask infra to allow those actions, and apply the workaround if the approval takes too long. Vladimir
Re: GitHub actions disabled
Francis> gradle/wrapper-validation-action@v1 It looks like wrapper-validation-action is already allowed for use. Francis>burrunan/gradle-cache-action I've filed https://issues.apache.org/jira/browse/INFRA-21275 to allow that action. Francis>- Avoid using third party actions, possibly replace with shell scripts? That depends. I would like to refrain from shell scripts as much as it is possible. Shell scripts have subtle differences across platforms, and it is hard to maintain them :( Vladimir
Re: GitHub actions disabled
Thanks for bringing this up, Julian. I was reading the thread on builds@ and infra [1] earlier this afternoon and I think they are just blocking actions / steps that are not found the github.com/actions, github.com/apache or github.com/github organizations. I just when through our repos to see if there are any non-apache or non-github steps / actions and here are my findings: ## Avatica-Go https://github.com/apache/calcite-avatica-go/blob/master/.github/workflows/tests.yaml - OK (no third party actions) ## Avatica https://github.com/apache/calcite-avatica/blob/master/.github/workflows/gradle-wrapper-validation.yml - Uses gradle/wrapper-validation-action@v1 in a step, so will fail https://github.com/apache/calcite-avatica/blob/master/.github/workflows/main.yml - OK (no third party actions) ## Calcite https://github.com/apache/calcite/blob/master/.github/workflows/gradle-wrapper-validation.yml - Uses gradle/wrapper-validation-action@v1 in a step, so will fail https://github.com/apache/calcite/blob/master/.github/workflows/main.yml - Uses burrunan/gradle-cache-action@v1, so will fail. Potential solutions: - Avoid using third party actions, possibly replace with shell scripts? - Try to get those whitelisted or moved into the apache/ org. - Bring those actions into our own repositories (maintenance nightmare?) Francis [1] https://lists.apache.org/thread.html/r900f8f9a874006ed8121bdc901a0d1acccbb340882c1f94dad61a5e9%40%3Cusers.infra.apache.org%3E On 29/12/2020 6:14 pm, Julian Hyde wrote: As of yesterday, GitHub actions are disabled for any non-Apache repository. I think this means that GitHub actions will not fire for PRs (which are typically in the contributor’s repository). See https://issues.apache.org/jira/browse/INFRA-21234 <https://issues.apache.org/jira/browse/INFRA-21234> for more details. And I see that Arrow are facing the same problem [1]. Please be wary of merging PRs that have not passed the usual checks. Julian [1] https://lists.apache.org/thread.html/rc120149c464f8767daf042bc6f8016ce4a3d87e3727cbb8fe024ca99%40%3Cdev.arrow.apache.org%3E <https://lists.apache.org/thread.html/rc120149c464f8767daf042bc6f8016ce4a3d87e3727cbb8fe024ca99@%3Cdev.arrow.apache.org%3E>
GitHub actions disabled
As of yesterday, GitHub actions are disabled for any non-Apache repository. I think this means that GitHub actions will not fire for PRs (which are typically in the contributor’s repository). See https://issues.apache.org/jira/browse/INFRA-21234 <https://issues.apache.org/jira/browse/INFRA-21234> for more details. And I see that Arrow are facing the same problem [1]. Please be wary of merging PRs that have not passed the usual checks. Julian [1] https://lists.apache.org/thread.html/rc120149c464f8767daf042bc6f8016ce4a3d87e3727cbb8fe024ca99%40%3Cdev.arrow.apache.org%3E <https://lists.apache.org/thread.html/rc120149c464f8767daf042bc6f8016ce4a3d87e3727cbb8fe024ca99@%3Cdev.arrow.apache.org%3E>
[jira] [Created] (CALCITE-3904) Upgrade github actions dependencies
Francis Chuang created CALCITE-3904: --- Summary: Upgrade github actions dependencies Key: CALCITE-3904 URL: https://issues.apache.org/jira/browse/CALCITE-3904 Project: Calcite Issue Type: Task Components: avatica-go Reporter: Francis Chuang Assignee: Francis Chuang Fix For: avatica-go-5.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Avatica + GitHub Actions + Windows = AvaticaSpnegoTest - Principal: HTTP/stratum.antpool....@example.com is not known
The error is caused by private static final boolean USE_CANONICAL_HOSTNAME = true; in https://github.com/apache/calcite-avatica/blob/73fac75befaaf3caa3871656d0d0b7adbd91f7cf/core/src/main/java/org/apache/calcite/avatica/remote/AvaticaCommonsHttpClientSpnegoImpl.java#L58 In other words, httpclient is using the "canonical hostname" when sending the request rather than localhost. I guess the property should be configurable via connection URL. I've added -D for now. CI is OK, and it would be great if someone could create a connection parameter for configuring the use of "canonical hostname". Vladimir
Re: Avatica + GitHub Actions + Windows = AvaticaSpnegoTest - Principal: HTTP/stratum.antpool....@example.com is not known
It's strange that you see this on Windows. Everything should be using localhost in the tests. Maybe that means it's something specific to Windows? I don't know enough to say if that's a reasonable guess or not. The principal looked up stems from the hostname you issue a request to. e.g. if you try to SPNEGO with http://foobar.google.com, the client would look up HTTP/foobar.google.com in the KDC. On 11/11/19 3:54 PM, Vladimir Sitnikov wrote: localhost but the principal being looked up is FQDN. Where that FQDN is taken from? Vladimir
Re: Avatica + GitHub Actions + Windows = AvaticaSpnegoTest - Principal: HTTP/stratum.antpool....@example.com is not known
>localhost but the principal being looked up is FQDN. Where that FQDN is taken from? Vladimir
Re: Avatica + GitHub Actions + Windows = AvaticaSpnegoTest - Principal: HTTP/stratum.antpool....@example.com is not known
Yeah, the expectation is that all of the tests are using localhost (to specifically avoid issues around trying to pull the FQDN and relying on the developer to have specific setups). On Mon, Nov 11, 2019 at 3:38 PM Kevin Risden wrote: > > > > > 2019-11-10 21:29:30,443 [pool-1-thread-2] ERROR - Principal: HTTP/ > > stratum.antpool@example.com is not known > > 2019-11-10 21:29:30,459 [Test worker] WARN - NEGOTIATE authentication > > error: No valid credentials provided (Mechanism level: No valid credentials > > provided (Mechanism level: Server not found in Kerberos database (7) - > > Server not found in Kerberos database)) > > > Usually means DNS or the SPN being acquired is not correct. Should look at > the test and see if it is trying to something like localhost and instead > doing FQDN instead. Looks like > https://github.com/apache/calcite-avatica/blob/master/server/src/test/java/org/apache/calcite/avatica/SpnegoTestUtil.java#L58 > does > localhost but the principal being looked up is FQDN. > Kevin Risden > > > > On Sun, Nov 10, 2019 at 4:53 PM Vladimir Sitnikov < > sitnikov.vladi...@gmail.com> wrote: > > > Hi, > > > > I've added GitHub Actions for CI, and it looks like AvaticaSpnegoTest fails > > again. > > > > The failure is the same for Maven and Gradle, so I expect it is not caused > > by Gradle migration > > > > Maven: https://github.com/vlsi/calcite-avatica/runs/296688961#step:4:31296 > > Gradle: > > > > https://github.com/vlsi/calcite-avatica/commit/d1bb6f04c905ed65931800cde0af6a9899e585b6/checks?check_suite_id=304254116#step:4:117 > > > > org.apache.calcite.avatica.AvaticaSpnegoTest > testAuthenticatedClient[0] > > STANDARD_OUT > > 2019-11-10 21:29:29,707 [pool-1-thread-1] INFO - The preauth data is > > empty. > > 2019-11-10 21:29:29,707 [pool-1-thread-1] INFO - KRB error occurred > > while processing request: Additional pre-authentication required > > 2019-11-10 21:29:29,785 [pool-1-thread-2] INFO - AS_REQ ISSUE: > > authtime 1573421369770,cli...@example.com for krbtgt/ > > example@example.com > > 2019-11-10 21:29:30,443 [pool-1-thread-2] ERROR - Principal: HTTP/ > > stratum.antpool@example.com is not known > > 2019-11-10 21:29:30,459 [Test worker] WARN - NEGOTIATE authentication > > error: No valid credentials provided (Mechanism level: No valid credentials > > provided (Mechanism level: Server not found in Kerberos database (7) - > > Server not found in Kerberos database)) > > > > org.apache.calcite.avatica.AvaticaSpnegoTest > testAuthenticatedClient[0] > > FAILED > > java.lang.RuntimeException: Failed to execute HTTP Request, got > > HTTP/401 > > at > > > > org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientSpnegoImpl.send(AvaticaCommonsHttpClientSpnegoImpl.java:134) > > at > > > > org.apache.calcite.avatica.remote.RemoteService.apply(RemoteService.java:34) > > at > > org.apache.calcite.avatica.remote.JsonService.apply(JsonService.java:172) > > at > > org.apache.calcite.avatica.remote.Driver.connect(Driver.java:176) > > at > > java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) > > at > > java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251) > > at > > > > org.apache.calcite.avatica.AvaticaSpnegoTest$1.run(AvaticaSpnegoTest.java:225) > > at > > > > org.apache.calcite.avatica.AvaticaSpnegoTest$1.run(AvaticaSpnegoTest.java:223) > > at java.base/java.security.AccessController.doPrivileged(Native > > Method) > > at java.base/javax.security.auth.Subject.doAs(Subject.java:423) > > at > > > > org.apache.calcite.avatica.AvaticaSpnegoTest.testAuthenticatedClient(AvaticaSpnegoTest.java:223) > > > > Is it something known? > > > > PS. The failure is only for GitHub Actions + Windows. GitHub Actions macOS > > test passes. > > > > Vladimir > >
Re: Avatica + GitHub Actions + Windows = AvaticaSpnegoTest - Principal: HTTP/stratum.antpool....@example.com is not known
> > 2019-11-10 21:29:30,443 [pool-1-thread-2] ERROR - Principal: HTTP/ > stratum.antpool@example.com is not known > 2019-11-10 21:29:30,459 [Test worker] WARN - NEGOTIATE authentication > error: No valid credentials provided (Mechanism level: No valid credentials > provided (Mechanism level: Server not found in Kerberos database (7) - > Server not found in Kerberos database)) Usually means DNS or the SPN being acquired is not correct. Should look at the test and see if it is trying to something like localhost and instead doing FQDN instead. Looks like https://github.com/apache/calcite-avatica/blob/master/server/src/test/java/org/apache/calcite/avatica/SpnegoTestUtil.java#L58 does localhost but the principal being looked up is FQDN. Kevin Risden On Sun, Nov 10, 2019 at 4:53 PM Vladimir Sitnikov < sitnikov.vladi...@gmail.com> wrote: > Hi, > > I've added GitHub Actions for CI, and it looks like AvaticaSpnegoTest fails > again. > > The failure is the same for Maven and Gradle, so I expect it is not caused > by Gradle migration > > Maven: https://github.com/vlsi/calcite-avatica/runs/296688961#step:4:31296 > Gradle: > > https://github.com/vlsi/calcite-avatica/commit/d1bb6f04c905ed65931800cde0af6a9899e585b6/checks?check_suite_id=304254116#step:4:117 > > org.apache.calcite.avatica.AvaticaSpnegoTest > testAuthenticatedClient[0] > STANDARD_OUT > 2019-11-10 21:29:29,707 [pool-1-thread-1] INFO - The preauth data is > empty. > 2019-11-10 21:29:29,707 [pool-1-thread-1] INFO - KRB error occurred > while processing request: Additional pre-authentication required > 2019-11-10 21:29:29,785 [pool-1-thread-2] INFO - AS_REQ ISSUE: > authtime 1573421369770,cli...@example.com for krbtgt/ > example@example.com > 2019-11-10 21:29:30,443 [pool-1-thread-2] ERROR - Principal: HTTP/ > stratum.antpool@example.com is not known > 2019-11-10 21:29:30,459 [Test worker] WARN - NEGOTIATE authentication > error: No valid credentials provided (Mechanism level: No valid credentials > provided (Mechanism level: Server not found in Kerberos database (7) - > Server not found in Kerberos database)) > > org.apache.calcite.avatica.AvaticaSpnegoTest > testAuthenticatedClient[0] > FAILED > java.lang.RuntimeException: Failed to execute HTTP Request, got > HTTP/401 > at > > org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientSpnegoImpl.send(AvaticaCommonsHttpClientSpnegoImpl.java:134) > at > > org.apache.calcite.avatica.remote.RemoteService.apply(RemoteService.java:34) > at > org.apache.calcite.avatica.remote.JsonService.apply(JsonService.java:172) > at > org.apache.calcite.avatica.remote.Driver.connect(Driver.java:176) > at > java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) > at > java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251) > at > > org.apache.calcite.avatica.AvaticaSpnegoTest$1.run(AvaticaSpnegoTest.java:225) > at > > org.apache.calcite.avatica.AvaticaSpnegoTest$1.run(AvaticaSpnegoTest.java:223) > at java.base/java.security.AccessController.doPrivileged(Native > Method) > at java.base/javax.security.auth.Subject.doAs(Subject.java:423) > at > > org.apache.calcite.avatica.AvaticaSpnegoTest.testAuthenticatedClient(AvaticaSpnegoTest.java:223) > > Is it something known? > > PS. The failure is only for GitHub Actions + Windows. GitHub Actions macOS > test passes. > > Vladimir >
Avatica + GitHub Actions + Windows = AvaticaSpnegoTest - Principal: HTTP/stratum.antpool....@example.com is not known
Hi, I've added GitHub Actions for CI, and it looks like AvaticaSpnegoTest fails again. The failure is the same for Maven and Gradle, so I expect it is not caused by Gradle migration Maven: https://github.com/vlsi/calcite-avatica/runs/296688961#step:4:31296 Gradle: https://github.com/vlsi/calcite-avatica/commit/d1bb6f04c905ed65931800cde0af6a9899e585b6/checks?check_suite_id=304254116#step:4:117 org.apache.calcite.avatica.AvaticaSpnegoTest > testAuthenticatedClient[0] STANDARD_OUT 2019-11-10 21:29:29,707 [pool-1-thread-1] INFO - The preauth data is empty. 2019-11-10 21:29:29,707 [pool-1-thread-1] INFO - KRB error occurred while processing request: Additional pre-authentication required 2019-11-10 21:29:29,785 [pool-1-thread-2] INFO - AS_REQ ISSUE: authtime 1573421369770,cli...@example.com for krbtgt/example@example.com 2019-11-10 21:29:30,443 [pool-1-thread-2] ERROR - Principal: HTTP/ stratum.antpool@example.com is not known 2019-11-10 21:29:30,459 [Test worker] WARN - NEGOTIATE authentication error: No valid credentials provided (Mechanism level: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - Server not found in Kerberos database)) org.apache.calcite.avatica.AvaticaSpnegoTest > testAuthenticatedClient[0] FAILED java.lang.RuntimeException: Failed to execute HTTP Request, got HTTP/401 at org.apache.calcite.avatica.remote.AvaticaCommonsHttpClientSpnegoImpl.send(AvaticaCommonsHttpClientSpnegoImpl.java:134) at org.apache.calcite.avatica.remote.RemoteService.apply(RemoteService.java:34) at org.apache.calcite.avatica.remote.JsonService.apply(JsonService.java:172) at org.apache.calcite.avatica.remote.Driver.connect(Driver.java:176) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677) at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:251) at org.apache.calcite.avatica.AvaticaSpnegoTest$1.run(AvaticaSpnegoTest.java:225) at org.apache.calcite.avatica.AvaticaSpnegoTest$1.run(AvaticaSpnegoTest.java:223) at java.base/java.security.AccessController.doPrivileged(Native Method) at java.base/javax.security.auth.Subject.doAs(Subject.java:423) at org.apache.calcite.avatica.AvaticaSpnegoTest.testAuthenticatedClient(AvaticaSpnegoTest.java:223) Is it something known? PS. The failure is only for GitHub Actions + Windows. GitHub Actions macOS test passes. Vladimir
Re: Github Actions for CI
>Definitely would simplify some of >the RM steps for a release. I've recently added GitHub Actions config to test Apache JMeter for Windows and macOS. So far I'm impressed. Pros: * It starts quite fast. Appveyor might take 2-3hours to even start the build. Actions start very fast, and it does catch Windows-specific issues like CRLF, "un-ability to remove a file that is not closed" and "un-ability to remove a read-only file" Cons: * It always fetches all the refs from the Git repository (not just master+pr, but it fetches even gh-pages branch) which causes issues like https://github.com/junit-team/junit5/issues/2048 . It is fixable, but the defaults are odd. * Caching seems to be not there. Dependencies seem to be downloaded on each build. * There's a lot of jitter. JMeter does have several tests like "thread.sleep(200) + assert actual duration", and they fail way more often in Actions. * GitHub Actions is likely unavailable for the forked repositories. In other words, I cannot easily use Actions in my fork repository. I can use Travis, and Calcite's travis.yml works for my repository. However, if Calcite migrates to Actions, then forks would be harder to test (PR would be required). Vladimir
Re: Github Actions for CI
So I've been wanting to sit down and look at this closer so its exciting to see progress :) Thanks for digging in. I'd be interested in looking at replacing Travis and seeing what it takes for the build with separate JVMs. One thing that would be interesting would be to move the integration > tests for Calcite to docker images and run each test for each backend in > its own job to maximize parallelism and allow each job to maximize the > available hardware resources. > I think this is a separate but related idea. It would be good to have integration tests run without a beefy VM. Definitely would simplify some of the RM steps for a release. Kevin Risden On Tue, Sep 17, 2019 at 7:20 PM Francis Chuang wrote: > A month or so ago, I started a thread on Github Actions on the list. I > have just replaced Travis with Github Actions for Avatica-Go > (CALCITE-3356). > > For the workflows in Avatica-Go (currently just 1 for CI), see: > https://github.com/apache/calcite-avatica-go/tree/master/.github/workflows > > The latest run is here: > https://github.com/apache/calcite-avatica-go/runs/226087805 > > The VMs provided by Github are a lot beefier (2 core CPUs, 7GB RAM, 14GB > SSD disk). The tests are slightly slower compare to Travis (in the order > of seconds), but this is most likely due to us having to check out the > code during run time and the need to compile the Github Action for > checking out the code for each run. I don't think this is too much of an > issue and the direct integration with Github is much nicer. > > I think it shouldn't be too difficult to move Avatica and Calcite over. > We can probably get rid of Appveyor as well since Github Actions > provides Windows and OS X nodes as well. > > One thing that would be interesting would be to move the integration > tests for Calcite to docker images and run each test for each backend in > its own job to maximize parallelism and allow each job to maximize the > available hardware resources. > > Francis >
Github Actions for CI
A month or so ago, I started a thread on Github Actions on the list. I have just replaced Travis with Github Actions for Avatica-Go (CALCITE-3356). For the workflows in Avatica-Go (currently just 1 for CI), see: https://github.com/apache/calcite-avatica-go/tree/master/.github/workflows The latest run is here: https://github.com/apache/calcite-avatica-go/runs/226087805 The VMs provided by Github are a lot beefier (2 core CPUs, 7GB RAM, 14GB SSD disk). The tests are slightly slower compare to Travis (in the order of seconds), but this is most likely due to us having to check out the code during run time and the need to compile the Github Action for checking out the code for each run. I don't think this is too much of an issue and the direct integration with Github is much nicer. I think it shouldn't be too difficult to move Avatica and Calcite over. We can probably get rid of Appveyor as well since Github Actions provides Windows and OS X nodes as well. One thing that would be interesting would be to move the integration tests for Calcite to docker images and run each test for each backend in its own job to maximize parallelism and allow each job to maximize the available hardware resources. Francis
[jira] [Created] (CALCITE-3356) Use Github Actions for continuous integration
Francis Chuang created CALCITE-3356: --- Summary: Use Github Actions for continuous integration Key: CALCITE-3356 URL: https://issues.apache.org/jira/browse/CALCITE-3356 Project: Calcite Issue Type: Improvement Components: avatica-go Reporter: Francis Chuang Assignee: Francis Chuang Fix For: avatica-go-5.0.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Github Actions
That's great news, thanks a lot Francis! The option of using actions for CI is really appealing especially if you say that is fast. I am trying to enable slow tests in Travis but I don't think it is going to happen since they are really slow. I will try to see what can be done using actions. Best, Stamatis On Tue, Aug 20, 2019 at 1:24 PM Michael Mior wrote: > Sounds great! I've been wanting to play around with GitHub Actions > more myself and this sounds encouraging. > -- > Michael Mior > mm...@apache.org > > Le lun. 19 août 2019 à 18:15, Francis Chuang > a écrit : > > > > Hey everyone, > > > > Previously I mentioned that I was working on automated website builds > > for the Calcite (and avatica and avatica-go subwebsite) Website [1] and > > I wanted to give you guys some updates. > > > > The initial attempt was to use the git-websites jenkins node to build > > the site, however I ran into a few issues: > > - I couldn't get a Jenkins build to trigger only when a new tag that > > matches a regex is pushed. > > - The jenkins build can sometimes be unreliable and not trigger. > > > > Recently, Github released Github Actions, which allows not just CI/CD, > > but the ability to run any piece of code in Javascript, a docker > > container or a VM when an event triggers it [2]. The newest beta of > > Github Actions is available to all projects under the ASF organization > > on Github, so do check it out, it's pretty cool! > > > > I have successfully implemented automated site builds using Github > > Actions in the test-site branch [3]. The builds now trigger correctly > > and the only missing piece left is to let Github Actions have access to > > the token to push to the Calcite-site repository. This is currently > > tracked in INFRA-18874 [4]. > > > > Github Actions can also be used for CI/CD, so it's a good candidate for > > replacing Travis and Appveyor as Github also provides Windows nodes for > > building. I did some initial work to test the waters with migrating > > Avatica-Go [5]. Unfortunately, it seems to be missing some things, such > > as the ability to set the command when running a service container. I am > > sure these are things that will definitely be fixed/implemented in the > > future as Actions is still in beta. > > > > From my testing, Github Actions builds are really fast (9 minutes to > > build the Javadoc for Calcite), so it's something we should consider for > > Calcite + Avatica + Avatica-Go in the future. > > > > Any thoughts? > > > > I've also started some discussion regarding Github Actions on the build@ > > list for more general discussion [6]. > > > > Francis > > > > [1] > > > https://lists.apache.org/thread.html/a86bf62901f4032027d7a247c33aa22e0f74ad423914967a86f769b8@%3Cdev.calcite.apache.org%3E > > > > [2] https://github.com/features/actions > > > > [3] > > > https://github.com/apache/calcite/commit/eb44ae0ae6f9e5197067452f036bd07c92b4c92a > > > > [4] https://issues.apache.org/jira/browse/INFRA-18874 > > > > [5] > > > https://github.com/apache/calcite-avatica-go/commit/a83e09408aeac7706b2a482d704e5d1c758eafb2 > > > > [6] > > > https://lists.apache.org/thread.html/246c44267ddaede1cda516c4c825a3868cf617dbe09a6106a8a29d92@%3Cbuilds.apache.org%3E >
Re: Github Actions
Sounds great! I've been wanting to play around with GitHub Actions more myself and this sounds encouraging. -- Michael Mior mm...@apache.org Le lun. 19 août 2019 à 18:15, Francis Chuang a écrit : > > Hey everyone, > > Previously I mentioned that I was working on automated website builds > for the Calcite (and avatica and avatica-go subwebsite) Website [1] and > I wanted to give you guys some updates. > > The initial attempt was to use the git-websites jenkins node to build > the site, however I ran into a few issues: > - I couldn't get a Jenkins build to trigger only when a new tag that > matches a regex is pushed. > - The jenkins build can sometimes be unreliable and not trigger. > > Recently, Github released Github Actions, which allows not just CI/CD, > but the ability to run any piece of code in Javascript, a docker > container or a VM when an event triggers it [2]. The newest beta of > Github Actions is available to all projects under the ASF organization > on Github, so do check it out, it's pretty cool! > > I have successfully implemented automated site builds using Github > Actions in the test-site branch [3]. The builds now trigger correctly > and the only missing piece left is to let Github Actions have access to > the token to push to the Calcite-site repository. This is currently > tracked in INFRA-18874 [4]. > > Github Actions can also be used for CI/CD, so it's a good candidate for > replacing Travis and Appveyor as Github also provides Windows nodes for > building. I did some initial work to test the waters with migrating > Avatica-Go [5]. Unfortunately, it seems to be missing some things, such > as the ability to set the command when running a service container. I am > sure these are things that will definitely be fixed/implemented in the > future as Actions is still in beta. > > From my testing, Github Actions builds are really fast (9 minutes to > build the Javadoc for Calcite), so it's something we should consider for > Calcite + Avatica + Avatica-Go in the future. > > Any thoughts? > > I've also started some discussion regarding Github Actions on the build@ > list for more general discussion [6]. > > Francis > > [1] > https://lists.apache.org/thread.html/a86bf62901f4032027d7a247c33aa22e0f74ad423914967a86f769b8@%3Cdev.calcite.apache.org%3E > > [2] https://github.com/features/actions > > [3] > https://github.com/apache/calcite/commit/eb44ae0ae6f9e5197067452f036bd07c92b4c92a > > [4] https://issues.apache.org/jira/browse/INFRA-18874 > > [5] > https://github.com/apache/calcite-avatica-go/commit/a83e09408aeac7706b2a482d704e5d1c758eafb2 > > [6] > https://lists.apache.org/thread.html/246c44267ddaede1cda516c4c825a3868cf617dbe09a6106a8a29d92@%3Cbuilds.apache.org%3E
Github Actions
Hey everyone, Previously I mentioned that I was working on automated website builds for the Calcite (and avatica and avatica-go subwebsite) Website [1] and I wanted to give you guys some updates. The initial attempt was to use the git-websites jenkins node to build the site, however I ran into a few issues: - I couldn't get a Jenkins build to trigger only when a new tag that matches a regex is pushed. - The jenkins build can sometimes be unreliable and not trigger. Recently, Github released Github Actions, which allows not just CI/CD, but the ability to run any piece of code in Javascript, a docker container or a VM when an event triggers it [2]. The newest beta of Github Actions is available to all projects under the ASF organization on Github, so do check it out, it's pretty cool! I have successfully implemented automated site builds using Github Actions in the test-site branch [3]. The builds now trigger correctly and the only missing piece left is to let Github Actions have access to the token to push to the Calcite-site repository. This is currently tracked in INFRA-18874 [4]. Github Actions can also be used for CI/CD, so it's a good candidate for replacing Travis and Appveyor as Github also provides Windows nodes for building. I did some initial work to test the waters with migrating Avatica-Go [5]. Unfortunately, it seems to be missing some things, such as the ability to set the command when running a service container. I am sure these are things that will definitely be fixed/implemented in the future as Actions is still in beta. From my testing, Github Actions builds are really fast (9 minutes to build the Javadoc for Calcite), so it's something we should consider for Calcite + Avatica + Avatica-Go in the future. Any thoughts? I've also started some discussion regarding Github Actions on the build@ list for more general discussion [6]. Francis [1] https://lists.apache.org/thread.html/a86bf62901f4032027d7a247c33aa22e0f74ad423914967a86f769b8@%3Cdev.calcite.apache.org%3E [2] https://github.com/features/actions [3] https://github.com/apache/calcite/commit/eb44ae0ae6f9e5197067452f036bd07c92b4c92a [4] https://issues.apache.org/jira/browse/INFRA-18874 [5] https://github.com/apache/calcite-avatica-go/commit/a83e09408aeac7706b2a482d704e5d1c758eafb2 [6] https://lists.apache.org/thread.html/246c44267ddaede1cda516c4c825a3868cf617dbe09a6106a8a29d92@%3Cbuilds.apache.org%3E
