Re: [Twisted-Python] GitHub Actions parallelism limit increase
On Tue, 6 Apr 2021 at 19:58, Kyle Altendorf wrote: > On 2021-04-01 09:36, Adi Roiban wrote: > > Thanks Kyle for the GitHub communication. I have checked Twisted billing and we have GitHub Teams enabled. So yes, 60x concurrent jobs on GitHub own VMs. GitHub billing now shows $124.00 / month to cover our Team plan but we don't have to pay. For reference Twisted has 41 members [0] Upgrading to enterprise will be $903.00 / month :) -- We now use 13x jobs for a build - 10 - python + 3 lint/mypy/docs. Python jobs now take a maximum of 8 minutes (on 2xCPU VMs with 4 parallel jobs) [1] ... from the previous 30+ minutes In the last month there were about 250 jobs so about 8 per day. So I think that we are ok for now. -- We have 12 extra jobs on Azure. I think that as an experiment we can move 1 Windows and 1 macOS job from Azure to GitHub Actions. I would still keep Azure pipelines jobs to use those extra VMs that are available. Azure allows retrying a single job on failure , or all failed jobs. This is a bit nicery for flaky tests. In GitHub, you have to re-run the whole workflow. - We can get more jobs by hosting our own runners... but that is a different story. [snip] Sounds good. Sounds like exactly what I suggested. :] Except for > maybe misunderstanding about what level of separation is needed for the > interjob dependencies. > > Hopefully this clarifies a bit. I think we actually intend the same > thing. Unless I've missed something else? > I guess that is best to discuss directly on the change made for a PR and see how the final matrix will look. Top priority for me is enabling trial parallel runs on Windows (PR waiting for review) to use the extra CPU available on the VM. Next is enabling some kind of PR triage helper. [0] https://github.com/orgs/twisted/people [1] https://github.com/twisted/twisted/actions/runs/735543893 [2] https://dev.azure.com/twistedmatrix/twisted/_build/results?buildId=3717=results -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On 2021-04-01 09:36, Adi Roiban wrote: For example, I don't understand why you need to create an sdist and from sdist to create a wheel and then install that wheel inside tox for each tox test environment. Why not create the wheel once and install the same wheel in all environments. Are you referring to the use of tox-wheel? Note that the workflow I setup in towncrier has a single build job that creates an sdist and from that a wheel one time, uploads those as artifacts in the workflow, and then every other test job and the publish job use those exact pre-created files. So, I think we agree here. I'll let Grainger weigh in on their present take on tox-wheel if they want. There is an extra 1 minute required for each job to generate the coverage XML file. We can download all the coverage artifacts and generate the XML only once. I did coverage collection from all jobs for pymodbus. Looks like I still had the XML report generating in each job, but that could presumably be shifted. Anyways, I'm sure several of us could do this off the top of our heads but in case it is of any interest here's what I did. Or maybe you just point out something silly I did there and should learn from. :] https://github.com/riptideio/pymodbus/pull/592/files That is one big matrix :) ... I don't understand why you need test and check and coverage, each with a separate matrix. I don't think it really is as big as you think. The test and coverage matrix are separated (and the coverage matrix is kind of not a matrix, what with one job) because that's the level at which GitHub allows you to describe dependencies. If you want to collect the coverage during test runs and then combine all the results together before uploading to codecov etc a single time, I think this is just a mandatory structure. The coverage job must be separated so it can depend on the test matrix. Having checks be a separate matrix is kind of neither here nor there on this topic, I think. And the All job is compensation for a missing GitHub Actions feature. I am thinking of something like this: * A matrix to run jobs with coverage on each environment (os + py combination + extra stuff noipv6, nodeps, etc) * Each job from the matrix will run an initial combine to combine sub-process coverage * Each job will upload once python coverage raw file. * A single job will download those coverage file, combine them once again to generate an XML file to be pushed to codecov.io Sounds good. Sounds like exactly what I suggested. :] Except for maybe misunderstanding about what level of separation is needed for the interjob dependencies. Hopefully this clarifies a bit. I think we actually intend the same thing. Unless I've missed something else? Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On 2021-03-30 17:14, Kyle Altendorf wrote: On 2021-03-30 15:24, Glyph wrote: On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: Hi All, Has anyone contacted GitHub to see if they would be willing to increase the parallelism limit in Actions? My understanding is that we maintain two CI systems (GitHub Actions and Azure Pipelines) for the sake of more parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub would be willing to help out. Not as far as I know. Do you want to give it a shot? That was the plan. Submitted. I'll let you all know. From GitHub: Hi Kyle, Thanks for taking the time to write in. You would need to upgrade the twisted organization account to Enterprise Cloud for a higher concurrent jobs limit. However, the discount on the twisted organization account is only for GitHub Team, so an upgrade would require payment. https://docs.github.com/en/actions/reference/usage-limits-billing-and-administration#usage-limits All the best, Jimmy https://docs.github.com/en/github/getting-started-with-github/githubs-products#github-enterprise https://docs.github.com/en/github/getting-started-with-github/githubs-products#github-team So it seems we already have 60x parallelism and a discount ($4/month per user if we have Teams for free) and at least a few of us didn't realize it? Admittedly macOS remains at 5x until you get to the enterprise level. So, case closed on that I suppose. I'm not sure where the rest of the discussion about just-GitHub-for-CI stands, but that can be separated from thread. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On Thu, 1 Apr 2021 at 13:46, Kyle Altendorf wrote: > On 2021-03-31 05:45, Adi Roiban wrote: > > > On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, wrote: > > > >> On 2021-03-30 15:24, Glyph wrote: > On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: > > Hi All, > > Has anyone contacted GitHub to see if they would be willing to > increase the parallelism limit in Actions? My understanding is that > we maintain two CI systems (GitHub Actions and Azure Pipelines) for > the sake of more parallelism. While perhaps worthwhile, this > doesn't > seem fun. Maybe GitHub would be willing to help out. > >>> > >>> Not as far as I know. Do you want to give it a shot? > >> > >> That was the plan. Submitted. I'll let you all know. > > > > Thanks. > > > > On a related note I am working to reduce the general duration of > > Twisted CI jobs. > > > > This should result in keeping the jobs slot busy for less time and > > allowing more total jobs per day. > > > > The first step was to enable parallel trial tests. Use both CPUs > > available to the CI VMs. > > This was already done for Linux and MacOS. > > > > Distributed trial is not yet supported on Windows, but I am working on > > it. > > I already have a working distributed trial for Windows on my system. > > It needs various fixed and I have submitted smaller PR for review. > > > > Just getting distrial to work on Windows is not all...as we need to > > update and clean the Twsited test to reduce the side effects. > > Linux and macOS are more relaxed in terms of file access and for > > example you can delete a file, even if its opened by another process or > > by the same process. > > Windows is not happy about that. > > > > And then there are the 2 pypy jobs taking 30 minutes each - working to > > fix it here https://github.com/twisted/twisted/pull/1543/ > > > > And then there are also general CI jobs improvements. > > > > For example we can save about 40 seconds if we decide to stop sending > > coverage to coveralls and only use codecov.io > > Another example is save another 1 minute by stop using `tox --notests` > > and replace it with something like > > https://github.com/twisted/python-info-action to show the dependencies. > > With `tox --notest` we now create the wheels and install the > > dependencies twice for each job. > > Would `tox --skip-pkg-install` cut it for the second invocation? > > > https://github.com/ericaltendorf/plotman/pull/66/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR145 > > I don't know. I still don't understand the tox install and test process... and tox is not my friend as every week tox hits me with various errors that I don't understand. Most probably, tox is an advanced tool and I am a poor developer who doesn't know much and sticks to the manual labour of calling venv + pip + trial :) For example, I don't understand why you need to create an sdist and from sdist to create a wheel and then install that wheel inside tox for each tox test environment. Why not create the wheel once and install the same wheel in all environments. > > There is an extra 1 minute required for each job to generate the > > coverage XML file. We can download all the coverage artifacts and > > generate the XML only once. > > I did coverage collection from all jobs for pymodbus. Looks like I > still had the XML report generating in each job, but that could > presumably be shifted. Anyways, I'm sure several of us could do this > off the top of our heads but in case it is of any interest here's what I > did. Or maybe you just point out something silly I did there and should > learn from. :] > > https://github.com/riptideio/pymodbus/pull/592/files > That is one big matrix :) ... I don't understand why you need test and check and coverage, each with a separate matrix. I am thinking of something like this: * A matrix to run jobs with coverage on each environment (os + py combination + extra stuff noipv6, nodeps, etc) * Each job from the matrix will run an initial combine to combine sub-process coverage * Each job will upload once python coverage raw file. * A single job will download those coverage file, combine them once again to generate an XML file to be pushed to codecov.io The maximum jobs time will continue to be the same...as you still need to wait for the slowest env, but the total run time can be reduced by more than 20 minutes as each job will no longer have to spend 2 minutes creating XML and reporting JSON to coveralls. --- Note that coveralls API is different. Instead of receiving an XML file, with coverall there are separate API calls for reporting the coverage in JSON format. This is why coveralls reporting takes so long Also, for coveralls, the python uploader is a 3rd party tool, and not an official tool from the Coveralls team. So maybe the python coveralls uploader tool can be improved to send the reports faster. Cheers -- Adi Roiban
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On 2021-03-31 05:45, Adi Roiban wrote: On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, wrote: On 2021-03-30 15:24, Glyph wrote: On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: Hi All, Has anyone contacted GitHub to see if they would be willing to increase the parallelism limit in Actions? My understanding is that we maintain two CI systems (GitHub Actions and Azure Pipelines) for the sake of more parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub would be willing to help out. Not as far as I know. Do you want to give it a shot? That was the plan. Submitted. I'll let you all know. Thanks. On a related note I am working to reduce the general duration of Twisted CI jobs. This should result in keeping the jobs slot busy for less time and allowing more total jobs per day. The first step was to enable parallel trial tests. Use both CPUs available to the CI VMs. This was already done for Linux and MacOS. Distributed trial is not yet supported on Windows, but I am working on it. I already have a working distributed trial for Windows on my system. It needs various fixed and I have submitted smaller PR for review. Just getting distrial to work on Windows is not all...as we need to update and clean the Twsited test to reduce the side effects. Linux and macOS are more relaxed in terms of file access and for example you can delete a file, even if its opened by another process or by the same process. Windows is not happy about that. And then there are the 2 pypy jobs taking 30 minutes each - working to fix it here https://github.com/twisted/twisted/pull/1543/ And then there are also general CI jobs improvements. For example we can save about 40 seconds if we decide to stop sending coverage to coveralls and only use codecov.io Another example is save another 1 minute by stop using `tox --notests` and replace it with something like https://github.com/twisted/python-info-action to show the dependencies. With `tox --notest` we now create the wheels and install the dependencies twice for each job. Would `tox --skip-pkg-install` cut it for the second invocation? https://github.com/ericaltendorf/plotman/pull/66/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR145 There is an extra 1 minute required for each job to generate the coverage XML file. We can download all the coverage artifacts and generate the XML only once. I did coverage collection from all jobs for pymodbus. Looks like I still had the XML report generating in each job, but that could presumably be shifted. Anyways, I'm sure several of us could do this off the top of our heads but in case it is of any interest here's what I did. Or maybe you just point out something silly I did there and should learn from. :] https://github.com/riptideio/pymodbus/pull/592/files Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On Tue, 30 Mar 2021 at 15:57, Kyle Altendorf wrote: > Hi All, > SNIP > My understanding is that we maintain > two CI systems (GitHub Actions and Azure Pipelines) for the sake of more > parallelism. > SNIP > -kyle > We also have 2 CI systems because migrating the Windows and macOS jobs from Azure Pipeline to GHA takes time. As an experiment I have enabled one Windows job on GitHub and the tests failed... so that needs extra work. To check what happened I got Windows 2019 server on a local VM and tried to execute `trial twisted` ...tests are failing. So for now, I am also not in a hurry to remove Azure Pipelines as it just works. Fixing is not hard but it takes time. For example some Windows tests are skipped only on Azure... and they fail both on Azure and local dev... I guess that a macOS migration is easier as macOS is similar to Linux, and on macOS we only run the default reactor, which I think is the select reactor. But in preparation for a future GitHub to Azure migration, I think that we can have a single Windows and a single macOS job on GitHub Actions, while keeping Azure VMs. In this way we can have them running for a while in trunk and see if there are any surprises . I guess that with GitHub being a MS company GitHub Actions and Azure Pipelines might use the exact same VM images and VM compute nodes. Cheers -- Adi Roiban ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On Tue, 30 Mar 2021, 22:23 Kyle Altendorf, wrote: > > > On 2021-03-30 15:24, Glyph wrote: > >> On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: > >> > >> Hi All, > >> > >> Has anyone contacted GitHub to see if they would be willing to > >> increase the parallelism limit in Actions? My understanding is that > >> we maintain two CI systems (GitHub Actions and Azure Pipelines) for > >> the sake of more parallelism. While perhaps worthwhile, this doesn't > >> seem fun. Maybe GitHub would be willing to help out. > > > > Not as far as I know. Do you want to give it a shot? > > That was the plan. Submitted. I'll let you all know. > Thanks. On a related note I am working to reduce the general duration of Twisted CI jobs. This should result in keeping the jobs slot busy for less time and allowing more total jobs per day. The first step was to enable parallel trial tests. Use both CPUs available to the CI VMs. This was already done for Linux and MacOS. Distributed trial is not yet supported on Windows, but I am working on it. I already have a working distributed trial for Windows on my system. It needs various fixed and I have submitted smaller PR for review. Just getting distrial to work on Windows is not all...as we need to update and clean the Twsited test to reduce the side effects. Linux and macOS are more relaxed in terms of file access and for example you can delete a file, even if its opened by another process or by the same process. Windows is not happy about that. And then there are the 2 pypy jobs taking 30 minutes each - working to fix it here https://github.com/twisted/twisted/pull/1543/ And then there are also general CI jobs improvements. For example we can save about 40 seconds if we decide to stop sending coverage to coveralls and only use codecov.io Another example is save another 1 minute by stop using `tox --notests` and replace it with something like https://github.com/twisted/python-info-action to show the dependencies. With `tox --notest` we now create the wheels and install the dependencies twice for each job. There is an extra 1 minute required for each job to generate the coverage XML file. We can download all the coverage artifacts and generate the XML only once. - Now the actual job duration will differ... I see the pip cache download taking 2 seconds or 2 minutes. But on Linux it takes about 4 minutes to run the tests (inside tox with wheel creation)... and then from 2minutes and above is the extra overhead Cheers > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
On 2021-03-30 15:24, Glyph wrote: On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: Hi All, Has anyone contacted GitHub to see if they would be willing to increase the parallelism limit in Actions? My understanding is that we maintain two CI systems (GitHub Actions and Azure Pipelines) for the sake of more parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub would be willing to help out. Not as far as I know. Do you want to give it a shot? That was the plan. Submitted. I'll let you all know. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] GitHub Actions parallelism limit increase
> On Mar 30, 2021, at 7:57 AM, Kyle Altendorf wrote: > > Hi All, > > Has anyone contacted GitHub to see if they would be willing to increase the > parallelism limit in Actions? My understanding is that we maintain two CI > systems (GitHub Actions and Azure Pipelines) for the sake of more > parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub > would be willing to help out. Not as far as I know. Do you want to give it a shot? -g ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] GitHub Actions parallelism limit increase
Hi All, Has anyone contacted GitHub to see if they would be willing to increase the parallelism limit in Actions? My understanding is that we maintain two CI systems (GitHub Actions and Azure Pipelines) for the sake of more parallelism. While perhaps worthwhile, this doesn't seem fun. Maybe GitHub would be willing to help out. Cheers, -kyle ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
