[DISCUSS] KIP-942: Add Power(ppc64le) support
Hi Team, We need to add an additional job for ppc64le at https://ci-builds.apache.org/job/Kafka/job/kafka/ using a new Jenkins file created specific for ppc64le PR: https://github.com/apache/kafka/pull/13817 We have updated the Jenkinfile.ppc64le with required changes to run builds daily on trunk. But we need help to enable job at https://ci-builds.apache.org/job/Kafka/job/kafka/ specific to ppc64le. Thanks, Vaibhav
RE: [DISCUSS] KIP-942: Add Power(ppc64le) support
Hi Divij, 1.No the different versions of JVM don't work differently for power architecture. It will be sufficient if we just run it with the latest supported JDK (20) + latest supported scala (2.13) 2. The plan has been updated accordingly for rejected alternatives -Original Message- From: Divij Vaidya Sent: Thursday, August 3, 2023 2:33 PM To: dev@kafka.apache.org Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support Hey Vaibhav 1. KIP says "Enable CI for power architecture and run tests with Java 8, 11 and 17 and Scala 2.13". Do different versions of JVM work differently for power architecture? Would it be sufficient if we just run it with the latest supported JDK (20) + latest supported scala (2.13) ? 2. Can you also please add that we plan to run this only on branch builder and not on every PR. Note that we have two CI runs configured today, one is "branch builder" which runs when a commit is merged to trunk or preceding versions and another is "PR builder" which runs on every commit on every PR. From our earlier discussion on this thread, we discussed to only add it for "branch builder". Also, please add option of adding test to "PR builder" in the rejected alternative section. -- Divij Vaidya On Thu, Aug 3, 2023 at 8:40 AM Vaibhav Nazare wrote: > > Hi Divij > > Thanks for the response. Agree with you, also I have updated the KIP > accordingly. >
Re: [DISCUSS] KIP-942: Add Power(ppc64le) support
Hey Vaibhav 1. KIP says "Enable CI for power architecture and run tests with Java 8, 11 and 17 and Scala 2.13". Do different versions of JVM work differently for power architecture? Would it be sufficient if we just run it with the latest supported JDK (20) + latest supported scala (2.13) ? 2. Can you also please add that we plan to run this only on branch builder and not on every PR. Note that we have two CI runs configured today, one is "branch builder" which runs when a commit is merged to trunk or preceding versions and another is "PR builder" which runs on every commit on every PR. From our earlier discussion on this thread, we discussed to only add it for "branch builder". Also, please add option of adding test to "PR builder" in the rejected alternative section. -- Divij Vaidya On Thu, Aug 3, 2023 at 8:40 AM Vaibhav Nazare wrote: > > Hi Divij > > Thanks for the response. Agree with you, also I have updated the KIP > accordingly. >
[DISCUSS] KIP-942: Add Power(ppc64le) support
Hi Divij Thanks for the response. Agree with you, also I have updated the KIP accordingly.
[DISCUSS] KIP-942: Add Power(ppc64le) support
Hi Divij Thanks for the response. Agree with you, also I have updated the KIP accordingly.
Re: [DISCUSS] KIP-942: Add Power(ppc64le) support
Hi Vaibhav 1. Could you please add the details that we discussed here in this thread to the KIP (e.g. why is it safe to add this infra now and which other projects are using it). This would help to have KIP as a source of record if someone wants to read about this decision in future. 2. Given that we merge ~5-6 commits a day to trunk, I would be inclined towards running this infra on every merge to trunk (instead of a daily run as I suggested earlier). It won't be run for every PR though. These are the last two thoughts from my side. Otherwise, it looks good. -- Divij Vaidya On Mon, Jul 10, 2023 at 7:09 PM Vaibhav Nazare wrote: > > Hi Colin and Mickael, > > Yes I agree we don't have to run on every build , may be as mentioned we can > run the job once/twice a day at specific time. > According the test plan the tests which I mentioned are not specific to > ppc64le , I was running tests on the trunk branch which were common across > platforms on that specific commit and I have removed the table and mentioned > the test plan accordingly when to run the jobs. > > > > -Original Message- > From: Mickael Maison > Sent: Tuesday, July 4, 2023 3:12 PM > To: dev@kafka.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support > > Hi, > > Thanks for the KIP! > In the Test Plan section you mentioned a few unit and integration tests are > failing. Are these flaky tests or did you find issues related to ppc64le? If > they are just flaky tests, I think we can remove that table and instead > describe how often we intend to run the tests in the CI. > > Thanks, > Mickael > > On Tue, Jul 4, 2023 at 12:04 AM Colin McCabe wrote: > > > > I agree with Divij. A nightly Apache Kafka build for PowerPC would be > > welcome. But it shouldn't run on every build, since the extra time and > > complexity would not be worth it. > > > > By the way, are there any features or plugins we don't intend to support on > > PPC? If there are, this KIP would be a good place to spell them out. > > > > Naively, I would think all of our Java and Scala code should work on > > PPC without changes. However, there may be library dependencies that > > don't exist on PPC. (We have to remember that the last desktop PowerPC > > chip that an average user could buy shipped in 2005) > > > > best, > > Colin > > > > > > On Mon, Jun 19, 2023, at 23:12, Vaibhav Nazare wrote: > > > Thank you for response Divij. > > > > > > 1. We are going to use ASF infra provided nodes for better > > > availability and stability as there are 3 power9 nodes managed > > > officially by ASF infra team themselves. > > > Ref: > > > INVALID URI REMOVED > > > rg_jira_browse_INFRA-2D24663=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I > > > 3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11 > > > WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=EeWMMaaUqDLlqUjJmw3_ouI9lt > > > 4B_EUGyH8OZ_VTIRE= > > > INVALID URI REMOVED. > > > apache.org_view_Shared-2520-2D-2520ppc64le-2520nodes_=DwIFaQ=jf_ > > > iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw= > > > wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=O > > > R-kg7uZdehAVxU7TdCUruDpGb0yOAEa0wW2dC3P3yU= > > > previously used power node details for apache/kafka CI: > > > RAM- 16GB > > > VCPUs- 8 VCPU > > > Disk- 160GB > > > for shared VMs we need to check with ASF infra team to provide > > > details > > > > > > 2. We can run nightly builds once or twice in a day on specific > > > period of time instead of every commit 3. apache/camel > > > INVALID URI REMOVED > > > rg_job_Camel_job_el_=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=J35xYPMn5tCJRn91aARKgyiwas_cu4RGeSsh50tV4GI= > > > has already enabled CI for power platform they are using same H/W > > > resources as > > > RAM- 16GB > > > VCPUs- 8 VCPU > > > Disk- 160GB > > > > > > -Original Message- > > > From: Divij Vaidya > > > Sent: Monday, June 19, 2023 10:20 PM > > > To: dev@kafka.apache.org > > > Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) > > > support > > > > > > Thank you for the KIP Vaibhav. > > > > > > 1. Builds for power architecture were intentionally disabled in the > > > past since the infrastructure was flaky [1]. Could you please add to > >
RE: [DISCUSS] KIP-942: Add Power(ppc64le) support
Hi Colin and Mickael, Yes I agree we don't have to run on every build , may be as mentioned we can run the job once/twice a day at specific time. According the test plan the tests which I mentioned are not specific to ppc64le , I was running tests on the trunk branch which were common across platforms on that specific commit and I have removed the table and mentioned the test plan accordingly when to run the jobs. -Original Message- From: Mickael Maison Sent: Tuesday, July 4, 2023 3:12 PM To: dev@kafka.apache.org Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support Hi, Thanks for the KIP! In the Test Plan section you mentioned a few unit and integration tests are failing. Are these flaky tests or did you find issues related to ppc64le? If they are just flaky tests, I think we can remove that table and instead describe how often we intend to run the tests in the CI. Thanks, Mickael On Tue, Jul 4, 2023 at 12:04 AM Colin McCabe wrote: > > I agree with Divij. A nightly Apache Kafka build for PowerPC would be > welcome. But it shouldn't run on every build, since the extra time and > complexity would not be worth it. > > By the way, are there any features or plugins we don't intend to support on > PPC? If there are, this KIP would be a good place to spell them out. > > Naively, I would think all of our Java and Scala code should work on > PPC without changes. However, there may be library dependencies that > don't exist on PPC. (We have to remember that the last desktop PowerPC > chip that an average user could buy shipped in 2005) > > best, > Colin > > > On Mon, Jun 19, 2023, at 23:12, Vaibhav Nazare wrote: > > Thank you for response Divij. > > > > 1. We are going to use ASF infra provided nodes for better > > availability and stability as there are 3 power9 nodes managed > > officially by ASF infra team themselves. > > Ref: > > INVALID URI REMOVED > > rg_jira_browse_INFRA-2D24663=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I > > 3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11 > > WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=EeWMMaaUqDLlqUjJmw3_ouI9lt > > 4B_EUGyH8OZ_VTIRE= > > INVALID URI REMOVED. > > apache.org_view_Shared-2520-2D-2520ppc64le-2520nodes_=DwIFaQ=jf_ > > iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw= > > wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=O > > R-kg7uZdehAVxU7TdCUruDpGb0yOAEa0wW2dC3P3yU= > > previously used power node details for apache/kafka CI: > > RAM- 16GB > > VCPUs- 8 VCPU > > Disk- 160GB > > for shared VMs we need to check with ASF infra team to provide > > details > > > > 2. We can run nightly builds once or twice in a day on specific > > period of time instead of every commit 3. apache/camel > > INVALID URI REMOVED > > rg_job_Camel_job_el_=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH78J2VuRdvseBKrVLuXW9PtUJqs9BWO4=J35xYPMn5tCJRn91aARKgyiwas_cu4RGeSsh50tV4GI= > > has already enabled CI for power platform they are using same H/W > > resources as > > RAM- 16GB > > VCPUs- 8 VCPU > > Disk- 160GB > > > > -Original Message- > > From: Divij Vaidya > > Sent: Monday, June 19, 2023 10:20 PM > > To: dev@kafka.apache.org > > Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) > > support > > > > Thank you for the KIP Vaibhav. > > > > 1. Builds for power architecture were intentionally disabled in the > > past since the infrastructure was flaky [1]. Could you please add to > > the KIP on what has changed since then? > > 2. What do you think about an alternative solution where we run a > > nightly build for this platform instead of running the CI with every > > PR/commit? > > 3. To bolster the case for this KIP, could you please add > > information from other Apache projects who are already running CI > > for this platform? Is their CI stable on Apache Infra hosts? > > > > > > [1] > > INVALID URI REMOVED > > he_kafka_pull_12380=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHA > > urpHrTUoOkX8ByFHVUGD0XU1PTKfCiw=wok2fkRzJGACiUg-o6hyaJP11WLbfqYbH7 > > 8J2VuRdvseBKrVLuXW9PtUJqs9BWO4=Wo7q9L9UOHo8AkRRwIFG9570Y4isqaxmhIY > > W0wwVspk= > > > > -- > > Divij Vaidya > > > > > > > > On Mon, Jun 19, 2023 at 12:30 PM Vaibhav Nazare > > wrote: > > > >> > >> INVALID URI REMOVED > >> confluence_display_KAFKA_KIP-2D942-253A-2BAdd-2BPower-2528ppc64le-2 > >> 529 > >> -2Bsupport=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUo > >> OkX > >> 8ByFHVUGD0XU1PTKfCiw=z6ZZ_vt5XP--aKB5lpRRZxdVMA37hD_0ch7COCLdMtLh > >> Mve > >> 8AJcbKfwRtBac267r=BQtj2lbWlu32mK0TP37XeZanal33QOf5HB1-33QJIqc= > >>
Re: [DISCUSS] KIP-942: Add Power(ppc64le) support
Hi, Thanks for the KIP! In the Test Plan section you mentioned a few unit and integration tests are failing. Are these flaky tests or did you find issues related to ppc64le? If they are just flaky tests, I think we can remove that table and instead describe how often we intend to run the tests in the CI. Thanks, Mickael On Tue, Jul 4, 2023 at 12:04 AM Colin McCabe wrote: > > I agree with Divij. A nightly Apache Kafka build for PowerPC would be > welcome. But it shouldn't run on every build, since the extra time and > complexity would not be worth it. > > By the way, are there any features or plugins we don't intend to support on > PPC? If there are, this KIP would be a good place to spell them out. > > Naively, I would think all of our Java and Scala code should work on PPC > without changes. However, there may be library dependencies that don't exist > on PPC. (We have to remember that the last desktop PowerPC chip that an > average user could buy shipped in 2005) > > best, > Colin > > > On Mon, Jun 19, 2023, at 23:12, Vaibhav Nazare wrote: > > Thank you for response Divij. > > > > 1. We are going to use ASF infra provided nodes for better availability > > and stability as there are 3 power9 nodes managed officially by ASF > > infra team themselves. > > Ref: https://issues.apache.org/jira/browse/INFRA-24663 > > https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/ > > previously used power node details for apache/kafka CI: > > RAM- 16GB > > VCPUs- 8 VCPU > > Disk- 160GB > > for shared VMs we need to check with ASF infra team to provide details > > > > 2. We can run nightly builds once or twice in a day on specific period > > of time instead of every commit > > 3. apache/camel https://builds.apache.org/job/Camel/job/el/ has already > > enabled CI for power platform they are using same H/W resources as > > RAM- 16GB > > VCPUs- 8 VCPU > > Disk- 160GB > > > > -Original Message- > > From: Divij Vaidya > > Sent: Monday, June 19, 2023 10:20 PM > > To: dev@kafka.apache.org > > Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support > > > > Thank you for the KIP Vaibhav. > > > > 1. Builds for power architecture were intentionally disabled in the > > past since the infrastructure was flaky [1]. Could you please add to > > the KIP on what has changed since then? > > 2. What do you think about an alternative solution where we run a > > nightly build for this platform instead of running the CI with every > > PR/commit? > > 3. To bolster the case for this KIP, could you please add information > > from other Apache projects who are already running CI for this > > platform? Is their CI stable on Apache Infra hosts? > > > > > > [1] https://github.com/apache/kafka/pull/12380 > > > > -- > > Divij Vaidya > > > > > > > > On Mon, Jun 19, 2023 at 12:30 PM Vaibhav Nazare > > wrote: > > > >> > >> INVALID URI REMOVED > >> confluence_display_KAFKA_KIP-2D942-253A-2BAdd-2BPower-2528ppc64le-2529 > >> -2Bsupport=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX > >> 8ByFHVUGD0XU1PTKfCiw=z6ZZ_vt5XP--aKB5lpRRZxdVMA37hD_0ch7COCLdMtLhMve > >> 8AJcbKfwRtBac267r=BQtj2lbWlu32mK0TP37XeZanal33QOf5HB1-33QJIqc= > >>
Re: [DISCUSS] KIP-942: Add Power(ppc64le) support
I agree with Divij. A nightly Apache Kafka build for PowerPC would be welcome. But it shouldn't run on every build, since the extra time and complexity would not be worth it. By the way, are there any features or plugins we don't intend to support on PPC? If there are, this KIP would be a good place to spell them out. Naively, I would think all of our Java and Scala code should work on PPC without changes. However, there may be library dependencies that don't exist on PPC. (We have to remember that the last desktop PowerPC chip that an average user could buy shipped in 2005) best, Colin On Mon, Jun 19, 2023, at 23:12, Vaibhav Nazare wrote: > Thank you for response Divij. > > 1. We are going to use ASF infra provided nodes for better availability > and stability as there are 3 power9 nodes managed officially by ASF > infra team themselves. > Ref: https://issues.apache.org/jira/browse/INFRA-24663 > https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/ > previously used power node details for apache/kafka CI: > RAM- 16GB > VCPUs- 8 VCPU > Disk- 160GB > for shared VMs we need to check with ASF infra team to provide details > > 2. We can run nightly builds once or twice in a day on specific period > of time instead of every commit > 3. apache/camel https://builds.apache.org/job/Camel/job/el/ has already > enabled CI for power platform they are using same H/W resources as > RAM- 16GB > VCPUs- 8 VCPU > Disk- 160GB > > -Original Message- > From: Divij Vaidya > Sent: Monday, June 19, 2023 10:20 PM > To: dev@kafka.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support > > Thank you for the KIP Vaibhav. > > 1. Builds for power architecture were intentionally disabled in the > past since the infrastructure was flaky [1]. Could you please add to > the KIP on what has changed since then? > 2. What do you think about an alternative solution where we run a > nightly build for this platform instead of running the CI with every > PR/commit? > 3. To bolster the case for this KIP, could you please add information > from other Apache projects who are already running CI for this > platform? Is their CI stable on Apache Infra hosts? > > > [1] https://github.com/apache/kafka/pull/12380 > > -- > Divij Vaidya > > > > On Mon, Jun 19, 2023 at 12:30 PM Vaibhav Nazare > wrote: > >> >> INVALID URI REMOVED >> confluence_display_KAFKA_KIP-2D942-253A-2BAdd-2BPower-2528ppc64le-2529 >> -2Bsupport=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX >> 8ByFHVUGD0XU1PTKfCiw=z6ZZ_vt5XP--aKB5lpRRZxdVMA37hD_0ch7COCLdMtLhMve >> 8AJcbKfwRtBac267r=BQtj2lbWlu32mK0TP37XeZanal33QOf5HB1-33QJIqc= >>
Re: [DISCUSS] KIP-942: Add Power(ppc64le) support
Thank you for response Divij. 1. We are going to use ASF infra provided nodes for better availability and stability as there are 3 power9 nodes managed officially by ASF infra team themselves. Ref: https://issues.apache.org/jira/browse/INFRA-24663 https://jenkins-ccos.apache.org/view/Shared%20-%20ppc64le%20nodes/ previously used power node details for apache/kafka CI: RAM- 16GB VCPUs- 8 VCPU Disk- 160GB for shared VMs we need to check with ASF infra team to provide details 2. We can run nightly builds once or twice in a day on specific period of time instead of every commit 3. apache/camel https://builds.apache.org/job/Camel/job/el/ has already enabled CI for power platform they are using same H/W resources as RAM- 16GB VCPUs- 8 VCPU Disk- 160GB -Original Message- From: Divij Vaidya Sent: Monday, June 19, 2023 10:20 PM To: dev@kafka.apache.org Subject: [EXTERNAL] Re: [DISCUSS] KIP-942: Add Power(ppc64le) support Thank you for the KIP Vaibhav. 1. Builds for power architecture were intentionally disabled in the past since the infrastructure was flaky [1]. Could you please add to the KIP on what has changed since then? 2. What do you think about an alternative solution where we run a nightly build for this platform instead of running the CI with every PR/commit? 3. To bolster the case for this KIP, could you please add information from other Apache projects who are already running CI for this platform? Is their CI stable on Apache Infra hosts? [1] https://github.com/apache/kafka/pull/12380 -- Divij Vaidya On Mon, Jun 19, 2023 at 12:30 PM Vaibhav Nazare wrote: > > INVALID URI REMOVED > confluence_display_KAFKA_KIP-2D942-253A-2BAdd-2BPower-2528ppc64le-2529 > -2Bsupport=DwIFaQ=jf_iaSHvJObTbx-siA1ZOg=s9I3h_d72lHAurpHrTUoOkX > 8ByFHVUGD0XU1PTKfCiw=z6ZZ_vt5XP--aKB5lpRRZxdVMA37hD_0ch7COCLdMtLhMve > 8AJcbKfwRtBac267r=BQtj2lbWlu32mK0TP37XeZanal33QOf5HB1-33QJIqc= >
Re: [DISCUSS] KIP-942: Add Power(ppc64le) support
Thank you for the KIP Vaibhav. 1. Builds for power architecture were intentionally disabled in the past since the infrastructure was flaky [1]. Could you please add to the KIP on what has changed since then? 2. What do you think about an alternative solution where we run a nightly build for this platform instead of running the CI with every PR/commit? 3. To bolster the case for this KIP, could you please add information from other Apache projects who are already running CI for this platform? Is their CI stable on Apache Infra hosts? [1] https://github.com/apache/kafka/pull/12380 -- Divij Vaidya On Mon, Jun 19, 2023 at 12:30 PM Vaibhav Nazare wrote: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-942%3A+Add+Power%28ppc64le%29+support >
[DISCUSS] KIP-942: Add Power(ppc64le) support
https://cwiki.apache.org/confluence/display/KAFKA/KIP-942%3A+Add+Power%28ppc64le%29+support
[DISCUSS] KIP-942: Add Power(ppc64le) support
https://cwiki.apache.org/confluence/display/KAFKA/KIP-942%3A+Add+Power%28ppc64le%29+support
