Re: [DISCUSSION] Deprecation of YuniKorn plugin deployment mode

[Weiwei Yang]

Hi Craig

Thanks for the write-up and it clearly explains the rationale behind this.
I appreciate you listing all these things in detail to help all community
members understand
in depth about the issue. Thank you!

I agree with all the points you listed. My concern is from another angle,
though the plugin mode is not as elegant as the standalone mode, it works
and
serves the purpose of bringing yunikorn values without breaking existing
things. This helps some
users that run mixed things on large existing K8s clusters. Removing this
feature means the users
have to migrate which can be risky for production.

Dig a little more into the "migration concerns" section, if we suggest the
users migrate from the plugin to
standalone mode, what are the implications to the existing workloads
running with the default scheduler?
I checked the scheduler configuration
, and
theoretically, I think we can support most of them, but I am not sure
if we can support the plugins that implement multiple extension points.
Should we have a document to explain
the compatibility of the standalone yunikorn scheduler vs the default
scheduler, and potential limitations if any,
to help the existing plugin mode users better understand the impact of
migration?

Another thing is I want to understand all the concerns if we want to keep
this feature, I think you made it clear about
1. Maintenance effort - make sure the quality of the plugin mode same level
as the standalone mode
2. Complexity of the e2e tests
3. Deeper dependency on the Kubernetes private APIs
I want to understand more about #3, since now we are leveraging scheduling
framework APIs, is this still an outstanding issue?

Hope this makes sense.
Weiwei





On Tue, Apr 9, 2024 at 12:03 PM Peter Bacsko  wrote:

> Hi all,
>
> thanks Craig for writing this excellent, detailed summary, including
> historical context.
>
> As we already talked about it on Slack, I'm definitely +1 for removing the
> plugin. My main gripes are:
>
> 1. It overcomplicates the codebase. Two branches for plugin vs non-plugin
> mode, scheduler cache tracks way too much state because of it (
> SchedulerCache: pendingAllocations, inProgressAllocations, schedulingTasks,
> taskBloomFilterRef; cache.Task: schedulingState).This adds significant
> complexity to the code, making maintainability and debugging difficult.
>
> 2. Some issues can only be solved with completely ugly hacks:
> a) Two scheduler caches can become out-of-sync and detecting this is a real
> challenge. We can run a goroutine which checks whether Yunikorn scheduled
> pods are indeed scheduled, but what if they aren't? Then these pods must be
> invalidated inside Yunikorn. We can remove and add it again which just
> feels wrong. We can add new code for it, which again, just complicates the
> code base. A shared scheduler cache would require a lot of copy-paste code
> so we can actually access it from the default scheduler. We probably need
> to create an intermediate layer which adapts one type to another. I can't
> even estimate how much effort this possibly could be.
>
> b) The extra K8s API call to activate Unschedulable pods (place them to the
> activeQ again). If there's no scheduling decision for a given pod, we mark
> it Unschedulable, but we need to re-trigger scheduling as soon as Yunikorn
> selects a target node. So we need to update the pod with a network round
> trip call to do this... Or we can retrieve the activeQ instance, which
> requires some copy-pasting from the existing kube-scheduler so we can
> obtain the reference. This also means that from time to time, we need to
> see how much our copy-pasted code deviates from the actual K8s code.
>
> 3. I believe all e2e tests runs on a single machine inside VMs/containers.
> So having a smaller test matrix means faster e2e test execution
> (hopefully).
>
> Suggested timeline looks very reasonable to me.
>
> Thanks,
> Peter
>
> On Tue, Apr 9, 2024 at 5:26 PM Craig Condit  wrote:
>
> > ---
> > Preface
> > ===
> >
> > There have been quite a few informal discussions on this topic, and it's
> > time that we bring this formally to the yunikorn-dev mailing list for
> > further discussion...
> >
> >
> > ---
> > Summary
> > ===
> >
> > We are actively planning to deprecate the YuniKorn plugin mode for
> > eventual removal. This has been an experimental feature since YuniKorn
> > 1.0.0, but has not proven to be as stable or performant as our default
> > deployment mode. Additionally, it has proven to be a large maintenance
> > burden -- even for contributors who do not actively use it.
> >
> >
> > ---
> > History
> > ===
> >
> > To adequately explain the current situation and why this is being
> planned,
> > it's helpful to understand some of the history of both Kubernetes and
> > YuniKorn and how they interact.
> >
> > Approximately three years ago, the Kubernetes community decided to
> > implement an internal 

Re: [ANNOUNCE] Apache YuniKorn v1.5.0 release

[Weiwei Yang]

Congrats on the new release!
Thank you for the hard work Tingyao!

On Thu, Mar 14, 2024 at 9:47 AM TingYao  wrote:

> Hi all,
>
> It gives me great pleasure to announce that the Apache YuniKorn community
> has
> voted to release Apache YuniKorn v1.5.0.
>
> It contains 219 fixes and improvements. The release details, list of
> major features and incompatible changes are on the v1.5.0 announcement
> page [1].
>
> You can also download the release from the Downloads page [2],
>
> Many thanks to everyone who contributed to the release.
>
> Tingyao
>
> [1] https://yunikorn.apache.org/release-announce/1.5.0
> [2] https://yunikorn.apache.org/community/download/
>

Re: Should we remove Chinese documents

[Weiwei Yang]

Good points, thanks Chia-Ping for bringing this up.
I can see the difficulties of keeping the 2 versions of docs up-to-date, I
am +1 on removing the Chinese version.


On Fri, Feb 9, 2024 at 8:26 AM Craig Condit  wrote:

> As a fellow non-Chinese speaker I’ll defer to others on the decision
> whether to stop providing Chinese docs. *If* we decide to stop, I’d be in
> favor of removing the existing docs due to staleness (as Chia-Ping Tsai
> indicated). We won’t lose them permanently; there is still git history if
> we would ever change our minds (such as if our Chinese-speaking developer
> base grows and the docs become sustainable to maintain).
>
> Craig
>
>
> > On Feb 9, 2024, at 7:56 AM, Chia-Ping Tsai  wrote:
> >
> > hi Wilfred
> >
> >> If the decision is to stop: do we remove current content or let it age
> out?
> >
> > Personally, it should be removed totally since users will get
> misdirected by the out-of-date docs.
> >
> > On 2024/02/09 07:15:41 Wilfred Spiegelenburg wrote:
> >> As a non-Chinese speaker I am leaving that decision to our Chinese
> >> speaking community members.
> >>
> >> Would be good to get some feedback on the following point:
> >> If the decision is to stop: do we remove current content or let it age
> out?
> >>
> >> Wilfred
> >>
> >>
> >>
> >> On Fri, 9 Feb 2024 at 17:59, Chia-Ping Tsai  wrote:
> >>>
> >>> Dear all,
> >>>
> >>> We have a topic in slack (
> https://yunikornworkspace.slack.com/archives/CL9CRJ1KM/p1707414644039739)
> to discuss the future of Chinese documents.
> >>>
> >>> As a YK developer, I’d like to get rid of Chinese documents because it
> is hard to make it up-to-date. Also, we ought to focus on other more
> important features/fixes.
> >>>
> >>> As a (native) Chinese speaker, I’d like to get rid of Chinese
> documents because the translation tools are good enough to help me to read
> the English documents. Also, there are some native-Chinese developers in
> the YK community, and hence I can have discussion with them by Chinese.
> >>>
> >>> As a YK committer, I’d like to see more feedbacks from the community
> before making this difficult decision. Hence, PLEASE feel free to raise
> objection to this proposal via mail or slack.
> >>>
> >>>
> >>> --
> >>> Happy Lunar New Year
> >>> Chia-Ping Tsai (蔡嘉平)
> >>> -
> >>> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> >>> For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >>>
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> >> For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >>
> >>
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

[jira] [Created] (YUNIKORN-2271) Incorrect handling of GPU only resources

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-2271:
-

 Summary: Incorrect handling of GPU only resources
 Key: YUNIKORN-2271
 URL: https://issues.apache.org/jira/browse/YUNIKORN-2271
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: shim - kubernetes
Reporter: Weiwei Yang


https://github.com/apache/yunikorn-k8shim/blob/a118ba6c4d84804e2a407f9d91196ece4690cf09/pkg/common/resource.go#L61-L63
 this code seems to have a bug. When I define resource like this:

{code}
request:
  nvidia.com/gpu: 1
limit:
  nvidia.com/gpu: 1
 {code}

this is considered as QoS best effort and returned with just

{code}
Resources:
  pod:1
{code}

but I think this is a valid configuration that a pod only specifies GPU 
resource without memory or CPU. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-2270) GPU Preemption is not triggered as expected when all available GPUs are used

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-2270:
-

 Summary: GPU Preemption is not triggered as expected when all 
available GPUs are used
 Key: YUNIKORN-2270
 URL: https://issues.apache.org/jira/browse/YUNIKORN-2270
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: core - scheduler
Reporter: Weiwei Yang


I am testing an important scenario of preemption for GPU. The design a scenario 
is like the following:

queue structure is pretty simple:

{code}
root.a (min=100, max=300)
root.b (min=0, max=300)
{code}

the cluster has a total of 300 GPUs available, no autoscaling. Reproducing 
steps:

1. Create 600 pods in root.b queue, each needs 1 GPU. This will consume all 300 
GPUs available in the cluster, and 300 pods pending
2. Create 100 pods in root.a queue, each needs 1 GPU. The expectation is queue 
a will preempt 100 GPU from queue b reach the guarantee. 

observation: a small number of pods preempted resources from queue b got 
started on queue a, the result is not stable. it could not reach guaranteed 
resources. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-2255) Minor updates on who-we-are page

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-2255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-2255.
---
Fix Version/s: 1.5.0
   Resolution: Fixed

> Minor updates on who-we-are page
> 
>
> Key: YUNIKORN-2255
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2255
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: documentation
>    Reporter: Weiwei Yang
>    Assignee: Weiwei Yang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.5.0
>
>
> Minor updates for Junping, changing the organization to Datastrato 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-2255) Minor updates on who-we-are page

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-2255:
-

 Summary: Minor updates on who-we-are page
 Key: YUNIKORN-2255
 URL: https://issues.apache.org/jira/browse/YUNIKORN-2255
 Project: Apache YuniKorn
  Issue Type: Task
  Components: documentation
Reporter: Weiwei Yang


Minor updates for Junping, changing the organization to Datastrato 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [DISCUSS] YuniKorn community sync: Mandarin

[Weiwei Yang]

I used to run that meeting for quite a while, and it was a good series of
meetings to meet with APAC folks whose first language is Mandarin. I don't
think I have the bandwidth to run that meeting anytime soon, nor do I
feel the need to have this series of meetings besides the English one. I'd
suggest removing it from the calendar. unless someone from PMC has a strong
intention to host it.

Thanks for bringing this up for discussion.
Weiwei

On Sun, Dec 3, 2023 at 6:45 PM Wilfred Spiegelenburg 
wrote:

> Hi,
>
> We have had a Mandarin sync on the calendar for a long time. These
> syncs have not been held in months. Probably not since we graduated as
> a TLP. We currently have no one from the PMC or even one of the
> committers to run it.
>
> Do we still need to keep it on the calendar or can we suffice with the
> English sync only? There has been a good turn out to the English
> language sync as it is an APAC friendly time.
> If we keep it:
> * How would we get feedback on things that were brought up in that
> meeting to the rest of the community that do not speak Mandarin?
> * Is someone who speaks Mandarin prepared to step up and lead the
> sync. I can host the sync on zoom but I cannot lead the sync.
> * Is the current once per month enough, if we resurrect this sync?
>
> Wilfred
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [ANNOUNCE] New PMC member: Rainie Li

[Weiwei Yang]

Hi Rainie, Congrats!
Look forward to more contributions from you!

On Thu, Nov 16, 2023 at 9:07 AM Wei-Chiu Chuang  wrote:

> Congratulations!!
>
> On Thu, Nov 16, 2023 at 8:37 AM Desai, Mit 
> wrote:
>
> > Congratulations Rainie! Very well deserved.
> >
> > -Mit
> >
> > On 11/15/23, 6:21 PM, "Wilfred Spiegelenburg"  > > wrote:
> >
> >
> > The Project Management Committee (PMC) for Apache YuniKorn has invited
> > Rainie Li to become a PMC member and we are pleased to announce
> > that she has accepted.
> >
> >
> > On behalf of the Apache YuniKorn PMC
> > Wilfred
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org  > dev-unsubscr...@yunikorn.apache.org>
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org  > dev-h...@yunikorn.apache.org>
> >
> >
> >
> >
> >
> >
>

Re: [VOTE] Release Apache YuniKorn 1.4.0 RC1

[Weiwei Yang]

+1

- Build images from source with go 1.21 on arm64
- Verified the image names are correct
- Run the scheduler with a local K8s cluster, run a few examples and worked
fine
- Verified the REST APIs outputs
- Reviewed README, NOTICE and LICENSE files


On Wed, Nov 15, 2023 at 1:42 AM Wilfred Spiegelenburg 
wrote:

> Hello everyone,
>
> I would like to call a vote for releasing Apache YuniKorn 1.4.0 RC1.
> It is a large release with 250+ jiras included.
> Please note that K8s v1.23 and earlier are no longer supported.
>
> The release artefacts have been uploaded here:
>   https://dist.apache.org/repos/dist/dev/yunikorn/1.4.0-RC1/
>
> My public key is located in the KEYS file:
>   https://downloads.apache.org//yunikorn/KEYS
>
> JIRA issues that have been resolved in this release:
>   https://issues.apache.org/jira/issues/?filter=12352769
>
> The release contains a number of incompatible changes that could
> impact the release verification. Please read the draft release notes
> attached to this vote for further details.
>
> Git tags for each component are as follows:
> yunikorn-scheduler-interface: v1.4.0-1
> yunikorn-core: v1.4.0-1
> yunikorn-k8shim: v1.4.0-2
> yunikorn-web: v1.4.0-2
> yunikorn-release: v1.4.0-1
>
> Once the release is voted on and approved, all repos will be tagged
> 1.4.0 for consistency.
>
> Please review and vote. The vote will be open for at least 72 hours
> and closes on Saturday 18 November 2023, 10:00:00 UTC
>
> [ ] +1 Approve
> [ ] +0 No opinion
> [ ] -1 Disapprove (and the reason why)
>
>
> Thank you,
> Wilfred
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [DISCUSS] removal of the yunikorn application CRD

[Weiwei Yang]

Remove YuniKorn application CRD should be fine, I don't think we ever
reached the GA for this feature, and I don't think anyone is using it.
The operator plugin, however, I don't think we should remove, for example
the Spark CRD plugin

provides
the hook to YK in order to update the Spark job status more consistently.

Weiwei

On Fri, Jul 28, 2023 at 12:06 PM Craig Condit  wrote:

> I’m definitely in favor of removing the CRD, and sooner rather than later.
> It negatively impacts some of the ongoing refactoring tasks as it
> influences the recovery pipeline. I think given the alpha state of the CRD,
> if there are no objections we can remove this in 1.4.0.
>
> Craig
>
>
> > On Jul 27, 2023, at 9:41 PM, Wilfred Spiegelenburg 
> wrote:
> >
> > We added the YuniKorn application CRD via YUNIKORN-170 [1]. The idea
> > was back then that we used the CRD to implement gang scheduling.
> > During the later design of gang scheduling we completely stepped away
> > from using the CRD as the basis for gang scheduling. Some of the other
> > advantages that we were expecting from the CRD were never observed
> > (finished state for an application, one point to manage pods). The
> > Spark CRD integration was mostly reversed [2] in favour of normal pod
> > handling due to issues.
> > The second phase for the application CRD YUNIKORN-599 [3] was never
> > started due to the limited advantages we expected to get.
> >
> > There have been no changes in the code or jiras logged against the CRD
> > for two years besides making the build work on later K8s versions [4]
> > The current usage of the application CRD is limited to the
> > TaskGrooupDefinition being used for gang scheduling.
> >
> > Based on all this I would like to start the discussion on removing the
> > application CRD from YuniKorn. Frank Yang has looked at the changes
> > needed to remove the CRD and the impact on the code for the K8shim. A
> > commit with all the changes can be seen in his repo [5] to reference.
> > The change will remove almost 3,000 lines of code just from the K8shim
> > repository. There will be some further changes needed to clean up the
> > build (K8shim) and helmchart (release). Those changes will be removal
> > of scripts and code only.
> >
> > Before we progress with this further we would like to know:
> > * If the application CRD is used by anyone.
> > * If it is used, what part(s) of the CRD are used and what is it used
> for?
> > * Is removing the CRD in YuniKorn 1.4.0 OK or do we need to push to a
> > later release.
> >
> > Objections, comments please let us know.
> >
> > Thank you,
> > Wilfred
> >
> > [1] https://issues.apache.org/jira/browse/YUNIKORN-170
> > [2] https://issues.apache.org/jira/browse/YUNIKORN-643
> > [3] https://issues.apache.org/jira/browse/YUNIKORN-599
> > [4]
> https://github.com/apache/yunikorn-k8shim/commits/master/pkg/apis/yunikorn.apache.org
> > [5]
> https://github.com/FrankYang0529/yunikorn-k8shim/commit/354248b3c24accd679ff2d5a557c599123a58408
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [ANNOUNCE] New PMC member: YuTen Chen

[Weiwei Yang]

Congrats, Yu Teng!

On Mon, Jun 5, 2023 at 10:17 PM Qi Zhu  wrote:

> Congrats!
>
> On 2023/06/06 02:10:29 Wilfred Spiegelenburg wrote:
> > The Project Management Committee (PMC) for Apache YuniKorn has invited
> > YuTen Chen to become a PMC member and we are pleased to announce
> > that he has accepted.
> >
> > On behalf of the Apache YuniKorn PMC
> > Wilfred
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [ANNOUNCE] New committer: Qi Zhu

[Weiwei Yang]

Congrats, Qi!

On Mon, Jun 5, 2023 at 10:22 PM Qi Zhu  wrote:

> Thanks Project Management Committee (PMC) for Apache YuniKorn, i will
> continue contributing more value to the Apache YuniKorn project!
>
> BR
> Qi Zhu
>
> On 2023/06/06 04:45:34 Wilfred Spiegelenburg wrote:
> > The Project Management Committee (PMC) for Apache YuniKorn has invited
> > Qi Zhu Li to become a committer and we are pleased to announce that he
> > has accepted.
> > Please join me in congratulating him.
> >
> > Congratulations & Welcome aboard Qi !
> >
> > Wilfred
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

[jira] [Created] (YUNIKORN-1642) Scheduler recovery failed due to listing operation timeout

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1642:
-

 Summary: Scheduler recovery failed due to listing operation timeout
 Key: YUNIKORN-1642
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1642
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: shim - kubernetes
Reporter: Weiwei Yang


The listing operation in the recovery phase: 
https://github.com/apache/yunikorn-k8shim/blob/c25ac60ffbc175c4966f917da21d184f34dea7b4/pkg/client/apifactory.go#L225.
 This could sometimes fail on some large clusters, the response time from API 
server is not guaranteed. And we see logs like this

{noformat}
2023-03-16T07:00:46.181ZWARNclient/apifactory.go:218Failed 
to sync informers{"error": "timeout waiting for condition"}
2023-03-16T07:00:46.182ZINFOgeneral/general.go:344  Pod list 
retrieved from api server  {"nr of pods": 0}
2023-03-16T07:00:46.182ZINFOgeneral/general.go:365  Application 
recovery statistics {"nr of recoverable apps": 0, "nr of total pods": 0, "nr of 
pods without application metadata": 0, "nr of pods to be recovered": 0}
I0316 07:00:51.319100   1 trace.go:205] Trace[140954425]: "Reflector 
ListAndWatch" 
name:pkg/mod/k8s.io/client-go@v0.20.11/tools/cache/reflector.go:167 
(16-Mar-2023 07:00:16.168) (total time: 35150ms):
{noformat}

Since it is a WARN, it continues but the informers did not return anything. 
This confuses the scheduler that nothing needs to be recovered, and it goes 
ahead doing the scheduling. This causes subsequential scheduler failures.  And 
eventually, nothing can be scheduled anymore.

This should be a FATAL error. So the scheduler can be restarted to retry 
recoverying.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1414) Adding Chinese translations of Sorting Policies

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1414.
---
Target Version: 1.3.0
Resolution: Fixed

> Adding Chinese translations of  Sorting Policies 
> -
>
> Key: YUNIKORN-1414
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1414
> Project: Apache YuniKorn
>  Issue Type: Task
>Reporter: Chen Yu Teng
>Assignee: Chen, Kai-Chun
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.3.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Reopened] (YUNIKORN-1414) Adding Chinese translations of Sorting Policies

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang reopened YUNIKORN-1414:
---

> Adding Chinese translations of  Sorting Policies 
> -
>
> Key: YUNIKORN-1414
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1414
> Project: Apache YuniKorn
>  Issue Type: Task
>Reporter: Chen Yu Teng
>Assignee: Chen, Kai-Chun
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.3.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1414) Adding Chinese translations of Sorting Policies

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1414.
---
Fix Version/s: 1.3.0
   Resolution: Fixed

> Adding Chinese translations of  Sorting Policies 
> -
>
> Key: YUNIKORN-1414
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1414
> Project: Apache YuniKorn
>  Issue Type: Task
>Reporter: Chen Yu Teng
>Assignee: Chen, Kai-Chun
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.3.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1512) Adding Chinese translation of Translation

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1512.
---
Fix Version/s: 1.3.0
   Resolution: Fixed

> Adding Chinese translation of Translation
> -
>
> Key: YUNIKORN-1512
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1512
> Project: Apache YuniKorn
>  Issue Type: Task
>Reporter: Chen Yu Teng
>Assignee: Chen Yu Teng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.3.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [VOTE] Release Apache YuniKorn 1.2.0 RC1

[Weiwei Yang]

+1 (Binding)
- Verified LICENSE and NOTICE files
- Verified the file structure in the release tarball and file size
- Build the scheduler image on M1 mac (arm64)
- Install the scheduler locally on a Kind cluster (v1.26)
- Verified the build by running some simple applications
- Verified REST API for nodes/apps/queues

Weiwei

On Mon, Jan 30, 2023 at 6:13 PM Wilfred Spiegelenburg 
wrote:

> +1 (binding):
> - Verified signatures
> - Verified sha512 sum
> - Verified the LICENSE and NOTICE files
> - Built ARM based image on Mac/amd64 (go 1.19.4)
> - Ran validation script to verify proper deployment and startup using K8s
> 1.25.3
> - Checked UI and submitted an application
> - Ran unit tests (all passed)
>
> Wilfred
>
> On Mon, 30 Jan 2023 at 19:31, Manikandan R  wrote:
>
> > Hello everyone,
> >
> > I’d like to call a vote for releasing Apache YuniKorn 1.2.0 RC1.
> >
> > The release artifacts have been uploaded here:
> >   https://dist.apache.org/repos/dist/dev/yunikorn/1.2.0-RC1/
> >
> > My public key is located here:
> >   https://downloads.apache.org/yunikorn/KEYS
> >
> > JIRA issues that have been resolved in this release:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20YUNIKORN%20AND%20%22Target%20Version%22%20%3D%201.2.0%20ORDER%20BY%20status%20ASC
> >
> > Git tags for each component are as follows:
> >
> > yunikorn-scheduler-interface: v1.2.0-1
> > yunicorn-core: v1.2.0-1
> > yunikorn-k8shim: v1.2.0-1
> > yunikorn-web: v1.2.0-1
> > yunikorn-release: v1.2.0-1
> >
> > One the release is voted on and approved, all repos will be tagged 1.2.0
> > for consistency.
> >
> > Please review and vote. The vote will be open for at least 72 hours and
> > closes on Thursday, Feb 2 2023, 8 AM GMT.
> >
> > [ ] +1 Approve
> > [ ] +0 No opinion
> > [ ] -1 Disapprove (and the reason why)
> >
> >
> > Thank you,
> > Mani
> >
>

Re: [DISCUSS] Inclusive language: change git default branch name

[Weiwei Yang]

Let's decide this with Apache way and vote to a consensus.
I am -1 on this. If this is an effortless change, I will be +0, but this
needs a ton of effort. Also lots of implications we couldn't even realize
today.  I don't think we need to debate if the "master" word is offensive
or not, this is something really personal, and I respect different
opinions. But as an OSS community, we focus on building good sustainable
software, it is not necessarily (or best not to) to do changes like this.
That's my opinion.

Weiwei

On Fri, Jan 27, 2023 at 8:59 AM Craig Condit  wrote:

> This will probably be an unpopular opinion, but I’m -1 on this. I agree
> with being inclusive where it makes sense, however this whole “master vs.
> main” controversy is rooted in an idea that is ill-informed, ill-conceived,
> and provides no technical benefit. I’ll make my case as follows:
>
> 1. Ill-informed
>   - Myth: “master” as used in Git is reflective of slavery
>   - Fact: Words mean drastically different things in different contexts.
> The word “master” has 26 distinct definitions (
> https://www.dictionary.com/browse/master), only 1 of which has any
> connection with slavery. The most appropriate definition of “master” as
> used within Git would probably be 23: “Computers. An original data file or
> disk from which duplicates may be made.” This definition also follows from
> the previous one as used in audio recording. 22b: "an audio disk or tape
> from which duplicates may be made”.
>   - Conclusion: The desire to eradicate “master” from technology is rooted
> in a false premise that it has connections with slavery. Clearly, **when
> used in context** it does not. If we believe there is merit in removing
> "master” from git, should we also remove “master’s degrees”, honorary
> titles (Master Smith), ship captains (Master Mariner), recording masters,
> photographic masters, descriptions of the highly skilled: “chess master”,
> “grandmaster”, “schoolmaster”? Where does it stop?
>
> 2. Ill-Conceived
>   - Myth: Changing the branch name is free (or low cost)
>   - Fact: A change like this is anything but free. Having gone through
> this several times in the past, I can confidently say we will be dealing
> with the technical fallout of this for months if not longer. Every
> reference to the code, both publicly and privately, has to be updated. This
> includes not only working copies, but CI/CD pipelines, including for any
> user or company that maintains their own. There’s a “flag day” where
> everything prior is now irrevocably divorced from what comes after. Since
> we have multiple repositories, there’s now inter-repository work as well.
>   - Conclusion: This change would create a tremendous amount of technical
> debt.
>
> Racism, slavery, and exploitation are real issues that have been plaguing
> humanity since there have been multiple different people around. They won’t
> be solved by excising words from our vocabulary (especially those with
> legitimate technical definitions) and only serve to let people give
> themselves a pat on the back.  Want to make a real difference? Donate time
> or resources to one of many organizations working to end things like human
> trafficking. That will provide far more benefit than any mass rename of
> code.
>
> To those who may be tempted to find the use of the word “master” as used
> in Git offensive: I would encourage you to educate yourself about the
> nuance of language and recognize that the offense is in how people treat
> each other, and not in what words are deemed today as “ok” and tomorrow as
> “off-limits”.
>
>
> Craig
>
>
> > On Jan 26, 2023, at 6:25 PM, Wilfred Spiegelenburg 
> wrote:
> >
> > The naming of our default branch master, while very common, can be seen
> as
> > offensive.
> > I would like to propose that after the release of YuniKorn 1.2 we rename
> > our default branch from master to "trunk". There are other options like:
> > "main" or "development".
> >
> > We cannot execute the rename ourselves and need the help of the Apache
> > Infra team. After this change we will need to update github workflows,
> > documentation and some of the tools we use.
> >
> > After the renaming, all local clones will need to be updated to point to
> > the trunk. The process of how to do that is well documented [1] but it
> does
> > require action from everyone.
> >
> > two things that we need to confirm:
> > * the timing first action after the release of YuniKorn 1.2
> > * the name of the new default branch to become "trunk"
> >
> > Wilfred
> >
> > [1]
> >
> https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-branches-in-your-repository/renaming-a-branch#updating-a-local-clone-after-a-branch-name-changes
>
>

[jira] [Resolved] (YUNIKORN-1539) Add KubeConf talk info to events page

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1539.
---
Fix Version/s: 1.2.0
 Assignee: Weiwei Yang
   Resolution: Fixed

> Add KubeConf talk info to events page
> -
>
> Key: YUNIKORN-1539
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1539
> Project: Apache YuniKorn
>  Issue Type: Bug
>  Components: website
>    Reporter: Weiwei Yang
>    Assignee: Weiwei Yang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.2.0
>
>
> Add the recent KubeConf talk info to the events page



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1539) Add KubeConf talk info to events page

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1539:
-

 Summary: Add KubeConf talk info to events page
 Key: YUNIKORN-1539
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1539
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: website
Reporter: Weiwei Yang


Add the recent KubeConf talk info to the events page



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: YuniKorn talks at the recent ApacheCon

[Weiwei Yang]

Yes. And I have a KubeConf talk that hasn't been updated on the website yet.
I will submit a PR for that. Chaoran / Wilfred can you please update the
ApacheConf one as well?

Weiwei

On Tue, Jan 10, 2023 at 9:34 AM Wei-Chiu Chuang
 wrote:

> Hi
> I checked Yunikorn's event page
> <
> https://yunikorn.apache.org/community/events/#past-conference--meetup-recordings
> >
> and found no talks that took place at the recent ApacheCon there.
>
> Is there a plan to add the talks (recording or presentation slides) to the
> page? I am aware that Wilfred's talk is here in this ApacheCon slides
> repository . There's another
> talk at ApacheCon given by Bowen and Chaoran, which I sadly missed at the
> venue.
>
> Thanks,
> Weichiu
>

[jira] [Resolved] (YUNIKORN-1522) Add Chinese translation for release announcement 1.1

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1522.
---
Fix Version/s: 1.2.0
   Resolution: Fixed

> Add Chinese translation for release announcement 1.1
> 
>
> Key: YUNIKORN-1522
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1522
> Project: Apache YuniKorn
>  Issue Type: Bug
>  Components: documentation
>Reporter: Wilfred Spiegelenburg
>Assignee: Wu hsuang zong
>Priority: Critical
>  Labels: newbie, pull-request-available
> Fix For: 1.2.0
>
>
> Release announcement for the release 1.1 is missing from the zh-cn translated 
> documents



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [DISCUSS] Release manager for YuniKorn 1.2

[Weiwei Yang]

Thanks a lot Mani!
Look forward to 1.2 release!

On Tue, Jan 3, 2023 at 3:20 PM Wilfred Spiegelenburg 
wrote:

> Thank you Mani,
>
> I have assigned the umbrella jira for the release YUNIKORN-1517 to you.
>
> Wilfred
>
> On Tue, 3 Jan 2023 at 15:31, Manikandan R  wrote:
>
> > Hi Everyone,
> >
> > Happy New Year !
> >
> > I will take this up.
> >
> > Thanks,
> > Mani
> >
> >
> >
> > On Tue, Jan 3, 2023 at 7:10 AM Wilfred Spiegelenburg <
> wilfr...@apache.org>
> > wrote:
> >
> > > Hi all,
> > >
> > > Happy New Year ! We're getting close to the YuniKorn 1.2 release date
> > that
> > > we planned in the roadmap [1].
> > > That same roadmap does not have a release manager yet. Any committer
> can
> > be
> > > the release manager. If you want to be the release manager this is the
> > time
> > > to let everyone know.
> > >
> > > If this is the first time managing a release don't worry, you can
> always
> > > ask for help and the process is documented [2] and mostly automated via
> > > tools.
> > >
> > > Wilfred
> > >
> > > [1] https://yunikorn.apache.org/community/roadmap/
> > > [2] https://yunikorn.apache.org/community/release_procedure
> > >
> >
>

[jira] [Resolved] (YUNIKORN-1409) Adding Chinese translation of User and Group Resolution

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1409.
---
Fix Version/s: 1.2.0
   Resolution: Fixed

> Adding Chinese translation of User and Group Resolution
> ---
>
> Key: YUNIKORN-1409
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1409
> Project: Apache YuniKorn
>  Issue Type: Task
>Reporter: Chen Yu Teng
>Assignee: Wu hsuang zong
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1387) Adding Chinese translation of partition and queue configuration

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1387.
---
Fix Version/s: 1.2.0
   Resolution: Fixed

> Adding Chinese translation of partition and queue configuration
> ---
>
> Key: YUNIKORN-1387
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1387
> Project: Apache YuniKorn
>  Issue Type: Task
>Reporter: Chen Yu Teng
>Assignee: Chen Yu Teng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [DISCUSS] Removal of periodic state dumps

[Weiwei Yang]

+1 to remove this, I don't believe users heavily depend on this, but let's
keep this thread for at least a few days to collect feedback.

Thanks
Weiwei

On Mon, Dec 19, 2022 at 4:21 PM Craig Condit  wrote:

> Re-adding existing YUNIKORN-1483 text as formatting broke badly. I’m not
> proposing this is the way to go, just referencing the JIRA for discussion:
>
>
> 
>
> The current support for generating periodic state dumps implemented in
> YUNIKORN-940 has several warts:
>
> 1. The configuration in YUNIKORN-949 is done via the core scheduler
> configuration, leading to a random option on partitions which doesn't
> belong there and has nothing to do with scheduling.
>
> 2. Changing the frequency of the state dumps is done via the unsecured
> REST API. This is a potential denial-of-service vector.
>
> 3. Configuration V2 is now complete, which standardizes on using a
> ConfigMap to configure all YuniKorn options that make sense to be
> reconfigured. However, allowing the location to be changed at runtime makes
> no sense in a containerized environment.
>
> 4. Retrieving the state dumps requires mounting of external storage. This
> is necessarily a site-specific configuration and currently requires a
> custom Helm deployment.
>
> 5. The state dumps, though JSON, are emitted as text files with JSON
> appended to them, making parsing difficult.
>
> To address these issues:
>
> 1. Deprecate existing REST API configuration for frequency, and make it a
> no-op now for security reasons. We can remove it completely in 2.0.
>
> 2. Deprecate the statedumpfilepath option on partitions. Ignore it for
> security reasons now (and warn if found), and remove completely in 2.0.
>
> 3. Disable the feature by default. To enable it, we should require setting
> a specific environment variable:
>- YUNIKORN_STATE_DUMP_LOCATION=/path/to/dir : This would be required to
> enable the feature at all. Making it an env var makes sense as it is not an
> option that should be reconfigured (or even visible) in configuration.
>
> 4. Via configmap, we should allow the feature to be enabled / disabled and
> its frequency set. These options would have no effect if
> YUNIKORN_STATE_DUMP_LOCATION is not defined:
>   - periodicStateDump.enabled: "true" | "false" (default "false")
>   - periodicStateDump.frequency: "15m" (default value, do not allow more
> frequently than 1m intervals)
>   - periodicStateDump.count: 10 (default value)
>
> 5. Create an empty directory /yunkorn-state in the Docker image to store
> state dumps.
>
> 6. Add support to Helm for enabling state dump support as well as setting
> custom mount options (including quota). Enabling support should set the env
> var YUNIKORN_STATE_DUMP_LOCATION=/yunikorn-state and mount this directory
> via the options specified.
>
> 7. Output a single json file per dump and remove oldest files until count
> <= periodicStateDump.count entries: yunikorn-state-dump-MMDD-HHMM.json
>
> 
>
> > On Dec 19, 2022, at 6:18 PM, Craig Condit  wrote:
> >
> > All,
> >
> > I’d like to open a discussion about the future of the periodic state
> dump feature. To jumpstart the discussion, I opened
> https://issues.apache.org/jira/browse/YUNIKORN-1483, which is copied
> below for context. In the process of writing this up, it seems to me that
> we might actually be better off simply removing the feature, and relying
> solely on the REST API to retrieve state dumps on demand.
> >
> > In the current state, periodic state dumps need to be enabled, at which
> point they write to a local filesystem within the YuniKorn scheduler. This
> maps onto ephemeral storage, so to avoid out-of-space scenarios, an
> administrator needs to customize the YK Helm deployment with additional
> resource quota. Additionally, to even access the dumps, the filesystem
> needs to be mounted as a persistent volume and external code written to
> interact with the saved dumps. Given the mixed text-and-json format of
> these dumps, this can be rather complicated.
> >
> > Alternatively, users could simply deploy a cron container which pulls
> the state dump on-demand from the existing REST API. This ends up being
> considerably simpler.
> >
> > Are there objections to removing the existing periodic state dump
> functionality? Existing users who would be impacted greatly? To be clear,
> I’m not proposing removing the state dump itself; the version available via
> the REST API has proven extremely valuable. All that is on the table is
> removal of the automatic, periodic state dump which writes to local files.
> >
> > Looking forward to feedback,
> >
> > Craig
> >
> >
> >
> > 
> > YUNIKORN-1483 write-up:
> >
> > The current support for generating periodic state dumps implemented in
> YUNIKORN-940  has
> several warts:
> > The configuration in YUNIKORN-949 <
> https://issues.apache.org/jira/browse/YUNIKORN-949> is done via the core
> scheduler configuration, 

Re: [ANNOUNCE] New committer: Rainie Li

[Weiwei Yang]

Congrats, Rainie!

On Thu, Dec 15, 2022 at 7:15 AM Craig Condit  wrote:

> Congratulations Rainie!
>
>
> > On Dec 14, 2022, at 10:57 PM, Wilfred Spiegelenburg 
> wrote:
> >
> > The Project Management Committee (PMC) for Apache YuniKorn has invited
> > Rainie Li to become a committer and we are pleased to announce that she
> has
> > accepted.
> > Please join me in congratulating her.
> >
> > Congratulations & Welcome aboard Rainie !
> >
> > Wilfred
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [DISCUSS] Priority design

[Weiwei Yang]

Hi Wilfred

Thanks for putting things together, this looks really good.
I really like the concepts introduced in the proposal, such as priority
fencing and offset brilliant ideas!
I have one major doubt here, which is more related to how users can consume
this feature. I have added some comments in the doc, pls check.
The key idea is I want to make sure this is something users can easily
understand and use these concepts to achieve what they cannot do in the
past. And at the same time, they won't get confused by lots of implications.

Thanks
Weiwei

On Thu, Dec 1, 2022 at 5:42 PM Wilfred Spiegelenburg 
wrote:

> Hi,
>
> Priorities are a standard feature in K8s. YuniKorn has only had a very
> limited use of priorities in Daemon Set preemption and request sorting.
> Extending this into the rest of the scheduling cycle has been on the cards
> for a long time.
>
> This design does not change the sorting of requests, i.e. AllocationAsk,
> within an application. Priority based sorting has been used since our
> earliest releases. Adding a different sorting for requests falls outside of
> the design.
>
> Implementation will be tracked under YUNIKORN-1
>
> Please provide feedback on the design
>
> Wilfred
>
> [1]
>
> https://docs.google.com/document/d/1P_xlWSj65BiwAglvDvFyrnh8wZ7M72FRtIg199fyHNo/edit?usp=sharing
>

[jira] [Resolved] (YUNIKORN-1422) Adding Chinese translations of Run TensorFlow Jobs

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1422.
---
Fix Version/s: 1.2.0
   Resolution: Fixed

> Adding Chinese translations of Run TensorFlow Jobs
> --
>
> Key: YUNIKORN-1422
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1422
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: documentation
>Reporter: Wu hsuang zong
>Assignee: Wu hsuang zong
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.2.0
>
>
> While YUNIKORN-1339 add tutorial about time-slicing in Run TensorFlow Jobs, 
> the Chinese version isn't synced yet.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1386) Adding Chinese translation of deployment modes

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1386.
---
Fix Version/s: 1.2.0
   Resolution: Fixed

> Adding Chinese translation of deployment modes
> --
>
> Key: YUNIKORN-1386
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1386
> Project: Apache YuniKorn
>  Issue Type: Sub-task
>Reporter: Chen Yu Teng
>Assignee: Chen Yu Teng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1379) update meeting links on the community page

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1379.
---
Fix Version/s: 1.2.0
   Resolution: Fixed

> update meeting links on the community page
> --
>
> Key: YUNIKORN-1379
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1379
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: website
>Reporter: Wilfred Spiegelenburg
>Assignee: Jagadeesan A S
>Priority: Minor
>  Labels: newbie, pull-request-available
> Fix For: 1.2.0
>
>
> The community page is not in sync with the google doc for the community sync.
> The zoom links on the page are the old expired links and should be updated to 
> the new ones from the doc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1408) Adding Chinese translation of workload/Overview

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1408.
---
Fix Version/s: 1.2.0
   Resolution: Fixed

> Adding Chinese translation of workload/Overview
> ---
>
> Key: YUNIKORN-1408
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1408
> Project: Apache YuniKorn
>  Issue Type: Task
>Reporter: Chen Yu Teng
>Assignee: Chen Yu Teng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [DISCUSS] Removal of configmap update logic via REST API

[Weiwei Yang]

+1

Sounds good. The new config consolidation also looks nice, thanks for moving 
this forward.
Some minor comments added to the design doc, we can discuss offline for that.

Thanks
Weiwei



> On Oct 31, 2022, at 9:08 PM, Wilfred Spiegelenburg  
> wrote:
> 
> +1 I am for removing this functionality just from a security point alone.
> 
> The change also includes watching the configmap and reloading directly in
> the shim [1]. We would no longer rely on the mounted file to be changed by
> the kubelet. This will also minimise latency between changing and applying
> the change to the core. It will probably be faster and much simpler than
> using the file watch we have now and rely on the kubelet process.
> 
> Wilfred
> 
> [1] https://issues.apache.org/jira/browse/YUNIKORN-1365
> 
> On Tue, 1 Nov 2022 at 10:27, Craig Condit  wrote:
> 
>> Hello everyone,
>> 
>> I’d like to open a discussion around the idea of removing the current
>> logic that supports writing a new scheduler configuration via the REST API.
>> 
>> Currently, this logic is disabled in two different ways (and has been for
>> several releases). First, it is turned off if config auto-refresh is
>> enabled (the default). Second, it doesn’t work unless a non-standard and
>> elevated set of permissions is granted to the service account running
>> YuniKorn. This happened as part of YUNIKORN-997: Use K8s fine-grained
>> access control for YuniKorn scheduler [1]. So, it has not been easily
>> activated since 1.0.0 shipped.
>> 
>> This functionality was originally created to workaround potential delays
>> in configmap update detection and allow for more predictable updates since
>> it’s possible that a kubelet doesn’t refresh the on-disk copy of the
>> configmap in a timely manner. However, this hasn’t been true by default in
>> modern Kubernetes versions and no instances of this have been reported in
>> quite some time, and config auto-refresh has been active by default for
>> many releases.
>> 
>> Additionally, this functionality is extremely insecure, as the REST API
>> has no security on it. Any user who can talk to the REST API could upload
>> any configuration they want, and this was why YUNIKORN-997 disabled this
>> functionality.
>> 
>> With the upcoming design for YUNIKORN-1221: Configuration V2 [2], we’d
>> like to remove this functionality once and for all as it greatly simplifies
>> configuration handling moving forward.
>> 
>> Let me know if you have any concerns,
>> 
>> Craig Condit
>> 
>> 
>> [1] https://issues.apache.org/jira/browse/YUNIKORN-997 <
>> https://issues.apache.org/jira/browse/YUNIKORN-997>
>> [2] https://issues.apache.org/jira/browse/YUNIKORN-1221 <
>> https://issues.apache.org/jira/browse/YUNIKORN-1221>
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: YuniKorn community meetup: Santa Clara, CA

[Weiwei Yang]

Sounds good. Wilfred, could you pls create a meetup link on Linkedin so we
can start to spread the messages?
I'd like to cover 3 topics:
 - Karpenter support
 - Job priority high-level discussion
 - GPU support

I'll draft something and share it in the meeting.

Weiwei

On Mon, Oct 3, 2022 at 10:56 AM Wilfred Spiegelenburg 
wrote:

> I checked with the Cloudera office team and they prefer Thursday 20 October
> 2022.
> Let's lock in that day to start with.
>
> For the program this is some of the things we should cover:
> * progress I have made with the K8s scheduling sig on the pre-enqueue
> filtering.
> * feedback from the ApacheCON presentations (Chaoran, Bowen & Wilfred).
> * discussion: K8s version support.
> * General Roadmap
>
> Wilfred
>
>
> On Mon, 3 Oct 2022 at 12:12, Weiwei Yang  wrote:
>
> > Hi Wilfred
> >
> > I suggest having this on Friday afternoon, Oct 21, so we have enough time
> > to spread the messages and possibly invite more people to join.
> > We can have a few planned presentations and preserve some time for
> general
> > and roadmap discussions.
> > I can't wait for meeting you guys in person!!
> >
> > Weiwei
> >
> > On Mon, Oct 3, 2022 at 8:26 AM Chenya Zhang  >
> > wrote:
> >
> > > Big +1!
> > >
> > > I'll also be in the Bay area the week of Oct 17th. I can also help
> > Wilfred
> > > to organize this meetup.
> > >
> > > Let me check if there are topics of interest from our side and share
> them
> > > here. This is very exciting!
> > >
> > > Best,
> > > Chenya
> > >
> > > On Fri, Sep 30, 2022 at 4:38 PM Weiwei Yang  wrote:
> > >
> > > > That sounds great!!
> > > > Can we start to collect the topics and finalize the date ASAP?
> > > >
> > > > Thanks
> > > > Weiwei
> > > >
> > > > On Thu, Sep 29, 2022 at 10:38 PM Wilfred Spiegelenburg <
> > > > wilfr...@apache.org>
> > > > wrote:
> > > >
> > > > > HI all,
> > > > >
> > > > > There has not been a community meetup in a while. With ApacheCON
> next
> > > > week
> > > > > in New Orleans I will be in the USA for a while.
> > > > >
> > > > > I would like to propose a meetup in Santa Clara, California in the
> > 3rd
> > > > week
> > > > > of October. That is the week from 17-21 October. Cloudera is
> willing
> > to
> > > > > host and organise the meetup at their office.
> > > > > We can work on the exact content over the next weeks.
> > > > >
> > > > > Anyone is welcome, please let me know what you think.
> > > > >
> > > > > Wilfred
> > > > >
> > > >
> > >
> >
>

Re: YuniKorn community meetup: Santa Clara, CA

[Weiwei Yang]

Hi Wilfred

I suggest having this on Friday afternoon, Oct 21, so we have enough time
to spread the messages and possibly invite more people to join.
We can have a few planned presentations and preserve some time for general
and roadmap discussions.
I can't wait for meeting you guys in person!!

Weiwei

On Mon, Oct 3, 2022 at 8:26 AM Chenya Zhang 
wrote:

> Big +1!
>
> I'll also be in the Bay area the week of Oct 17th. I can also help Wilfred
> to organize this meetup.
>
> Let me check if there are topics of interest from our side and share them
> here. This is very exciting!
>
> Best,
> Chenya
>
> On Fri, Sep 30, 2022 at 4:38 PM Weiwei Yang  wrote:
>
> > That sounds great!!
> > Can we start to collect the topics and finalize the date ASAP?
> >
> > Thanks
> > Weiwei
> >
> > On Thu, Sep 29, 2022 at 10:38 PM Wilfred Spiegelenburg <
> > wilfr...@apache.org>
> > wrote:
> >
> > > HI all,
> > >
> > > There has not been a community meetup in a while. With ApacheCON next
> > week
> > > in New Orleans I will be in the USA for a while.
> > >
> > > I would like to propose a meetup in Santa Clara, California in the 3rd
> > week
> > > of October. That is the week from 17-21 October. Cloudera is willing to
> > > host and organise the meetup at their office.
> > > We can work on the exact content over the next weeks.
> > >
> > > Anyone is welcome, please let me know what you think.
> > >
> > > Wilfred
> > >
> >
>

Re: YuniKorn community meetup: Santa Clara, CA

[Weiwei Yang]

That sounds great!!
Can we start to collect the topics and finalize the date ASAP?

Thanks
Weiwei

On Thu, Sep 29, 2022 at 10:38 PM Wilfred Spiegelenburg 
wrote:

> HI all,
>
> There has not been a community meetup in a while. With ApacheCON next week
> in New Orleans I will be in the USA for a while.
>
> I would like to propose a meetup in Santa Clara, California in the 3rd week
> of October. That is the week from 17-21 October. Cloudera is willing to
> host and organise the meetup at their office.
> We can work on the exact content over the next weeks.
>
> Anyone is welcome, please let me know what you think.
>
> Wilfred
>

Re: [DISCUSSION] User and group handling

[Weiwei Yang]

Hi Wilfred

IIRC, Visa was interested in this feature, could you please ask for some
feedback from them?

Thanks
Weiwei

On Thu, Sep 8, 2022 at 12:21 AM Wilfred Spiegelenburg 
wrote:

> Peter Bacsko and I have been working on an enhancement to our user and
> group handling as part of the user quota design that was shared earlier. We
> have come up with a design that will allow us to retrieve user details,
> including groups, from the Kubernetes authentication information.
>
> The user information is passed onto YuniKorn as an annotation on the pod.
> This will replace the current user label that can be added to a pod. The
> new design should also improve the content and reliability of the user
> information.
>
> Feedback and comment welcome,
> Wilfred
>
> [1]
>
> https://docs.google.com/document/d/1HTY1s4k3wascg3GQqu6-ewoXCl_nO84gLAtzjSxnFSQ/edit?usp=sharing
>

Re: [VOTE] Release Apache YuniKorn 1.1.0 RC1

[Weiwei Yang]

+1 (binding)

 - built the docker images from source on M1 mac
 - installed yunikorn locally with the released helm charts
 - run simple workloads to verify the scheduler works fine
 - verified the web UI pages, home / apps / queues / nodes, look good
 - verified the queue utilization bar is working properly

Peter, thanks for helping the release,  now you have 6 binding +1 votes : )

Weiwei


On Tue, Sep 6, 2022 at 6:40 AM Peter Bacsko  wrote:

> +1 (binding) from me
>
> - built images from source
> - ran unit tests
> - ran validate_cluster.sh to install the release on KIND
> - checked REST endpoints
> - checked UI
>
> Thank you all for the voting on the RC1 for 1.1.0.
>
> That concludes the voting: it has passed with 5 binding +1, 0 non binding
> +1, no 0 or -1 votes.
>
> After publishing the artifacts and finishing the release process, I'll send
> out an announcement email.
>
> Thank you,
> Peter
>
> On Tue, Sep 6, 2022 at 3:12 PM TingYao  wrote:
>
> > +1
> >
> > - build images(ubuntu/amd64)
> > - run unit test
> > - deploy Yunikorn to kubernetes cluster
> >
> > Thanks Peter
> >
> > Tingyao
> >
> > Peter Bacsko  於 2022年9月5日 週一 下午4:20寫道：
> >
> > > Thanks for the votes so far.
> > >
> > > Some folks are still out, so I'm extending the voting deadline until
> > > September 7, 2020 12:00pm CEST.
> > >
> > > Thanks,
> > > Peter
> > >
> > > On Sat, Sep 3, 2022 at 5:05 PM 陳Chen宥騰Yuteng 
> wrote:
> > >
> > > > +1
> > > > - Built ARM based image(arm64v8)
> > > > - Ran unit tests
> > > > - Deployed Yunikorn on Kubernetes cluster.
> > > > - Tried the sleep example. Pods could be scheduled by Yunikorn.
> > > >
> > > > Wilfred Spiegelenburg  於 2022年9月2日 週五
> 上午10:35寫道：
> > > >
> > > > > +1 (binding)
> > > > >
> > > > > - Verified signatures
> > > > > - Verified sha512 sum
> > > > > - Verified the LICENSE changes
> > > > > - Built ARM based image on Mac/amd64
> > > > > - Ran validation script to verify proper startup
> > > > > - Ran unit tests
> > > > >
> > > > > Filed YUNIKORN-1305 to fix the sha labeling in the generated image
> > but
> > > > that
> > > > > is not important enough to stop the release.
> > > > >
> > > > > Wilfred
> > > > >
> > > > > On Fri, 2 Sept 2022 at 03:23, Craig Condit 
> > > > wrote:
> > > > >
> > > > > > +1.
> > > > > >
> > > > > > - Verified signatures
> > > > > > - Verified sha512 sum
> > > > > > - Built release (on Mac/amd64)
> > > > > > - Ran validation script to verify proper startup
> > > > > > - Executed e2e tests on both standard and plugin mode
> > > > > >
> > > > > > Other than a couple of flaky tests (known issue) everything looks
> > > > great.
> > > > > >
> > > > > > Thanks Peter for the release work!
> > > > > >
> > > > > >
> > > > > > Craig
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Sep 1, 2022, at 7:03 AM, Peter Bacsko 
> > > wrote:
> > > > > > >
> > > > > > > Thanks Craig.
> > > > > > >
> > > > > > > I re-generated the artifacts with just using "1.1.0" in
> > > > > > > release-configs.json.
> > > > > > >
> > > > > > > It's based the same code base/tags, so it's still the RC1.
> > > > > > >
> > > > > > > I removed the previous artifacts and the new ones are available
> > > here:
> > > > > > >  https://dist.apache.org/repos/dist/dev/yunikorn/1.1.0-RC1/
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Peter
> > > > > > >
> > > > > > > On Wed, Aug 31, 2022 at 11:34 PM Wilfred Spiegelenburg <
> > > > > > wilfr...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> The version that is used in the name is also in the
> > substitutions
> > > in
> > > > > the
> > > > > > >> Makefile, helm charts etc.
> > > > > > >>
> > > > > > >> Updating the release-configs.json should be clarified in:
> > > > > > >> * update the version with the release version: examples:
> 0.12.2
> > or
> > > > > 1.1.0
> > > > > > >> * update the tags with the current release tag examples:
> > 0.12.2-1
> > > or
> > > > > > >> 1.0.0-1
> > > > > > >>
> > > > > > >> We need to re-run the release tool with the update in the
> > version.
> > > > > > >>
> > > > > > >> Wilfred
> > > > > > >>
> > > > > > >> On Thu, 1 Sept 2022 at 00:17, Craig Condit <
> ccon...@apache.org>
> > > > > wrote:
> > > > > > >>
> > > > > > >>> Unfortunately, I have to vote -1 on this.
> > > > > > >>>
> > > > > > >>> Release candidates, if they are to become actual releases,
> must
> > > be
> > > > > > binary
> > > > > > >>> identical once voted upon and released. This archive contains
> > > > > numerous
> > > > > > >>> references to “RC1” including the top-level directory within
> > the
> > > > > > archive
> > > > > > >>> itself. All of these will at a minimum change the sha512sum
> of
> > > the
> > > > > > >> archive,
> > > > > > >>> and logically result in different contents.
> > > > > > >>>
> > > > > > >>> However, I don’t see an issue with the source code, so simply
> > > > > > >> re-releasing
> > > > > > >>> the tarball without ‘-RC1’ in the version will probably work.
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> 

Re: [DISCUSSION] Planning for v1.2 and beyond

[Weiwei Yang]

Dear YuniKorn devs/users, if you have any feature requests, this is the
time to call them out, so that we can discuss and see if they can fit into
the 1.2.0 timeline. Look forward to hearing from you.
Thank you Wilfred for putting everything together.

On Thu, Aug 25, 2022 at 12:10 AM Wilfred Spiegelenburg 
wrote:

> Now that we are really close to getting v1.1 out it is time to look
> forward to v1.2 and beyond.
> The release v1.1 has been focussed on ARM builds and stability for
> recovery and the REST interface.
> The release v1.2 is going into the direction of enhancing quota
> tracking and enforcement. The current work that is already in progress
> is:
> * maxRunningApplications for queues (YUNIKORN-790)
> * generic resources for dynamic queues (YUNIKORN-1275)
> * user usage tracking (design doc released)
>
> User based usage tracking will have a follow up with enforcement,
> design doc for that is in progress. There is more room to add new
> things into v1.2. The scope does not have to be limited to the above.
>
> We also started looking at a design for an overall YuniKorn
> configuration. That should be picked back up again. There are some
> smaller fixes and enhancements being worked on, like exposing more
> details in the REST api around reservations, plenty of scope left for
> things like that. We do not have to list each jira but if there are
> other larger pieces that people would like to work on it would be nice
> to add them to the roadmap.
>
> I will create a jira to update the roadmap with the new points when we
> have finalised the discussion here.
>
> Wilfred
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

[jira] [Created] (YUNIKORN-1293) Add custom redirects to the current version doc

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1293:
-

 Summary: Add custom redirects to the current version doc
 Key: YUNIKORN-1293
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1293
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: website
Reporter: Weiwei Yang


Docusaurus hides the version number in the URL for the latest versions, e.g 
right now our latest version is 1.0.0, but http://localhost:3000/docs/1.0.0 
gives a 404. Ideally, we should make have an accessible URL for this version as 
well. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [DISCUSSION] k8shim: running e2e tests only on successful build

[Weiwei Yang]

Hi Wilfred

Thanks for the proposals. I am

+1 to

  e2e-tests:
needs: build
runs-on: ubuntu-latest

I feel a little concerned about the second proposal. Are we seeing any
issues with today's pipeline that inspire you with this proposal?

On Tue, Aug 23, 2022 at 8:02 AM Craig Condit  wrote:

> I think I would be +1 on the e2e-after-build proposal, but I’m a strong -1
> on pre-commit enforcement. Due to the interdependencies between our
> different modules, there have been (and likely will be again) scenarios
> where we need to push multiple changes to get a build back into a “healthy
> state” after some refactoring. This gets very complicated if we can’t
> commit to a branch without a passing build. Conversely, we have yet (to my
> knowledge) encountered a scenario where the build was broken for an
> extended (or even short?) time in the master branch - we already get
> notifications if a build fails after commit and in my experience these have
> been dealt with very swiftly.
>
> Craig
>
>
> > On Aug 22, 2022, at 11:42 PM, Wilfred Spiegelenburg 
> wrote:
> >
> > While looking at the way we run our build jobs at the moment we start
> > the e2etest job in parallel with the build job of the k8shim. These
> > tests run for 25-30 minutes. They all get started even if the
> > compilation fails or we do not pass the unit tests.
> > Should we not wait for a successful build job before we start the e2e
> > tests? If the build job does not succeed we should never commit the
> > change anyway.
> >
> > Proposal is to change the flow and commit requirements.
> >
> > First proposal is to update the k8shim workflow [1] with the following
> > change that requires a successful build job before starting the
> > e2e-tests:
> > ---
> >  e2e-tests:
> >needs: build
> >runs-on: ubuntu-latest
> > ---
> > The build which includes a number of checks and the unit tests is
> > quick. We really do not want or need to proceed with running the e2e
> > tests if the build job has failed. The job build will run and on
> > successful completion the e2e-tests job will run automatically.
> >
> > Second proposal is that we also enforce a passed build job before
> > commit. Turning on branch protection and requiring the build job to
> > pass before a commit is manageable from the asf.yaml file we already
> > maintain. This makes sure that we have at least the base tests
> > passing. Not all e2e tests might pass but confidence is at least
> > higher.
> > -
> > github:
> >  protected_branches:
> >master:
> >  required_status_checks:
> >strict: false
> >contexts:
> >  - build
> >  required_pull_request_reviews:
> >dismiss_stale_reviews: false
> >required_approving_review_count: 1
> >  required_linear_history: true
> >  required_signatures: false
> >
> > # The following branch protections only ensure that force pushes are not
> allowed
> > branch-1.1: {}
> > branch-1.0: {}
> > branch-0.12: {}
> > branch-0.11: {}
> > branch-0.10: {}
> > branch-0.9: {}
> > branch-0.8: {}
> > -
> >
> > This branch protection we can add to all code repos: core, k8shim,
> > scheduler-interface and web to stay consistent.
> >
> > Comments are welcome. If there are no comments in the next week I'll
> > log a jira and start the work to update the workflow and branch
> > protection early September.
> >
> > Wilfred
> >
> > [1]
> https://github.com/apache/yunikorn-k8shim/blob/master/.github/workflows/main.yml
> > [2]
> https://cwiki.apache.org/confluence/display/INFRA/Git+-+.asf.yaml+features#Git.asf.yamlfeatures-Branchprotection
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

[jira] [Resolved] (YUNIKORN-1292) Fix github pages issues after moving to TLP

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1292.
---
Fix Version/s: 1.1.0
 Assignee: Weiwei Yang
   Resolution: Fixed

> Fix github pages issues after moving to TLP
> ---
>
> Key: YUNIKORN-1292
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1292
> Project: Apache YuniKorn
>  Issue Type: Bug
>    Reporter: Weiwei Yang
>    Assignee: Weiwei Yang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.1.0
>
>
> https://apache.github.io/yunikorn-release/ still has the out-of-dated issue. 
> This was pointed out by the Spark community in this PR: 
> https://github.com/apache/spark/pull/37622#discussion_r952877147



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1292) Fix github pages issues after moving to TLP

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1292:
-

 Summary: Fix github pages issues after moving to TLP
 Key: YUNIKORN-1292
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1292
 Project: Apache YuniKorn
  Issue Type: Bug
Reporter: Weiwei Yang


https://apache.github.io/yunikorn-release/ still has the out-of-dated issue. 
This was pointed out by the Spark community in this PR: 
https://github.com/apache/spark/pull/37622#discussion_r952877147



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [DISCUSSION] Replacing the namespace quota annotations

[Weiwei Yang]

Looks good to me.
Add some minor comments, thanks!

On Thu, Aug 4, 2022 at 12:12 AM Wilfred Spiegelenburg 
wrote:

> I would like to propose a replacement of the current namespace quota
> annotation in an upcoming release. The current annotation only allows
> specifying cpu and or memory. It does not allow the same flexibility
> as we allow for quotas in the queue configuration, or what can be
> specified in the gang definitions.
>
> I have created a document with the details of the proposal, see this
> google doc: [1]  The document is open for commenting to everyone. I
> have also logged YUNIKORN-1275 [2] for tracking the work going
> forward. You can also leave comments in the jira.
>
> Wilfred
>
> [1]
> https://docs.google.com/document/d/1XFB-_TQJTa_vgl97ATwU3hqqDkuFpUsfYglVpdyK55k/edit#
> [2] https://issues.apache.org/jira/browse/YUNIKORN-1275
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [DISCUSSION] Yunikorn release 1.1.0

[Weiwei Yang]

Hi Peter

Thanks for driving the next release. The plan looks good.

Weiwei

On Mon, Aug 8, 2022 at 4:43 PM Wilfred Spiegelenburg 
wrote:

> Sounds like a good plan.
>
> Wilfred
>
> On Tue, 9 Aug 2022 at 09:05, Craig Condit  wrote:
> >
> > Hi Peter,
> >
> > Thanks for taking on the 1.1.0 release. I think this plan looks good.
> >
> > - Craig
> >
> >
> > > On Aug 8, 2022, at 7:30 AM, Peter Bacsko  wrote:
> > >
> > > Hi all,
> > >
> > > we've been discussing the next release of Apache Yunikorn. I was asked
> to
> > > be the release manager of 1.1.0.
> > >
> > > Besides misc bug fixes, the following larger items are to be delivered:
> > > * More details exposed on the REST API
> > > * Multi-architecture build
> > > * Recovery stabilization
> > > * DaemonSet scheduling
> > >
> > > Resource tracking, group retrieval and user/group limit enforcement are
> > > expected to be released in Yunikorn 1.2.0.
> > >
> > > There are some open items with target version 1.1.0:
> > >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20YUNIKORN%20AND%20%22Target%20Version%22%20%3D%201.1.0%20ORDER%20BY%20status%20ASC
> > >
> > > Please have a look at this list and decide whether it's feasible to
> > > complete them before code freeze. If not, then retarget the tickets
> > > accordingly.
> > >
> > > Branching should be done next week.
> > > Suggested date for code freeze 26th Aug.
> > > RC package release date: 29th Aug.
> > >
> > > If the RC passes the voting, then we can officially release Yunikorn
> 1.1.0
> > > in the early days September.
> > >
> > > Thoughts, opinions?
> > >
> > > Thanks,
> > > Peter
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [ANNOUNCE] New PMC member: Peter Bacsko

[Weiwei Yang]

Well deserved & congrats, Peter!

Weiwei

On Mon, Jul 25, 2022 at 9:36 PM Manikandan R  wrote:

> Congratulations !
>
> On Tue, Jul 26, 2022 at 8:39 AM Wei-Chiu Chuang 
> wrote:
>
> > Congrats!
> >
> > On Mon, Jul 25, 2022 at 6:03 PM Wilfred Spiegelenburg <
> wilfr...@apache.org
> > >
> > wrote:
> >
> > > The Project Management Committee (PMC) for Apache YuniKorn has invited
> > > Peter Bacsko to become a PMC member and we are pleased to announce
> > > that he has accepted.
> > >
> > > On behalf of the Apache YuniKorn PMC
> > > Wilfred
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> > >
> > >
> >
>

[ANNOUNCE] New committer: Ted Lin

[Weiwei Yang]

The Project Management Committee (PMC) for Apache YuniKorn
has invited Ted Lin to become a committer and we are pleased
to announce that he has accepted.

Being a committer enables easier contribution to the
project since there is no need to go via the patch
submission process. This should enable better productivity.
A PMC member helps manage and guide the direction of the project.

Weiwei

[ANNOUNCE] New PMC: Manikandan R

[Weiwei Yang]

The Project Management Committee (PMC) for Apache YuniKorn has
invited Manikandan R to become a PMC member and we are pleased to announce that
he has accepted.

On behalf of
Apache YuniKorn PMC

Re: [DISCUSS] removal of latest tag from docker hub

[Weiwei Yang]

We used to have GitHub action to auto-build and push "latest" images for
the convenience of test & dev
but I guess that was removed a long time ago. Still worth checking if
anywhere still uses "latest" images. For e2e tests, IIRC, we are pulling
the source code and building the docker images locally on the testing VM,
it probably won't break.



On Mon, Jun 20, 2022 at 6:27 AM Craig Condit  wrote:

> We need to be careful about how we do this, as it will cause breakage. Off
> the top of my head, e2e tests for the shim will definitely break, as they
> will be unable to deploy the web UI successfully if the web:latest tag goes
> away. There may be others as well…
>
>
> - Craig
>
>
> > On Jun 19, 2022, at 8:32 PM, Wilfred Spiegelenburg 
> wrote:
> >
> > On docker hub we publish images with the tag "latest". According to
> > the Apache release policy we can release compiled packages as part of
> > our release. The artifacts have to comply with a couple of simple
> > rules [1]. An image with the "latest" tag does not comply with these
> > rules as it does not have the same version number as the source
> > release.
> >
> > We should clean up the docker hub and remove the "latest" tags [2].
> > Before we make that change we should add a note to the Readme on the
> > docker hub to show what we publish. After a short grace period we can
> > then remove the latest tag. A good point would be to have that
> > coincide with the next release for YuniKorn.
> >
> > Wilfred
> >
> > [1] https://www.apache.org/legal/release-policy.html#compiled-packages
> > [2] https://hub.docker.com/r/apache/yunikorn/tags?page=1=latest
> > [3] https://hub.docker.com/repository/docker/apache/yunikorn
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [DISCUSS] Building multi architecture images

[Weiwei Yang]

So looking forward to seeing this capability to be added in 1.1.0
Awesome work, thank you!

On Thu, Jun 16, 2022 at 7:33 AM Craig Condit  wrote:

> I think this is a great start. I would additionally like to see
> instructions that allow for building multi-arch images on a (relatively)
> stock Linux install without third-party software other than Docker itself.
> I know it’s possible, and probably mostly there, but I think it’s important
> to limit the third-party requirements for an RM as much as possible.
>
> Thanks for working on this.
>
> - Craig
>
>
> > On Jun 15, 2022, at 8:01 PM, Wilfred Spiegelenburg 
> wrote:
> >
> > Hi,
> >
> > By default we only publish images with the AMD64 architecture.
> > YUNIKORN-725 was implemented to allow builds to at least succeed on
> > both AMD64 and ARM architectures.
> >
> > Building images is a bit more involved and it was briefly discussed as
> > part of YUNIKORN-725, and then moved to YUNIKORN-1215 for
> > implementation.
> > Over the last week I have been working on getting a build working and
> > generating both AMD (amd64) and ARM (arm64v8) images combined into one
> > image tag.
> > I have a solution that works seamlessly if:
> > * Docker Desktop [1] is installed.
> > * Rancher Desktop [2] is installed with "dockerd (moby)" as the
> > container runtime
> > The code impact will be in the Docker and Makefile for the web and
> > k8shim repo. Plus a new tool in the release repo.
> >
> > Both desktops are free for use in open source development. Docker
> > desktop is only free within limits and might not work for some of us.
> > Rancher Desktop is the always free option. The CPU impact of running
> > Rancher Desktop on my local machine is slightly higher than docker
> > desktop. Besides that it works the same.
> >
> > The next step could be moving to multi stage docker images for the
> > build [3]. That should also open up the possibility for nedrctl to be
> > used for building the multi architecture images.
> >
> > My proposal is to:
> > * release multi architecture images for the 1.1.0 release
> > * delay multi stage build to a later release
> >
> > Any comments on this plan? I will raise the PRs against the different
> > repos as part of YUNIKORN-1215 in the next day or so.
> >
> > Wilfred
> >
> > [1] https://www.docker.com/products/docker-desktop/
> > [2] https://rancherdesktop.io
> > [3]
> https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

[jira] [Created] (YUNIKORN-1237) Queue usage bar in web UI is broken

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1237:
-

 Summary: Queue usage bar in web UI is broken
 Key: YUNIKORN-1237
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1237
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: webapp
Reporter: Weiwei Yang


After upgrading to 1.0, the queue usage is no longer working. Screenshot 
attached



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1226) Deprecate maturity page in YuniKorn

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1226.
---
Resolution: Fixed

> Deprecate maturity page in YuniKorn
> ---
>
> Key: YUNIKORN-1226
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1226
> Project: Apache YuniKorn
>  Issue Type: Bug
>  Components: website
>Reporter: Sunil G
>Assignee: Saad Ur Rahman
>Priority: Major
>  Labels: newbee, pull-request-available, trivial
> Fix For: 1.1.0
>
>
> https://yunikorn.apache.org/community/maturity needs to be removed.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Reopened] (YUNIKORN-1226) Deprecate maturity page in YuniKorn

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang reopened YUNIKORN-1226:
---

> Deprecate maturity page in YuniKorn
> ---
>
> Key: YUNIKORN-1226
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1226
> Project: Apache YuniKorn
>  Issue Type: Bug
>  Components: website
>Reporter: Sunil G
>Assignee: Saad Ur Rahman
>Priority: Major
>  Labels: newbee, pull-request-available, trivial
> Fix For: 1.1.0
>
>
> https://yunikorn.apache.org/community/maturity needs to be removed.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1226) Deprecate maturity page in YuniKorn

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1226.
---
Fix Version/s: 1.1.0
   Resolution: Fixed

Merged, thanks for taking care of this.

> Deprecate maturity page in YuniKorn
> ---
>
> Key: YUNIKORN-1226
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1226
> Project: Apache YuniKorn
>  Issue Type: Bug
>  Components: website
>Reporter: Sunil G
>Assignee: Saad Ur Rahman
>Priority: Major
>  Labels: newbee, pull-request-available, trivial
> Fix For: 1.1.0
>
>
> https://yunikorn.apache.org/community/maturity needs to be removed.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1224) Failed to publish website due to incompatible node version

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1224.
---
Resolution: Fixed

Merged, thanks for the fix [~yuchaoran2011]

> Failed to publish website due to incompatible node version
> --
>
> Key: YUNIKORN-1224
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1224
> Project: Apache YuniKorn
>  Issue Type: Bug
>  Components: website
>Reporter: Chaoran Yu
>Assignee: Chaoran Yu
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.1.0
>
>
> Website publish no longer works. Example: 
> [https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:73.]
>  
> The error is:
>  
> yarn install v1.22.15 
> [64|https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:65]info
>  No lockfile found. 
> [65|https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:66][1/4]
>  Resolving packages... 
> [66|https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:67][2/4]
>  Fetching packages... 
> [67|https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:68]error
>  @docusaurus/core@2.0.0-beta.21: The engine "node" is incompatible with this 
> module. Expected version ">=16.14". Got "16.13.0" 
> [68|https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:69]info
>  Visit [https://yarnpkg.com/en/docs/cli/install] for documentation about this 
> command. 
> [69|https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:70]error
>  Found incompatible module. 
> [70|https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:71]The
>  command '/bin/sh -c yarn install' returned a non-zero code: 1 
> [71|https://github.com/apache/yunikorn-site/runs/6638395350?check_suite_focus=true#step:3:72]
>  
> Need to change the base image to use node 16.14 or higher



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1218) Scheduler crashed with concurrent map access error in health checker

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1218:
-

 Summary: Scheduler crashed with concurrent map access error in 
health checker
 Key: YUNIKORN-1218
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1218
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: core - scheduler
Reporter: Weiwei Yang
Assignee: Weiwei Yang


After YUNIKORN-1107, the health checker runs as a background thread in 30s 
interval. We observed a few scheduler restarts in the past week that seems to 
be caused by this thread, because it has an unsafe access to the partition 
context without proper read lock. I have uploaded a patch to reproduce this 
locally, and a file of the stack trace when crash happens. 





--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1213) The interval of the background health checker needs to be configurable

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1213:
-

 Summary: The interval of the background health checker needs to be 
configurable
 Key: YUNIKORN-1213
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1213
 Project: Apache YuniKorn
  Issue Type: Improvement
  Components: core - scheduler
Reporter: Weiwei Yang


YUNIKORN-1107 adds a background running health checker to verify the scheduler 
data correctness in the fixed time interval 30s: 
https://github.com/apache/yunikorn-core/blob/3ba91fb8a41c0fd0dd6243326e583dea5167199f/pkg/scheduler/health_checker.go#L34.
 We need to make this configurable, either let the user set a longer/shorter 
interval, or completely disable it.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1203) Missing pending/available resource metrics for queues

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1203.
---
Resolution: Invalid

> Missing pending/available resource metrics for queues
> -
>
> Key: YUNIKORN-1203
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1203
> Project: Apache YuniKorn
>  Issue Type: Sub-task
>    Reporter: Weiwei Yang
>Priority: Major
>
> Per document here: 
> https://yunikorn.apache.org/docs/next/performance/metrics#queue-metrics. 
> Right now only the usedResourceMetrics is available, pending and available 
> metrics are both missing.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1203) Missing pending/available resource metrics for queues

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1203:
-

 Summary: Missing pending/available resource metrics for queues
 Key: YUNIKORN-1203
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1203
 Project: Apache YuniKorn
  Issue Type: Sub-task
Reporter: Weiwei Yang


Per document here: 
https://yunikorn.apache.org/docs/next/performance/metrics#queue-metrics. Right 
now only the usedResourceMetrics is available, pending and available metrics 
are both missing.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1202) Add metrics to track partition resources

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1202:
-

 Summary: Add metrics to track partition resources
 Key: YUNIKORN-1202
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1202
 Project: Apache YuniKorn
  Issue Type: Sub-task
Reporter: Weiwei Yang


When we monitor the cluster resources, we need to track what is available vs 
what is used. In the queue metrics, currently, we have per queue used resource 
metrics e.g yunikorn_queue_root_xyz_used_resource. But we do not have metrics 
to track what's the total partition resources (both used and total), we need to 
add that too.




--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [ANNOUNCE] Apache YuniKorn v1.0.0 release

[Weiwei Yang]

Congrats, and thanks for everyone's contribution!!

On Wed, May 4, 2022 at 1:31 PM Wilfred Spiegelenburg 
wrote:

> Hi all,
>
> It gives me great pleasure to announce that the Apache YuniKorn community
> has
> voted to release Apache YuniKorn v1.0.0.
>
> Apache YuniKorn v1.0.0 is the first release for the project as an
> Apache top level project. It also marks a major milestone as the first
> major release.
>
> It contains 173 fixes and improvements. The release details, list of
> major features and incompatible changes are on the v1.0.0 announcement
> page [1].
>
> You can also download the release from the Downloads page [2],
>
> Many thanks to everyone who contributed to the release. This release
> is a direct result of your great contributions.
>
> Wilfred
>
> [1] https://yunikorn.apache.org/release-announce/1.0.0
> [2] https://yunikorn.apache.org/download
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re: [VOTE] Release Apache YuniKorn 1.0.0 RC1

[Weiwei Yang]

+1 (binding)

I wasn't able to build the docker image for yunikorn-web on my M1 chip Mac,
Wilfred provided a workaround.
This is not a blocker as we are going to publish docker images, we can fix
this after the release, logged YUNIKORN-1193
.

Things I’ve checked:
 - Verified the binary signature
 - Reviewed CHANGE_LOG, LICENSE, NOTICE,
 - Build the project from the source
 - Install the scheduler using helm-chart locally
 - Run a simple job to verify the scheduling
 - Verified the default placement rule works as expected
 - Verified the new REST API endpoint for queues and apps

Thanks
Weiwei

On Wed, Apr 27, 2022 at 9:08 PM Wilfred Spiegelenburg 
wrote:

> > In YUNIKORN-954, "ws/v1/apps" is replaced with
> > "ws/v1/partition/{partitionName}/queues/{queueName}/application", not
> > "ws/v1/partition/{partitionName}/apps".
>
> Thank you for pointing that out. I fixed it in the release notes that
> will be uploaded to the website.
>
> Wilfred
>
> On Thu, 28 Apr 2022 at 03:04, 陳宥騰  wrote:
> >
> > +1, but there is a wrong description in In draft-release-1.0.0.md.
> > In YUNIKORN-954, "ws/v1/apps" is replaced with
> > "ws/v1/partition/{partitionName}/queues/{queueName}/application", not
> > "ws/v1/partition/{partitionName}/apps".
> >
> > I checked these things and they worked.
> >
> >- Image build
> >- Use helm to deploy yunikorn on kubernetes v1.21.0 cluster node.
> >- Profiling is work(go tool pprof
> >http://localhost:9080/debug/pprof/profile, go tool pprof
> >http://localhost:9080/debug/pprof/heap)
> >- Test some api and it can receive correct resource information
> >including cpu, memory and gpu.
> >   - ws/v1/partitions/
> >   - ws/v1/metrics/
> >   - ws/v1/config/
> >   - ws/v1/partition/{partitionName}/queues
> >   - ws/v1/partition/{partitionName}/nodes
> >   - ws/v1/partition/{partitionName}/queues/{queueName}/application
> >- Run sleep pods and nginx pod examples.
> >
> >
> > Wilfred Spiegelenburg  於 2022年4月27日 週三 上午11:31寫道：
> >
> > > Hello everyone,
> > >
> > > I would like to call a vote for releasing Apache YuniKorn 1.0.0 RC1.
> > > The first release for Apache YuniKorn as a top level project. Please
> > > note that the location of artefacts and the KEYS file has changed as
> > > we are now a top level project.
> > >
> > > The release artefacts have been uploaded here:
> > >   https://dist.apache.org/repos/dist/dev/yunikorn/1.0.0-RC1/
> > >
> > > My public key is located in the KEYS file:
> > >   https://downloads.apache.org//yunikorn/KEYS
> > >
> > > JIRA issues that have been resolved in this release:
> > >   https://issues.apache.org/jira/issues/?filter=12350818
> > >
> > > The release contains a number of incompatible changes that could
> > > impact the release verification. Please read the draft release notes
> > > attached to this vote for further details.
> > >
> > > Git tags for each component are as follows:
> > > yunikorn-scheduler-interface: v1.0.0-1
> > > yunikorn-core: v1.0.0-1
> > > yunikorn-k8shim: v1.0.0-2
> > > yunikorn-web: v1.0.0-1
> > > yunikorn-release: v1.0.0-1
> > >
> > > Once the release is voted on and approved, all repos will be tagged
> > > 1.0.0 for consistency.
> > >
> > > Please review and vote. The vote will be open for at least 72 hours
> > > and closes on Sunday, 1 May 2022, midnight PDT.
> > >
> > > [ ] +1 Approve
> > > [ ] +0 No opinion
> > > [ ] -1 Disapprove (and the reason why)
> > >
> > >
> > > Thank you,
> > > Wilfred
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > > For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

[jira] [Created] (YUNIKORN-1193) Release build failed on arm64

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1193:
-

 Summary: Release build failed on arm64
 Key: YUNIKORN-1193
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1193
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: webapp
Affects Versions: 1.0.0
Reporter: Weiwei Yang


Download the released tarball, and build the docker images from source with 
"make", I got the following error on M1 chip Mac with arm64 arch.

{code}
 => ERROR [buildstage 5/6] RUN yarn install 

74.7s
--
 > [buildstage 5/6] RUN yarn install:
#11 0.467 yarn install v1.22.18
#11 0.602 [1/4] Resolving packages...
#11 0.963 [2/4] Fetching packages...
#11 54.52 [3/4] Linking dependencies...
#11 72.57 [4/4] Building fresh packages...
#11 73.85 info Visit https://yarnpkg.com/en/docs/cli/install for documentation 
about this command.
#11 73.85 error /usr/uiapp/node_modules/puppeteer: Command failed.
#11 73.85 Exit code: 1
#11 73.85 Command: node install.js
#11 73.85 Arguments:
#11 73.85 Directory: /usr/uiapp/node_modules/puppeteer
#11 73.85 Output:
#11 73.85 The chromium binary is not available for arm64.
#11 73.85 If you are on Ubuntu, you can install with:
#11 73.85
#11 73.85  sudo apt install chromium
#11 73.85
#11 73.85
#11 73.85  sudo apt install chromium-browser
#11 73.85
#11 73.85 
/usr/uiapp/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:119
#11 73.85 throw new Error();
#11 73.85 ^
#11 73.85
#11 73.85 Error
#11 73.85 at 
/usr/uiapp/node_modules/puppeteer/lib/cjs/puppeteer/node/BrowserFetcher.js:119:27
#11 73.85 at FSReqCallback.oncomplete (node:fs:198:21)
#11 73.93 warning Error running install script for optional dependency: 
"/usr/uiapp/node_modules/nice-napi: Command failed.
#11 73.93 Exit code: 1
#11 73.93 Command: node-gyp-build
#11 73.93 Arguments:
#11 73.93 Directory: /usr/uiapp/node_modules/nice-napi
#11 73.93 Output:
#11 73.93 gyp info it worked if it ends with ok
#11 73.93 gyp info using node-gyp@8.4.1
#11 73.93 gyp info using node@16.14.2 | linux | arm64
#11 73.93 gyp ERR! find Python
#11 73.93 gyp ERR! find Python Python is not set from command line or npm 
configuration
#11 73.93 gyp ERR! find Python Python is not set from environment variable 
PYTHON
#11 73.93 gyp ERR! find Python checking if \"python3\" can be used
#11 73.93 gyp ERR! find Python - \"python3\" is not in PATH or produced an error
#11 73.93 gyp ERR! find Python checking if \"python\" can be used
#11 73.93 gyp ERR! find Python - \"python\" is not in PATH or produced an error
#11 73.93 gyp ERR! find Python
#11 73.93 gyp ERR! find Python 
**
#11 73.93 gyp ERR! find Python You need to install the latest version of Python.
#11 73.93 gyp ERR! find Python Node-gyp should be able to find and use Python. 
If not,
#11 73.93 gyp ERR! find Python you can try one of the following options:
#11 73.93 gyp ERR! find Python - Use the switch 
--python=\"/path/to/pythonexecutable\"
#11 73.93 gyp ERR! find Python   (accepted by both node-gyp and npm)
#11 73.93 gyp ERR! find Python - Set the environment variable PYTHON
#11 73.93 gyp ERR! find Python - Set the npm configuration variable python:
#11 73.93 gyp ERR! find Python   npm config set python 
\"/path/to/pythonexecutable\"
#11 73.93 gyp ERR! find Python For more information consult the documentation 
at:
#11 73.93 gyp ERR! find Python https://github.com/nodejs/node-gyp#installation
#11 73.93 gyp ERR! find Python 
**
#11 73.93 gyp ERR! find Python
#11 73.93 gyp ERR! configure error
#11 73.93 gyp ERR! stack Error: Could not find any Python installation to use
#11 73.93 gyp ERR! stack at PythonFinder.fail 
(/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:330:47)
#11 73.93 gyp ERR! stack at PythonFinder.runChecks 
(/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:159:21)
#11 73.93 gyp ERR! stack at PythonFinder. 
(/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:202:16)
#11 73.93 gyp ERR! stack at PythonFinder.execFileCallback 
(/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/find-python.js:294:16)
#11 73.93 gyp ERR! stack at exithandler (node:child_process:406:5)
#11 73.93 gyp ERR! stack at ChildProcess.errorhandler 
(node:child_process:418:5)
#11 73.93 gyp ERR! stack at ChildProcess.emit (node:events:526:28)
#11 73.93 gyp ERR! stack at Process.ChildProcess._handle.onexit 
(node:internal/child_process:289:12)
#11 73.93 gyp ERR! sta

Re: [ANNOUNCE] Branching for v1.0.0

[Weiwei Yang]

Hi Wilfred

Great, thanks for the update!

On Fri, Apr 15, 2022 at 8:12 AM Wilfred Spiegelenburg 
wrote:

> Branches for 1.0 have been created in all repositories.
> We have a two minor changes that we still like to get in before we
> close the release and build the first RC:
>   YUNIKORN-764: PR is open to add simple script to allow cluster
> validations
>   YUNIKORN-1116: Mani is working on a PR around a locking issue.
>
> As soon as they are in I will tag and create the RC.
> Wilfred
>
> On Wed, 13 Apr 2022 at 13:45, Wilfred Spiegelenburg 
> wrote:
> >
> > We 're getting close to the v1.0.0 release. We have been working our
> > way through the jiras and the PRs. At the moment we think we need
> > another day or two to get the last couple of changes in. We are
> > currently in a state where we can proceed with branching.
> >
> > Planning is to have the new branch branch-1.0 available start of
> > Friday 15 April 2022 PST time.
> > Creating the first release candidate is scheduled for the week after
> > with a vote to follow soon after.
> >
> > We have moved out a number of changes to the next minor release v.1.1.0.
> > There are about 10 jiras still open for 1.0. You can see the list here
> > [1]. The plan is to get all these fixed for the 1.0 release. Not all
> > these jiras might make it in before branching, it all depends on
> > review cycles and timing.
> >
> > Wilfred
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12348416
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

Re-organize community meetings

[Weiwei Yang]

Hi all

The old meeting link in our google calendar no longer works, we are working
on updating them. We will send out a notification once we get everything
fixed. And this week (Apr 12th) meeting for APAC will be canceled, we'd
likely resume the meeting from the week after. Sorry for the
inconvenience and please be patient with us. Thank you!

Weiwei

[jira] [Resolved] (YUNIKORN-1160) Fix codecov after migration

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1160.
---
Fix Version/s: 1.0.0
   Resolution: Fixed

> Fix codecov after migration
> ---
>
> Key: YUNIKORN-1160
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1160
> Project: Apache YuniKorn
>  Issue Type: Sub-task
>Reporter: Craig Condit
>    Assignee: Weiwei Yang
>Priority: Major
> Fix For: 1.0.0
>
>
> After the rename of the repos, our code coverage reports are no longer 
> happening. This seems to be due to passing the new repo names into the 
> codecov API. Someone with credentials to that site will need to update to the 
> new structure.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Today's community sync up (中文) is canceled

[Weiwei Yang]

Hi all

I can't make it to today's meeting, and could not find a delegate on time.
Therefore, canceling the meeting. If you have any questions, please raise
up on the mailing list or slack channel for discussions. See you next time!

Thanks!
Weiwei

Re: Moving repositories as part of graduation

[Weiwei Yang]

Hi Wilfred

Thank you for the detailed write up, this is very helpful.
I have created the INFRA ticket as discussed:
https://issues.apache.org/jira/browse/INFRA-23028.
Once the infra team renames our repos, Craig will make the changes
accordingly to get everything back up and working.
It may be disruptive for 1 day or 2, we will keep that posted on the
mailing list.

Thanks

On Wed, Mar 23, 2022 at 11:47 PM Wilfred Spiegelenburg 
wrote:

> Hi,
>
> As part of the top level graduation we are removing the incubator
> references from our code and documentation.
> One of the steps is moving or renaming the code repositories. The
> normal impact of that change is that the remote set in a cloned
> repository _should_ be updated *). For our project this rename has a
> flow on effect to our code base because of the way Go handles
> dependencies.
>
> This email is just to explain the process of the code change. Multiple
> steps have to be executed, steps will be announced when needed.
>
> Renaming the repositories will require us to change the imports
> throughout our code base. This is a multi step process. The steps
> involve:
> * update the scheduler interface (SI) code to change the module and
> import statements,
>   commit the changes.
> * update the core repository code to change the module and import
> statements,
>   changes must include the SI pseudo ref, and commit the change
> * update the k8shim repository code to change the module and import
> statements,
>   changes must including the core & SI pseudo refs and commit the change
>
> After the changes all tests in each of the repositories should pass.
> We must do this for the master branch. After the change in master we
> can prepare for branching 1.0.0 not before.
>
> Depending on whether we still want, or need, to release from an older
> branch further changes might be required. The changes might be the
> same as for the master, we might get away with less. The proposal is
> to not change anything for the older branches until we have a real
> need to release an older branch.
> At the moment that seems unlikely, however we cannot dismiss that.
>
> The first step in the process is to request the rename from Apache
> Infra for which Weiwei volunteered.
> Import changes will follow after the renaming of all repositories is
> ready. Craig has stepped up to perform that change.
> More details can be found in the community sync up notes [1]. A list
> of further tasks for which help is needed is also part of the notes.
>
> Wilfred
>
> [1]
> https://docs.google.com/document/d/165gzC7uhcKc5XDWiMYSRKBiPQBy2tDtXADUPuhGlUa0/edit#heading=h.4ojjr45cn3ra
> *) github will redirect the old repository name incubator-yunikorn-*
> to the new name yunikorn-* for most actions which means you might not
> notice but it can get confusing
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

[jira] [Created] (YUNIKORN-1141) [Umbrella] Post graduation tasks

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1141:
-

 Summary: [Umbrella] Post graduation tasks
 Key: YUNIKORN-1141
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1141
 Project: Apache YuniKorn
  Issue Type: Improvement
  Components: core - common, release, scheduler-interface, shim - 
kubernetes, webapp, website
Reporter: Weiwei Yang


As of March 16th, YuniKorn has been officially graduated from the incubator and 
become an ASF top-level project, the roster has been established: 
https://whimsy.apache.org/roster/committee/yunikorn. This JIRA is an umbrella 
to track the post-graduation tasks, including:
# Press Releases for new TLPs
# Handover
# Transferring Resources
# Final Revision of Podling Incubation Records
related doc: 
https://svn.apache.org/repos/infra/websites/production/incubator/content/guides/graduation.html#top-level-board-proposal



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1137) Fix typo in placement related messages

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1137.
---
Fix Version/s: 1.0.0
   Resolution: Fixed

> Fix typo in placement related messages
> --
>
> Key: YUNIKORN-1137
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1137
> Project: Apache YuniKorn
>  Issue Type: Bug
>  Components: core - scheduler
>Reporter: ted
>Assignee: ted
>Priority: Minor
>  Labels: newbie, pull-request-available
> Fix For: 1.0.0
>
>
> Missing "e" in the word "placment"



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-951) Add perf-tool description into benchmarking tutorial page

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-951.
--
Fix Version/s: 1.0.0
   Resolution: Fixed

> Add perf-tool description into benchmarking tutorial page
> -
>
> Key: YUNIKORN-951
> URL: https://issues.apache.org/jira/browse/YUNIKORN-951
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: release, website
>Reporter: Chen Yu Teng
>Assignee: Chen Yu Teng
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> Describe performance tool and how to use this.
> Update perf tools doc into yunikorn 
> website([https://yunikorn.apache.org/docs/performance/performance_tutorial])
> Excepted context:
>  #  Cases setting in conf.yaml
>  ** Describe perf cases with default parameters according to conf.yaml 
> context  
>  ** Parameters description
>  #  How to start test
>  ** commands 
>  #  Meaning of outputs.
>  ** Explain what diagrams will produce according to default conf.yaml



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Dockershim removed in K8s 1.24

[Weiwei Yang]

Hi all

During today's community meeting, the following issue has been raised by
Ryan.
As dockershim will be removed in K8s 1.24 (
https://kubernetes.io/blog/2022/01/07/kubernetes-is-moving-on-from-dockershim/),
we need some assessment to see if we are affected. Such as

   - Do we have any issues supporting K8s 1.24?
   - Do we need any changes in our build process to remove docker
   dependency (e.g nerdctl )

BTW, we have quickly checked running YK on containerd runtime, it seems
everything is working fine. Please share your thoughts and spot potential
issues.

Thank you.
Weiwei

[jira] [Created] (YUNIKORN-1122) Move constants to scheduler interface

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1122:
-

 Summary: Move constants to scheduler interface
 Key: YUNIKORN-1122
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1122
 Project: Apache YuniKorn
  Issue Type: Improvement
  Components: core - common, scheduler-interface, shim - kubernetes
Reporter: Weiwei Yang
Assignee: TingYao Huang


While reviewing YUNIKORN-1103, I found there are quite some constants are still 
defined in shim/core repo. Since we have the ability to define constants in SI, 
we should move all COMMON constants to SI.  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1103) Support fetching queue name from pod annotation "yunikorn.apache.org/queue"

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1103.
---
Fix Version/s: 1.0.0
   Resolution: Fixed

> Support fetching queue name from pod annotation "yunikorn.apache.org/queue"
> ---
>
> Key: YUNIKORN-1103
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1103
> Project: Apache YuniKorn
>  Issue Type: Improvement
>Reporter: Weiwei Yang
>Assignee: Ting Yao,Huang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> Today, when we submit a Spark job, in order to know which queue this job will 
> be submitted to, we fetch the queue name from pod's spec, under the label:
> Pod {
>  Label:
>  queue: "root.abc"
> }
> besides that, we also want to support fetching queue names from pod 
> annotation:
> Pod {
>  annotation:
>  yunikorn.apache.org/queue: "root.abc"
> }
> BTW, this is for the static queue case, not with the placement rule.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1103) Support fetching queue name from pod annotation "yunikorn.apache.org/queue"

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1103:
-

 Summary: Support fetching queue name from pod annotation 
"yunikorn.apache.org/queue"
 Key: YUNIKORN-1103
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1103
 Project: Apache YuniKorn
  Issue Type: Improvement
Reporter: Weiwei Yang
Assignee: TingYao Huang


Today, when we submit a Spark job, in order to know which queue this job will 
be submitted to, we fetch the queue name from pod's spec, under the label:

Pod {
 Label:
 queue: "root.abc"
}

besides that, we also want to support fetching queue names from pod annotation:

Pod {
 annotation:
 yunikorn.apache.org/queue: "root.abc"
}

BTW, this is for the static queue case, not with the placement rule.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-990) DocSearch Migration

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-990.
--
Fix Version/s: 1.0.0
   Resolution: Fixed

> DocSearch Migration
> ---
>
> Key: YUNIKORN-990
> URL: https://issues.apache.org/jira/browse/YUNIKORN-990
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: website
>    Reporter: Weiwei Yang
>Assignee: Chen Yu Teng
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> h1. DocSearch is migrating!
> h1. 
> With DocSearch, we put developers needs first, so we always try to improve 
> their experience with our tools. We teamed up with our beloved Algolia 
> Crawler to provide a better platform for you to:
> * Access to the full Algolia Platform - Explore additional products and 
> features for free
> * Better collaboration - Invite and manage additional team members
> * Power and flexibility of the Algolia Crawler - Previously only available to 
> paid enterprise customers, it provides the ability to customize and refine 
> your indexing like never before
> * Work on your schedule - Schedule or trigger crawls based on your demands 
> from the Crawler interface or the GitHub action
> * Advanced tooling - The Crawler interface includes a live editor to maintain 
> your config and allow you to test your search results with DocSearch v3
> h1. What do I need to do?
> h1. 
> We tried making the migrations as smooth as possible, so all you have to do 
> to migrate your yunikorn index to our new infra is:
> # Create an Algolia account with your email address (the one that received 
> this email)
> # Join your own Algolia application (the invite is valid 7 days, please 
> contact us if you need a new one)
> # Update your DocSearch frontend integration with your new credentials: (ASK 
> PPMC)
> You can also read more about the migration on our documentation or contact us 
> at docsea...@algolia.com or on Discord
> h1. Can I still use my old credentials? When do I need to migrate by?
> h1. 
> Old credentials and indices will still be available, but crawl jobs will be 
> stopped 3 months after you've received this email.
> h1. Can I still use the legacy DocSearch scraper locally?
> h1. 
> Yes, the configs and the scraper will still be available, but not maintained.
> h1. How can I configure/update my new Crawler?
> h1. 
> After you've created your Algolia account and joined your Algolia 
> application, you can visit the Crawler interface and configure your crawlers!
> For any informations regarding your DocSearch configuration, please visit our 
> new documentation. If you were familiar with the legacy DocSearch scraper and 
> configs, please read our key parity page.You can also read more about the 
> Algolia Crawler on the Algolia Documentation.
> h1. Do I need to trigger my crawls manually?
> h1. 
> No, your crawls are already scheduled to run once a week, but you are now 
> able to trigger a new one whenever you want!
> h1. I'm not using this index/these credentials anymore
> h1. 
> Keep in mind that DocSearch is a community project, please let us know if 
> that's the case so we can disable it.
> Read more in our migration guide



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1073) AllocationAskRelease field allocationkey should be allocationKey

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1073.
---
Fix Version/s: 1.0.0
   Resolution: Fixed

> AllocationAskRelease field allocationkey should be allocationKey
> 
>
> Key: YUNIKORN-1073
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1073
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: core - common, scheduler-interface, shim - kubernetes
>Reporter: Wilfred Spiegelenburg
>Assignee: ted
>Priority: Major
>  Labels: newbie, pull-request-available
> Fix For: 1.0.0
>
>
> The case for the allocationkey in the AllocationAskRelease is incorrect and 
> should be fixed. Proper capitalisation would be allocationKey as it is used 
> in all other places in the interface.
> This has a flow on effect from the interface to the core and shim as the 
> field name in the message changes. It is a simple change that can almost be 
> done by a search and replace and 2 times a go.mod file updates
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: Wrong operation

[Weiwei Yang]

Hi Yuteng

No worries. Actually, I did not know we have that branch in our repo, we do
not need a dev branch like that and should be deleted anyway. Thanks for
letting us know, appreciate it.

On Thu, Feb 10, 2022 at 7:58 AM 陳宥騰  wrote:

> Sorry about my wrong operation when i checked website,
> I deleted wrong branch [YUNIKORN-701] which is wrong branch.
> I will be more careful in future.
>

[jira] [Resolved] (YUNIKORN-364) Add search support to yunikorn site

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-364.
--
Fix Version/s: 1.0.0
 Assignee: Yu Teng Chen
   Resolution: Fixed

> Add search support to yunikorn site
> ---
>
> Key: YUNIKORN-364
> URL: https://issues.apache.org/jira/browse/YUNIKORN-364
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: website
>Reporter: Adam Antal
>Assignee: Yu Teng Chen
>Priority: Major
> Fix For: 1.0.0
>
>
> It would be useful if the user were able to search in 
> http://yunikorn.apache.org/



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1064) Doc changes for YUNIKORN-958 & YUNIKORN-960

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1064.
---
Resolution: Fixed

Closing this as the PR has been merged.

> Doc changes for YUNIKORN-958 & YUNIKORN-960
> ---
>
> Key: YUNIKORN-1064
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1064
> Project: Apache YuniKorn
>  Issue Type: Sub-task
>Reporter: Manikandan R
>Assignee: Anuraag Nalluri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> Need to make doc changes for YUNIKORN-958 & YUNIKORN-960. Also, deprecation 
> info needs to be added.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Closed] (YUNIKORN-1072) Upgrade and sign-off status file

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang closed YUNIKORN-1072.
-
Fix Version/s: 1.0.0
   Resolution: Fixed

> Upgrade and sign-off status file
> 
>
> Key: YUNIKORN-1072
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1072
> Project: Apache YuniKorn
>  Issue Type: Sub-task
>  Components: community
>    Reporter: Weiwei Yang
>Assignee: Wilfred Spiegelenburg
>Priority: Major
> Fix For: 1.0.0
>
>
> The status file records and summarizes incubation-related information on the 
> podling. The PPMC is responsible for keeping this file current. Before you 
> are able to graduate, all tasks need to be completed.
> The status file is a great way of keeping tabs on how your project is doing 
> and what needs to be done to meet the graduation criteria. The Incubator PMC 
> will check this file when they vote on the graduation of your project. Once 
> all tasks are done and the listed criteria met, your project may be ready for 
> graduation.
> The status file of the JUDDI project is one example.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1072) Upgrade and sign-off status file

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1072:
-

 Summary: Upgrade and sign-off status file
 Key: YUNIKORN-1072
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1072
 Project: Apache YuniKorn
  Issue Type: Sub-task
  Components: community
Reporter: Weiwei Yang
Assignee: Weiwei Yang


The status file records and summarizes incubation-related information on the 
podling. The PPMC is responsible for keeping this file current. Before you are 
able to graduate, all tasks need to be completed.

The status file is a great way of keeping tabs on how your project is doing and 
what needs to be done to meet the graduation criteria. The Incubator PMC will 
check this file when they vote on the graduation of your project. Once all 
tasks are done and the listed criteria met, your project may be ready for 
graduation.

The status file of the JUDDI project is one example.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1068) Doc changes for state dump file path

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1068.
---
Resolution: Fixed

> Doc changes for state dump file path
> 
>
> Key: YUNIKORN-1068
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1068
> Project: Apache YuniKorn
>  Issue Type: Task
>Reporter: Anuraag Nalluri
>Assignee: Anuraag Nalluri
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> Need to document changes for YUNIKORN-949. Specifically, we need to document 
> the new partition-level config that can be set to change the output location 
> of periodic state dump. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1070) Potential scheduler memory leak

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1070.
---
Resolution: Duplicate

> Potential scheduler memory leak
> ---
>
> Key: YUNIKORN-1070
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1070
> Project: Apache YuniKorn
>  Issue Type: Bug
>    Reporter: Weiwei Yang
>Priority: Blocker
>
> Ben mentioned this in the slack, he runs 0.12.2 on EKS and runs into periodic 
> OOM cases for the scheduler in EKS after a few days.  Currently, the 
> scheduler is configured for 10GB of memory and eventually always seems to run 
> out of memory.In my environment, I have a lot of nodes coming in and out 
> of the cluster due to autoscaling.   Wondering if this could be a possible 
> reason or if you guys have any other ideas.   Let me know what kind of 
> troubleshooting information might be useful here, but there is just a 
> continuous growth of memory consumption that ends with OOMKilled.  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1070) Potential scheduler memory leak

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1070:
-

 Summary: Potential scheduler memory leak
 Key: YUNIKORN-1070
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1070
 Project: Apache YuniKorn
  Issue Type: Bug
Reporter: Weiwei Yang


Ben mentioned this in the slack, he runs 0.12.2 on EKS and runs into periodic 
OOM cases for the scheduler in EKS after a few days.  Currently, the scheduler 
is configured for 10GB of memory and eventually always seems to run out of 
memory.In my environment, I have a lot of nodes coming in and out of the 
cluster due to autoscaling.   Wondering if this could be a possible reason or 
if you guys have any other ideas.   Let me know what kind of troubleshooting 
information might be useful here, but there is just a continuous growth of 
memory consumption that ends with OOMKilled.  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[RESULT][VOTE] Apache YuniKorn (Incubating) - Community Graduation Vote

[Weiwei Yang]

Hi all

The Apache YuniKorn (Incubating) community graduation vote is now closed
and passed.
There are 15 +1 votes and no -1's:

10 PPMC
- Chenya Zhang
- Chaoran Yu
- Arun Suresh
- Jason Lowe
- Craig Condit
- Wilfred Spiegelenburg
- Sunil Govindan
- Akhil PB
- DB Tsai
- Subru Krishnan

5 Non PPMC
- Tingyao*
- Peter Bacsko*
- WenChih Lo
- Manikandan R*
- Li Gao*

(names with a suffix * are committers)

Thanks
Weiwei

Re: Apache YuniKorn (Incubating) - Community Graduation Vote

[Weiwei Yang]

Hi all

The vote has been running for a week, we are closing the vote now.
Thank you for participating in the votes, appreciate all the thoughts in
the previous thread:
https://lists.apache.org/thread/1stn48kr0q4t4zlt53m5nxwwbjg4rqb2 and the
community's effort to address them. Given the positive feedback received, I
will announce the result in another email.
Thank you.

On Fri, Jan 28, 2022 at 9:44 AM Li Gao  wrote:

> +1
>
> On Thu, Jan 27, 2022 at 9:46 PM Manikandan R  wrote:
>
> > +1
> >
> > On Fri, Jan 28, 2022 at 12:32 AM Subru Krishnan 
> wrote:
> >
> > > +1.
> > >
> > > On Wed, Jan 26, 2022 at 11:10 PM WenChih Lo 
> wrote:
> > >
> > > > +1
> > > >
> > > > DB Tsai  於 2022年1月27日 週四 下午2:36寫道：
> > > >
> > > >> +1
> > > >> DB Tsai  |  https://www.dbtsai.com/  |  PGP 42E5B25A8F7A82C1
> > > >>
> > > >>
> > > >> On Wed, Jan 26, 2022 at 10:29 PM Akhil PB 
> wrote:
> > > >>
> > > >> > +1
> > > >> >
> > > >> > On Thu, Jan 27, 2022 at 7:29 AM Sunil Govindan  >
> > > >> wrote:
> > > >> >
> > > >> >> +1
> > > >> >>
> > > >> >> Sunil
> > > >> >>
> > > >> >> On Tue, Jan 25, 2022 at 9:09 PM Weiwei Yang 
> > wrote:
> > > >> >>
> > > >> >>> Hi YuniKorn community and mentors
> > > >> >>>
> > > >> >>> Based on the discussion thread [1], after 2 years time of
> > > incubating,
> > > >> it
> > > >> >>> is considered that now is a good time to graduate YuniKorn from
> > the
> > > >> ASF
> > > >> >>> incubator and become a top-level Apache project. We have
> reviewed
> > > the
> > > >> ASF
> > > >> >>> project maturity model [2] and provided some assessment of the
> > > >> project's
> > > >> >>> maturity based on the guidelines. Details are included as the
> > > >> following. I
> > > >> >>> have enough reasons to believe the project has done sustainable
> > > >> development
> > > >> >>> successfully in the Apache way. Please read this and add your
> vote
> > > by
> > > >> >>> replying to this email, your feedback will be much
> appreciated!!!
> > > >> Note,
> > > >> >>> this vote is not just for committers or PPMC members, we welcome
> > > >> anyone in
> > > >> >>> the community to vote, thanks!
> > > >> >>>
> > > >> >>> *Code, License, and Copyright*
> > > >> >>>
> > > >> >>> All code is maintained on github, under Apache 2.0 license. We
> > have
> > > >> >>> reviewed all the dependencies and ensured they do not bring any
> > > >> license
> > > >> >>> issues. All the status files, license headers, and copyright are
> > up
> > > >> to date.
> > > >> >>>
> > > >> >>> *Release*
> > > >> >>>
> > > >> >>> The community has released 5 releases in the past 2 years, i.e
> > v0.8,
> > > >> >>> v0.9, v0.10, v0,11, and v0.12. These releases were done by 5
> > > different
> > > >> >>> release managers [3] and indicate the community can create
> > releases
> > > >> >>> independently. We have also a well-documented release process,
> > > >> automated
> > > >> >>> tools to help new release managers with the process.
> > > >> >>>
> > > >> >>> *Quality*
> > > >> >>>
> > > >> >>> The community has developed a comprehensive CI/CD pipeline as a
> > > guard
> > > >> of
> > > >> >>> the code quality. The pipeline runs per-commit license check,
> > > >> code-format
> > > >> >>> check, code-coverage check, UT, and end-to-end tests. All these
> > are
> > > >> built
> > > >> >>> as automated github actions, new contributors can easily trigger
> > and
> > > >> view
> > > >> >>> results when submitt

[jira] [Resolved] (YUNIKORN-935) Refactor resourcemanagercallback mock class

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-935.
--
Fix Version/s: 1.0.0
   Resolution: Fixed

> Refactor resourcemanagercallback mock class
> ---
>
> Key: YUNIKORN-935
> URL: https://issues.apache.org/jira/browse/YUNIKORN-935
> Project: Apache YuniKorn
>  Issue Type: Sub-task
>  Components: core - scheduler
>Reporter: Manikandan R
>Assignee: Manikandan R
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> Some test classes in core depending on rmcallback mocks ended up in having 
> redundant mocks/code as part of moving plugin interfaces from core to SI 
> (please refer https://issues.apache.org/jira/browse/YUNIKORN-907 for more 
> details)
> This redundancy should be avoided by creating resuable common mockup classes.
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: Local website build check 2nd language page not working

[Weiwei Yang]

Update:

Craig has posted a fix
https://github.com/apache/incubator-yunikorn-site/pull/122 and now
everything gets back to work again.
Thank you for the quick turnaround! Thank you everyone for collaborating on
this!!

On Fri, Jan 28, 2022 at 9:44 AM Weiwei Yang  wrote:

> Hi Chen Xiang
>
> I've merged https://github.com/apache/incubator-yunikorn-site/pull/117 but
> still see the build failure.
> Could you please help to take a look at how to fix this? Thanks
>
> On Fri, Jan 28, 2022 at 5:39 AM Wilfred Spiegelenburg 
> wrote:
>
>> Thank you for that, I found the reference in the docusaurus i18n
>> tutorial now too.
>> I'll open a jira to get the script updated so a choice can be made
>> which locale we should run in the dev setup.
>>
>> Wilfred
>>
>> On Fri, 28 Jan 2022 at 23:43, 陈 翔  wrote:
>> >
>> > You can run:  yarn run start -- --locale zh-cn
>> > Different languages need to be started independently in dev mode, but
>> they will be packaged together in the build phase.
>> >
>> > By the way, at present, my last submission of the master branch will
>> report an error when building. I have submitted a new PR to solve this
>> problem
>> >
>> >
>> > 发件人: Wilfred Spiegelenburg 
>> > 日期: 星期五, 2022年1月28日 下午6:49
>> > 收件人: dev@yunikorn.apache.org 
>> > 主题: Local website build check 2nd language page not working
>> > I have tried to build the website locally to confirm a change is
>> working.
>> > Is there anything special that needs to be done to allow browsing the
>> > translated pages?
>> > The only thing I get when I try to access the Chinese version (i.e.
>> > http://localhost:3000/zh-cn/) is an english page with "Page Not Found"
>> > message.
>> >
>> > I just execute the standard: local_build.sh run
>> > The logs show that it has built the zh-cn locale successfully and I
>> > can see the files in the build directory also. I tried cleaning up
>> > everything and re-ran the whole build from scratch.Used multiple
>> > browsers, cleaned the caches, nothing helped.
>> >
>> > Wilfred
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
>> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
>> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>>
>>

Re: Local website build check 2nd language page not working

[Weiwei Yang]

Hi Chen Xiang

I've merged https://github.com/apache/incubator-yunikorn-site/pull/117 but
still see the build failure.
Could you please help to take a look at how to fix this? Thanks

On Fri, Jan 28, 2022 at 5:39 AM Wilfred Spiegelenburg 
wrote:

> Thank you for that, I found the reference in the docusaurus i18n
> tutorial now too.
> I'll open a jira to get the script updated so a choice can be made
> which locale we should run in the dev setup.
>
> Wilfred
>
> On Fri, 28 Jan 2022 at 23:43, 陈 翔  wrote:
> >
> > You can run:  yarn run start -- --locale zh-cn
> > Different languages need to be started independently in dev mode, but
> they will be packaged together in the build phase.
> >
> > By the way, at present, my last submission of the master branch will
> report an error when building. I have submitted a new PR to solve this
> problem
> >
> >
> > 发件人: Wilfred Spiegelenburg 
> > 日期: 星期五, 2022年1月28日 下午6:49
> > 收件人: dev@yunikorn.apache.org 
> > 主题: Local website build check 2nd language page not working
> > I have tried to build the website locally to confirm a change is working.
> > Is there anything special that needs to be done to allow browsing the
> > translated pages?
> > The only thing I get when I try to access the Chinese version (i.e.
> > http://localhost:3000/zh-cn/) is an english page with "Page Not Found"
> > message.
> >
> > I just execute the standard: local_build.sh run
> > The logs show that it has built the zh-cn locale successfully and I
> > can see the files in the build directory also. I tried cleaning up
> > everything and re-ran the whole build from scratch.Used multiple
> > browsers, cleaned the caches, nothing helped.
> >
> > Wilfred
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> For additional commands, e-mail: dev-h...@yunikorn.apache.org
>
>

[jira] [Resolved] (YUNIKORN-1045) Add maturity matrix to the website

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1045.
---
Fix Version/s: 1.0.0
   Resolution: Fixed

> Add maturity matrix to the website
> --
>
> Key: YUNIKORN-1045
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1045
> Project: Apache YuniKorn
>  Issue Type: Sub-task
>  Components: website
>Reporter: Wilfred Spiegelenburg
>Assignee: Wilfred Spiegelenburg
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> As part of the graduation we need to fill out and make sure we have looked at 
> all points in the maturity model 
> ([https://community.apache.org/apache-way/apache-project-maturity-model.html)]
> We should document the answers in a clear way for everyone to access as part 
> of the website. 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Re: [VOTE] Release Apache YuniKorn (incubating) 0.12.2 RC4

[Weiwei Yang]

+1 (binding)

- Build the docker images from the source
- Verified the image SHAs are at the correct commit
- Install on a local cluster with helm charts, verified installation is good
- Run simple job and verify the K8s events


On Thu, Jan 27, 2022 at 9:04 PM Sunil Govindan  wrote:

> +1 (binding)
>
> Thanks Craig for the efforts
>
>
>- Verified checksum and signature
>- compiled and built the binaries from source code
>- brought a yunikorn cluster locally
>- Ran basic jobs
>
>
> Thanks
> Sunil
>
> On Wed, Jan 26, 2022 at 10:11 AM Craig Condit 
> wrote:
>
> > Sorry, description should say RC4.
> >
> >
> > > On Jan 26, 2022, at 12:10 PM, Craig Condit 
> > wrote:
> > >
> > > Hello everyone,
> > >
> > > I’d like to call a vote for releasing Apache YuniKorn (incubating)
> > 0.12.2 RC3.
> > >
> > > The release artifacts have been uploaded here:
> > https://dist.apache.org/repos/dist/dev/incubator/yunikorn/0.12.2-rc4/ <
> > https://dist.apache.org/repos/dist/dev/incubator/yunikorn/0.12.2-rc4/>
> > >
> > > My public key is located here:
> > https://downloads.apache.org/incubator/yunikorn/KEYS <
> > https://downloads.apache.org/incubator/yunikorn/KEYS>
> > >
> > > JIRA issues that have been resolved in this release:
> > https://issues.apache.org/jira/issues/?filter=12351270 <
> > https://issues.apache.org/jira/issues/?filter=12351270>
> > >
> > > Git tags for each component are as follows:
> > >
> > > incubator-yunikorn-scheduler-interface: v0.12.2-1
> > > incubator-yunikorn-core: v0.12.2-1
> > > incubator-yunikorn-k8shim: v0.12.2-4
> > > incubator-yunikorn-web: v0.12.2-1
> > > https://github.com/apache/incubator-yunikorn-release: <
> > https://github.com/apache/incubator-yunikorn-release:> v0.12.2-4
> > >
> > > One the release is voted on and approved, all repos will be tagged
> > 0.12.2 for consistency.
> > >
> > > Please review and vote. The vote will be open for at least 72 hours and
> > closes on Monday, January 31 2022, 1pm PDT.
> > >
> > > [ ] +1 Approve
> > > [ ] +0 No opinion
> > > [ ] -1 Disapprove (and the reason why)
> > >
> > >
> > > Thank you,
> > > Craig
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
> > For additional commands, e-mail: dev-h...@yunikorn.apache.org
> >
> >
>

[jira] [Resolved] (YUNIKORN-1061) Remove 1.19 from supported K8s versions in helm charts

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1061.
---
Fix Version/s: 1.0.0
   Resolution: Fixed

> Remove 1.19 from supported K8s versions in helm charts
> --
>
> Key: YUNIKORN-1061
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1061
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: release
>Reporter: Craig Condit
>Assignee: Craig Condit
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> Kubernetes 1.19 is EOL as of December 2021. We should mark it as EOL and 
> untested in our 1.0 helm charts.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1058) Update e2e tests to exercise plugin mode as well

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1058.
---
Fix Version/s: 1.0.0
   Resolution: Fixed

> Update e2e tests to exercise plugin mode as well
> 
>
> Key: YUNIKORN-1058
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1058
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: shim - kubernetes
>Reporter: Craig Condit
>Assignee: Craig Condit
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>
> We should update our e2e test runs to also test the scheduler in K8s plugin 
> mode.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Resolved] (YUNIKORN-1056) Update website with info about how to use scheduler plugin

[Weiwei Yang (Jira)]

 [ 
https://issues.apache.org/jira/browse/YUNIKORN-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang resolved YUNIKORN-1056.
---
Fix Version/s: 1.0.0
   Resolution: Fixed

> Update website with info about how to use scheduler plugin
> --
>
> Key: YUNIKORN-1056
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1056
> Project: Apache YuniKorn
>  Issue Type: Task
>  Components: website
>Reporter: Craig Condit
>Assignee: Craig Condit
>Priority: Major
>  Labels: pull-request-available
> Fix For: 1.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

[jira] [Created] (YUNIKORN-1055) Increase the web site build timeout in github action

[Weiwei Yang (Jira)]

Weiwei Yang created YUNIKORN-1055:
-

 Summary: Increase the web site build timeout in github action
 Key: YUNIKORN-1055
 URL: https://issues.apache.org/jira/browse/YUNIKORN-1055
 Project: Apache YuniKorn
  Issue Type: Bug
  Components: website
Reporter: Weiwei Yang


Seeing some failures in the auto-build because of timeout. I think we should 
increase the timeout for building the web: 
https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: dev-unsubscr...@yunikorn.apache.org
For additional commands, e-mail: dev-h...@yunikorn.apache.org

Apache YuniKorn (Incubating) - Community Graduation Vote

[Weiwei Yang]

Hi YuniKorn community and mentors

Based on the discussion thread [1], after 2 years time of incubating, it is
considered that now is a good time to graduate YuniKorn from the ASF
incubator and become a top-level Apache project. We have reviewed the ASF
project maturity model [2] and provided some assessment of the project's
maturity based on the guidelines. Details are included as the following. I
have enough reasons to believe the project has done sustainable development
successfully in the Apache way. Please read this and add your vote by
replying to this email, your feedback will be much appreciated!!! Note,
this vote is not just for committers or PPMC members, we welcome anyone in
the community to vote, thanks!

*Code, License, and Copyright*

All code is maintained on github, under Apache 2.0 license. We have
reviewed all the dependencies and ensured they do not bring any license
issues. All the status files, license headers, and copyright are up to date.

*Release*

The community has released 5 releases in the past 2 years, i.e v0.8, v0.9,
v0.10, v0,11, and v0.12. These releases were done by 5 different release
managers [3] and indicate the community can create releases independently.
We have also a well-documented release process, automated tools to help new
release managers with the process.

*Quality*

The community has developed a comprehensive CI/CD pipeline as a guard of
the code quality. The pipeline runs per-commit license check, code-format
check, code-coverage check, UT, and end-to-end tests. All these are built
as automated github actions, new contributors can easily trigger and view
results when submitting patches.

*Community*

The community has developed an easy-to-read homepage for the project [4],
the website hosts all the materials related to the project including
versioned documentation, user docs, developer docs, design docs,
performance docs. It provides the top-level navigation to the software
download page, where links to all our previous releases. It also has the
pages for the new contributors on-boarding with the project, such as how to
join community meetings, events links, etc.

The community shows appreciation to all contributors and welcomes all kinds
of contributions (not just for code). We have built an open, diverse
community and gathered many people to work together. With that, we have 41
unique code contributors and some non-code contributors as well. Many of
them have becoming to be committers and PPMC members while working with the
community. There were 2 new mentors, 8 new committers, 4 new PPMC from 6
different organizations [5] added in the incubating phase. And in total,
the project has 6 mentors, 23 PPMC, and 29 committers from at least 14
different organizations. All the info are generally available on the
project website, including some guidelines to help people become
committer/PPMC member [6]. Community collaboration was done in a
wide-public, open manner, we leverage regular bi-weekly/weekly community
meetings for 2 different timezones [7] and dev/user slack channels, mailing
lists for offline discussions.

*Independence*

The project was initially donated by Cloudera, but with a diverse open
source community, it has been operated as an independent project since it
entered into ASF incubator. The committers and PPMC members are a group of
passionate people from at least 14 different organizations, such as
Alibaba, Apple, Cloudera, Databricks, LinkedIn, Microsoft, Snowflake, etc.
The project's success is not depending on any single entity.

[1] https://lists.apache.org/thread/dno411y59g2pcy1d3kd7s3kdjz9jw65n
[2]
https://community.apache.org/apache-way/apache-project-maturity-model.html

[3] https://yunikorn.apache.org/community/download
[4] https://yunikorn.apache.org/
[5] https://incubator.apache.org/projects/yunikorn.html
[6] https://yunikorn.apache.org/community/people

[6]
https://docs.google.com/document/d/165gzC7uhcKc5XDWiMYSRKBiPQBy2tDtXADUPuhGlUa0

Re: Apache YuniKorn (Incubating) - Community Graduation Vote

[Weiwei Yang]

All sub-tasks under https://issues.apache.org/jira/browse/YUNIKORN-1005 are
now solved, and the name search is completed as well.
I will start a new thread for the vote.

On Tue, Jan 25, 2022 at 10:53 AM Chenya Zhang 
wrote:

> thanks Sunil! btw. I followed
> https://infra.apache.org/committer-email.html to configure my gmail
> settings to send from the apache email address. I originally used my gmail
> to subscribe to the private list and didn't receive a confirmation/error
> email. Hope others won't encounter the same problem.
>
> Best,
> Chenya
>
> On Tue, Jan 25, 2022 at 10:03 AM Sunil Govindan  wrote:
>
>> Yes. I reached out to a couple of them and added them to the list.
>> We still have a gap. Will try again.
>>
>> Thanks
>>  Sunil
>>
>> On Mon, Jan 24, 2022 at 10:45 PM Chenya Zhang <
>> chenyazhangche...@gmail.com>
>> wrote:
>>
>> > Subscribed to the private list! 3 -> 2 ? :)
>> >
>> > On Mon, Jan 24, 2022 at 6:02 PM Wilfred Spiegelenburg <
>> wilfr...@apache.org
>> > >
>> > wrote:
>> >
>> > > When you check the status page [1] you will see that a wiki is no
>> > > longer required.
>> > > We can skip adding it.
>> > >
>> > > BTW: I added Chenya to the roster that increases the PPMC members not
>> > > subscribed to 3 again after it was down to 2.
>> > >
>> > > Wilfred
>> > >
>> > > [1] https://incubator.apache.org/projects/yunikorn.html
>> > >
>> > >
>> > > On Mon, 24 Jan 2022 at 13:12, Weiwei Yang 
>> wrote:
>> > > >
>> > > > Hi Sunil
>> > > >
>> > > > I don’t think we ever have a wiki, do we still need to add that? I
>> see
>> > > some projects leave that empty as well.
>> > > >
>> > > > Sent from my iPhone
>> > > >
>> > > > > On Jan 23, 2022, at 2:07 PM, Sunil Govindan 
>> > wrote:
>> > > > >
>> > > > > @Weiwei Yang 
>> > > > > Could you please add WIKI as well to this?
>> > > > >
>> > > > > Thanks
>> > > > > Sunil
>> > > > >
>> > > > >> On Sun, Jan 23, 2022 at 1:33 PM Weiwei Yang 
>> > wrote:
>> > > > >>
>> > > > >> Thank you Felix.
>> > > > >> I have added the initial podling status file:
>> > > > >>
>> > > > >>
>> > >
>> >
>> https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings/yunikorn.yml
>> > > > >> .
>> > > > >> Please let me know if that looks good or not.
>> > > > >>
>> > > > >>> On Sat, Jan 22, 2022 at 10:18 PM Sunil Govindan <
>> sun...@apache.org
>> > >
>> > > wrote:
>> > > > >>>
>> > > > >>> I will reach out to them.
>> > > > >>>
>> > > > >>> Thanks
>> > > > >>> Sunil
>> > > > >>>
>> > > > >>> On Sat, Jan 22, 2022 at 9:00 PM Felix Cheung <
>> > > felixcheun...@hotmail.com>
>> > > > >>> wrote:
>> > > > >>>
>> > > > >>>> Pls add the podling status file
>> > > > >>>>
>> > > > >>>
>> > > > >>
>> > >
>> >
>> https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings/
>> > > > >>>>
>> > > > >>>> 3 ppmc members have not subscribed to private@
>> > > > >>>>
>> > > > >>>> These can be found on
>> > > > >>>> https://whimsy.apache.org/roster/ppmc/yunikorn
>> > > > >>>>
>> > > > >>>> <
>> > > > >>>
>> > > > >>
>> > >
>> >
>> https://svn.apache.org/repos/asf/incubator/public/trunk/content/podlings/
>> > > > >>>>>
>> > > > >>>> 
>> > > > >>>> From: Weiwei Yang 
>> > > > >>>> Sent: Thursday, January 20, 2022 10:05:55 PM
>> > > > >>>> To: dev@yunikorn.apache.org 
>> > > > >>>>