[
https://issues.apache.org/jira/browse/YUNIKORN-2287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17802389#comment-17802389
]
Wilfred Spiegelenburg commented on YUNIKORN-2287:
-------------------------------------------------
{quote}I don't think UT is enough
{quote}
I agree. The base64 decoding seems to happen way down in the K8s and yaml code
base when the object is translated from the yaml that is stored into the in
memory object. This is not part of our code base. That is also why it was not
detected in the unit tests for the compression.
We probably can trace down where it happens by looking at the processing that
is triggered in {{GetConfigMapObj()}} there we call {{Yaml2Obj()}} with the
file. The file has base64 encoded data and we seem to get back the decoded data
in the {{BinaryData}} field for the queues.yaml.gz entry in the
{{v1.Configmap}} that is returned
Would be good to find and document what and where the decode happens. Also need
to make sure that we document clearly that the data in BinaryData must be
base64 encoded.
> Decompress function doesn't need to decode base64
> -------------------------------------------------
>
> Key: YUNIKORN-2287
> URL: https://issues.apache.org/jira/browse/YUNIKORN-2287
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: PoAn Yang
> Assignee: PoAn Yang
> Priority: Major
> Labels: e2e, pull-request-available
>
> I'm working on https://issues.apache.org/jira/browse/YUNIKORN-2267. I added
> an example as following, but compression configmap doesn't work.
>
> 1. Use gzip to compress and use base64 to encode the config
> ```
> echo "
> partitions:
> - name: default
> placementrules:
> - name: tag
> value: namespace
> create: true
> queues:
> - name: root
> submitacl: '*'
> queues:
> - name: parent
> submitacl: '*'" | gzip | base64
> ```
> 2. Set the result to `queues.yaml.gz` in binaryData field.
>
> Finally, we can see an error log in scheduler:
> ```
> 2023-12-22T15:32:15.913Z ERROR shim.config conf/schedulerconf.go:458
> failed to decode schedulerConfig entry \{"error": "illegal base64 data
> at input byte 0"}
> github.com/apache/yunikorn-k8shim/pkg/conf.Decompress
> github.com/apache/yunikorn-k8shim/pkg/conf/schedulerconf.go:458
> github.com/apache/yunikorn-k8shim/pkg/conf.FlattenConfigMaps
> github.com/apache/yunikorn-k8shim/pkg/conf/schedulerconf.go:495
> github.com/apache/yunikorn-k8shim/pkg/cache.(*Context).triggerReloadConfig
> github.com/apache/yunikorn-k8shim/pkg/cache/context.go:496
> github.com/apache/yunikorn-k8shim/pkg/cache.(*Context).updateConfigMaps
> github.com/apache/yunikorn-k8shim/pkg/cache/context.go:394
> k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate
> k8s.io/[email protected]/tools/cache/controller.go:250
> k8s.io/client-go/tools/cache.FilteringResourceEventHandler.OnUpdate
> k8s.io/[email protected]/tools/cache/controller.go:315
> k8s.io/client-go/tools/cache.(*processorListener).run.func1
> k8s.io/[email protected]/tools/cache/shared_informer.go:971
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
> k8s.io/[email protected]/pkg/util/wait/backoff.go:226
> k8s.io/apimachinery/pkg/util/wait.BackoffUntil
> k8s.io/[email protected]/pkg/util/wait/backoff.go:227
> k8s.io/apimachinery/pkg/util/wait.JitterUntil
> k8s.io/[email protected]/pkg/util/wait/backoff.go:204
> k8s.io/apimachinery/pkg/util/wait.Until
> k8s.io/[email protected]/pkg/util/wait/backoff.go:161
> k8s.io/client-go/tools/cache.(*processorListener).run
> k8s.io/[email protected]/tools/cache/shared_informer.go:967
> k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1
> k8s.io/[email protected]/pkg/util/wait/wait.go:72
> ```
>
> I think the root cause is that the golang object is already decoded, so we
> don't need to decode it again.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
