[jira] [Commented] (SPARK-24894) Invalid DNS name due to hostname truncation

[Thi Nguyen (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-24894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17039580#comment-17039580
 ] 

Thi Nguyen commented on SPARK-24894:


Why is Fix Version 3.0.0? This looks like a bug fix to me, so should've been in 
a patch version?

> Invalid DNS name due to hostname truncation 
> 
>
> Key: SPARK-24894
> URL: https://issues.apache.org/jira/browse/SPARK-24894
> Project: Spark
>  Issue Type: Bug
>  Components: Kubernetes
>Affects Versions: 2.3.1
>Reporter: Dharmesh Kakadia
>Assignee: Marcelo Masiero Vanzin
>Priority: Major
> Fix For: 3.0.0
>
>
> The truncation for hostname happening here 
> [https://github.com/apache/spark/blob/5ff1b9ba1983d5601add62aef64a3e87d07050eb/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L77]
>   is a problematic and can lead to DNS names starting with "-". 
> Originally filled here : 
> https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/229
> ```
> {{2018-07-23 21:21:42 ERROR Utils:91 - Uncaught exception in thread 
> kubernetes-pod-allocator 
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: 
> POST at: https://kubernetes.default.svc/api/v1/namespaces/default/pods. 
> Message: Pod 
> "user-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9" is 
> invalid: spec.hostname: Invalid value: 
> "-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9": a DNS-1123 
> label must consist of lower case alphanumeric characters or '-', and must 
> start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', 
> regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'). Received 
> status: Status(apiVersion=v1, code=422, 
> details=StatusDetails(causes=[StatusCause(field=spec.hostname, 
> message=Invalid value: 
> "-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9": a DNS-1123 
> label must consist of lower case alphanumeric characters or '-', and must 
> start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', 
> regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), 
> reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Pod, 
> name=user-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9, 
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, 
> message=Pod 
> "user-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9" is 
> invalid: spec.hostname: Invalid value: 
> "-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9": a DNS-1123 
> label must consist of lower case alphanumeric characters or '-', and must 
> start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', 
> regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), 
> metadata=ListMeta(resourceVersion=null, selfLink=null, 
> additionalProperties={}), reason=Invalid, status=Failure, 
> additionalProperties={}). at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:470)
>  at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
>  at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
>  at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
>  at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:226)
>  at 
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:769)
>  at 
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:356)
>  at 
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$$anon$1$$anonfun$3$$anonfun$apply$3.apply(KubernetesClusterSchedulerBackend.scala:140)
>  at 
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$$anon$1$$anonfun$3$$anonfun$apply$3.apply(KubernetesClusterSchedulerBackend.scala:140)
>  at org.apache.spark.util.Utils$.tryLog(Utils.scala:1922) at 
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$$anon$1$$anonfun$3.apply(KubernetesClusterSchedulerBackend.scala:139)
>  at 
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$$anon$1$$anonfun$3.apply(KubernetesClusterSchedulerBackend.scala:138)
>  at 
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
>  at 
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
>  at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
>  at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) 
> at 
> 

[jira] [Commented] (SPARK-24894) Invalid DNS name due to hostname truncation

[Yinan Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-24894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554602#comment-16554602
 ] 

Yinan Li commented on SPARK-24894:
--

[~mcheah]. We need to make sure the truncation leads to a valid hostname.

> Invalid DNS name due to hostname truncation 
> 
>
> Key: SPARK-24894
> URL: https://issues.apache.org/jira/browse/SPARK-24894
> Project: Spark
>  Issue Type: Bug
>  Components: Kubernetes
>Affects Versions: 2.3.1
>Reporter: Dharmesh Kakadia
>Priority: Major
>
> The truncation for hostname happening here 
> [https://github.com/apache/spark/blob/5ff1b9ba1983d5601add62aef64a3e87d07050eb/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L77]
>   is a problematic and can lead to DNS names starting with "-". 
> Originally filled here : 
> https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/229
> ```
> {{2018-07-23 21:21:42 ERROR Utils:91 - Uncaught exception in thread 
> kubernetes-pod-allocator 
> io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: 
> POST at: https://kubernetes.default.svc/api/v1/namespaces/default/pods. 
> Message: Pod 
> "user-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9" is 
> invalid: spec.hostname: Invalid value: 
> "-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9": a DNS-1123 
> label must consist of lower case alphanumeric characters or '-', and must 
> start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', 
> regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'). Received 
> status: Status(apiVersion=v1, code=422, 
> details=StatusDetails(causes=[StatusCause(field=spec.hostname, 
> message=Invalid value: 
> "-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9": a DNS-1123 
> label must consist of lower case alphanumeric characters or '-', and must 
> start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', 
> regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), 
> reason=FieldValueInvalid, additionalProperties={})], group=null, kind=Pod, 
> name=user-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9, 
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, 
> message=Pod 
> "user-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9" is 
> invalid: spec.hostname: Invalid value: 
> "-archetypes-all-weekly-1532380861251850404-1532380862321-exec-9": a DNS-1123 
> label must consist of lower case alphanumeric characters or '-', and must 
> start and end with an alphanumeric character (e.g. 'my-name', or '123-abc', 
> regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), 
> metadata=ListMeta(resourceVersion=null, selfLink=null, 
> additionalProperties={}), reason=Invalid, status=Failure, 
> additionalProperties={}). at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:470)
>  at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
>  at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:379)
>  at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:343)
>  at 
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:226)
>  at 
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:769)
>  at 
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:356)
>  at 
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$$anon$1$$anonfun$3$$anonfun$apply$3.apply(KubernetesClusterSchedulerBackend.scala:140)
>  at 
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$$anon$1$$anonfun$3$$anonfun$apply$3.apply(KubernetesClusterSchedulerBackend.scala:140)
>  at org.apache.spark.util.Utils$.tryLog(Utils.scala:1922) at 
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$$anon$1$$anonfun$3.apply(KubernetesClusterSchedulerBackend.scala:139)
>  at 
> org.apache.spark.scheduler.cluster.k8s.KubernetesClusterSchedulerBackend$$anon$1$$anonfun$3.apply(KubernetesClusterSchedulerBackend.scala:138)
>  at 
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
>  at 
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
>  at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
>  at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) 
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99) 
> at