[spark] branch branch-3.0 updated: [SPARK-32824][CORE] Improve the error message when the user forgets the .amount in a resource config

[gurwls223]

This is an automated email from the ASF dual-hosted git repository.

gurwls223 pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.0 by this push:
     new e86d90b  [SPARK-32824][CORE] Improve the error message when the user 
forgets the .amount in a resource config
e86d90b is described below

commit e86d90b21d4b3d6658b3cb6dd30daafb32b0c1bd
Author: Thomas Graves <tgra...@nvidia.com>
AuthorDate: Wed Sep 9 10:28:40 2020 +0900

    [SPARK-32824][CORE] Improve the error message when the user forgets the 
.amount in a resource config
    
    ### What changes were proposed in this pull request?
    
    If the user forgets to specify .amount on a resource config like 
spark.executor.resource.gpu, the error message thrown is very confusing:
    
    ```
    ERROR SparkContext: Error initializing 
SparkContext.java.lang.StringIndexOutOfBoundsException: String index out of 
range:
    -1 at java.lang.String.substring(String.java:1967) at
    ```
    
    This makes it so we have a readable error thrown
    
    ### Why are the changes needed?
    
    confusing error for users
    ### Does this PR introduce _any_ user-facing change?
    
    just error message
    
    ### How was this patch tested?
    
    Tested manually on standalone cluster
    
    Closes #29685 from tgravescs/SPARK-32824.
    
    Authored-by: Thomas Graves <tgra...@nvidia.com>
    Signed-off-by: HyukjinKwon <gurwls...@apache.org>
    (cherry picked from commit e8634d8f6f8548852a284a32c1b7da24bedd8ff7)
    Signed-off-by: HyukjinKwon <gurwls...@apache.org>
---
 core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala 
b/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
index 994b363..16fe897 100644
--- a/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
+++ b/core/src/main/scala/org/apache/spark/resource/ResourceUtils.scala
@@ -148,7 +148,12 @@ private[spark] object ResourceUtils extends Logging {
 
   def listResourceIds(sparkConf: SparkConf, componentName: String): 
Seq[ResourceID] = {
     sparkConf.getAllWithPrefix(s"$componentName.$RESOURCE_PREFIX.").map { case 
(key, _) =>
-      key.substring(0, key.indexOf('.'))
+      val index = key.indexOf('.')
+      if (index < 0) {
+        throw new SparkException(s"You must specify an amount config for 
resource: $key " +
+          s"config: $componentName.$RESOURCE_PREFIX.$key")
+      }
+      key.substring(0, index)
     }.distinct.map(name => new ResourceID(componentName, name))
   }
 


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org