[ https://issues.apache.org/jira/browse/SPARK-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142394#comment-14142394 ]
Dale Richardson edited comment on SPARK-3620 at 9/21/14 9:30 AM: ----------------------------------------------------------------- Seems to be discussion about moving to typesafe config and back again for version 0.9 http://apache-spark-developers-list.1001551.n3.nabble.com/Moving-to-Typesafe-Config-td381.html was (Author: tigerquoll): Seems to be discussion about moving to typesafe config and back again at http://apache-spark-developers-list.1001551.n3.nabble.com/Moving-to-Typesafe-Config-td381.html > Refactor config option handling code for spark-submit > ----------------------------------------------------- > > Key: SPARK-3620 > URL: https://issues.apache.org/jira/browse/SPARK-3620 > Project: Spark > Issue Type: Improvement > Components: Deploy > Affects Versions: 1.0.0, 1.1.0 > Reporter: Dale Richardson > Assignee: Dale Richardson > Priority: Minor > > I'm proposing its time to refactor the configuration argument handling code > in spark-submit. The code has grown organically in a short period of time, > handles a pretty complicated logic flow, and is now pretty fragile. Some > issues that have been identified: > 1. Hand-crafted property file readers that do not support the property file > format as specified in > http://docs.oracle.com/javase/6/docs/api/java/util/Properties.html#load(java.io.Reader) > 2. ResolveURI not called on paths read from conf/prop files > 3. inconsistent means of merging / overriding values from different sources > (Some get overridden by file, others by manual settings of field on object, > Some by properties) > 4. Argument validation should be done after combining config files, system > properties and command line arguments, > 5. Alternate conf file location not handled in shell scripts > 6. Some options can only be passed as command line arguments > 7. Defaults for options are hard-coded (and sometimes overridden multiple > times) in many through-out the code e.g. master = local[*] -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org