[jira] [Updated] (SPARK-27760) Spark resources - user configs change .count to be .amount
[
https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-27760:
--
Description:
For the Spark resources, we created the config
spark.\{driver/executor}.resource.\{resourceName}.count
I think we should change .count to be .amount. That more easily allows users to
specify things with suffix like memory in a single config and they can combine
the value and unit. Without this they would have to specify 2 separate configs
(like .count and .unit) which seems more of a hassle for the user.
Note the yarn configs for resources use amount:
spark.yarn.\{executor/driver/am}.resource=, where the
is value and unit together. I think that makes a lot of sense. Filed a separate
Jira to add .amount to the yarn configs as well.
was:
For the Spark resources, we created the config
spark.\{driver/executor}.resource.\{resourceName}.count
I think we should change .count to be .amount. That more easily allows users to
specify things with suffix like memory in a single config and they can combine
the value and unit. Without this they would have to specify 2 separate configs
which seems more of a hassle for the user.
Note the yarn configs for resources use amount:
spark.yarn.\{executor/driver/am}.resource=, where the
is value and unit together. I think that makes a lot of sense. Filed a separate
Jira to add .amount to the yarn configs as well.
> Spark resources - user configs change .count to be .amount
> --
>
> Key: SPARK-27760
> URL: https://issues.apache.org/jira/browse/SPARK-27760
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Thomas Graves
>Assignee: Thomas Graves
>Priority: Major
>
> For the Spark resources, we created the config
> spark.\{driver/executor}.resource.\{resourceName}.count
> I think we should change .count to be .amount. That more easily allows users
> to specify things with suffix like memory in a single config and they can
> combine the value and unit. Without this they would have to specify 2
> separate configs (like .count and .unit) which seems more of a hassle for the
> user.
> Note the yarn configs for resources use amount:
> spark.yarn.\{executor/driver/am}.resource=, where the amont> is value and unit together. I think that makes a lot of sense. Filed a
> separate Jira to add .amount to the yarn configs as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27760) Spark resources - user configs change .count to be .amount
[
https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-27760:
--
Description:
For the Spark resources, we created the config
spark.\{driver/executor}.resource.\{resourceName}.count
I think we should change .count to be .amount. That more easily allows users to
specify things with suffix like memory in a single config and they can combine
the value and unit. Without this they would have to specify 2 separate configs
which seems more of a hassle for the user.
Note the yarn configs for resources use amount:
spark.yarn.\{executor/driver/am}.resource=, where the
is value and unit together. I think that makes a lot of sense. Filed a separate
Jira to add .amount to the yarn configs as well.
was:
For the Spark resources, we created the config
spark.\{driver/executor}.resource.\{resourceName}.count
I think we should change .count to be .amount. That more easily allows users to
specify things with suffix like memory in a single config and they can combine
the value and unit. Without this they would have to specify 2 separate configs
which seems more of a hassle for the user.
> Spark resources - user configs change .count to be .amount
> --
>
> Key: SPARK-27760
> URL: https://issues.apache.org/jira/browse/SPARK-27760
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Thomas Graves
>Assignee: Thomas Graves
>Priority: Major
>
> For the Spark resources, we created the config
> spark.\{driver/executor}.resource.\{resourceName}.count
> I think we should change .count to be .amount. That more easily allows users
> to specify things with suffix like memory in a single config and they can
> combine the value and unit. Without this they would have to specify 2
> separate configs which seems more of a hassle for the user.
> Note the yarn configs for resources use amount:
> spark.yarn.\{executor/driver/am}.resource=, where the amont> is value and unit together. I think that makes a lot of sense. Filed a
> separate Jira to add .amount to the yarn configs as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27760) Spark resources - user configs change .count to be .amount
[
https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-27760:
--
Summary: Spark resources - user configs change .count to be .amount (was:
Spark resources - user configs change .count to be .amount, and yarn configs
should match)
> Spark resources - user configs change .count to be .amount
> --
>
> Key: SPARK-27760
> URL: https://issues.apache.org/jira/browse/SPARK-27760
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Thomas Graves
>Priority: Major
>
> For the Spark resources, we created the config
> spark.\{driver/executor}.resource.\{resourceName}.count
> I think we should change .count to be .amount. That more easily allows users
> to specify things with suffix like memory in a single config and they can
> combine the value and unit. Without this they would have to specify 2
> separate configs which seems more of a hassle for the user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-27760) Spark resources - user configs change .count to be .amount, and yarn configs should match
[
https://issues.apache.org/jira/browse/SPARK-27760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-27760:
--
Description:
For the Spark resources, we created the config
spark.\{driver/executor}.resource.\{resourceName}.count
I think we should change .count to be .amount. That more easily allows users to
specify things with suffix like memory in a single config and they can combine
the value and unit. Without this they would have to specify 2 separate configs
which seems more of a hassle for the user.
was:
For the Spark resources, we created the config
spark.\{driver/executor}.resource.\{resourceName}.count
I think we should change .count to be .amount. That more easily allows users to
specify things with suffix like memory in a single config and they can combine
the value and unit. Without this they would have to specify 2 separate configs
which seems more of a hassle for the user.
Spark on yarn already had added configs to request resources via the configs
spark.yarn.\{executor/driver/am}.resource=, where the
is value and unit together. We should change those configs to have a .amount
suffix on them to match the spark configs and to allow future configs to be
more easily added. YARN itself already supports tags and attributes so if we
want the user to be able to pass those from spark at some point having a suffix
makes sense.
> Spark resources - user configs change .count to be .amount, and yarn configs
> should match
> -
>
> Key: SPARK-27760
> URL: https://issues.apache.org/jira/browse/SPARK-27760
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Thomas Graves
>Priority: Major
>
> For the Spark resources, we created the config
> spark.\{driver/executor}.resource.\{resourceName}.count
> I think we should change .count to be .amount. That more easily allows users
> to specify things with suffix like memory in a single config and they can
> combine the value and unit. Without this they would have to specify 2
> separate configs which seems more of a hassle for the user.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
