[jira] [Created] (SPARK-23291) SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1
Narendra created SPARK-23291: Summary: SparkR : substr : In SparkR dataframe , starting and ending position arguments in "substr" is giving wrong result when the position is greater than 1 Key: SPARK-23291 URL: https://issues.apache.org/jira/browse/SPARK-23291 Project: Spark Issue Type: Bug Components: SparkR Affects Versions: 2.2.1 Reporter: Narendra Defect Description : - For example ,an input string "2017-12-01" is read into a SparkR dataframe "df" with column name "col1". The target is to create a a new column named "col2" with the value "12" which is inside the string ."12" can be extracted with "starting position" as "6" and "Ending position" as "7" (the starting position of the first character is considered as "1" ) But,the current code that needs to be written is : df <- withColumn(df,"col2",substr(df$col1,7,8))) Observe that the first argument in the "substr" API , which indicates the 'starting position', is mentioned as "7" Also, observe that the second argument in the "substr" API , which indicates the 'ending position', is mentioned as "8" i.e the number that should be mentioned to indicate the position should be the "actual position + 1" Expected behavior : The code that needs to be written is : df <- withColumn(df,"col2",substr(df$col1,6,7))) Note : --- This defect is observed with only when the starting position is greater than 1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-18798) Expose the kill Executor in Yarn Mode
[ https://issues.apache.org/jira/browse/SPARK-18798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narendra closed SPARK-18798. > Expose the kill Executor in Yarn Mode > - > > Key: SPARK-18798 > URL: https://issues.apache.org/jira/browse/SPARK-18798 > Project: Spark > Issue Type: Improvement >Reporter: Narendra > > Expose the kill Executor in Yarn Mode > I can see spark already has exposed the kill Executor method through spark > context for Mesos , fi spark can expose the same method for Yarn it will > good feature if some want to test application stability by killing randomly > Executor > I see spark have kill Executor in YarnAllocator, it won't we much time > consuming to expose this , anyone can work I can also -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18798) Expose the kill Executor in Yarn Mode
[ https://issues.apache.org/jira/browse/SPARK-18798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736153#comment-15736153 ] Narendra commented on SPARK-18798: -- Thanks for Clarifying , It helped > Expose the kill Executor in Yarn Mode > - > > Key: SPARK-18798 > URL: https://issues.apache.org/jira/browse/SPARK-18798 > Project: Spark > Issue Type: Improvement >Reporter: Narendra > > Expose the kill Executor in Yarn Mode > I can see spark already has exposed the kill Executor method through spark > context for Mesos , fi spark can expose the same method for Yarn it will > good feature if some want to test application stability by killing randomly > Executor > I see spark have kill Executor in YarnAllocator, it won't we much time > consuming to expose this , anyone can work I can also -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18798) Expose the kill Executor in Yarn Mode
[ https://issues.apache.org/jira/browse/SPARK-18798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736067#comment-15736067 ] Narendra commented on SPARK-18798: -- i have put below explanation and this code from sparkContext > Expose the kill Executor in Yarn Mode > - > > Key: SPARK-18798 > URL: https://issues.apache.org/jira/browse/SPARK-18798 > Project: Spark > Issue Type: Improvement >Reporter: Narendra > > Expose the kill Executor in Yarn Mode > I can see spark already has exposed the kill Executor method through spark > context for Mesos , fi spark can expose the same method for Yarn it will > good feature if some want to test application stability by killing randomly > Executor > I see spark have kill Executor in YarnAllocator, it won't we much time > consuming to expose this , anyone can work I can also -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18798) Expose the kill Executor in Yarn Mode
[
https://issues.apache.org/jira/browse/SPARK-18798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736057#comment-15736057
]
Narendra commented on SPARK-18798:
--
I am putting below code it says
logWarning("Killing executors is only supported in coarse-grained mode") ,
this means it is supported only in Mesos ? if i am wrong please correct
/**
* :: DeveloperApi ::
* Request that the cluster manager kill the specified executors.
*
* @note This is an indication to the cluster manager that the application
wishes to adjust
* its resource usage downwards. If the application wishes to replace the
executors it kills
* through this method with new ones, it should follow up explicitly with a
call to
* {{SparkContext#requestExecutors}}.
*
* @return whether the request is received.
*/
@DeveloperApi
def killExecutors(executorIds: Seq[String]): Boolean = {
schedulerBackend match {
case b: CoarseGrainedSchedulerBackend =>
b.killExecutors(executorIds, replace = false, force = true).nonEmpty
case _ =>
logWarning("Killing executors is only supported in coarse-grained mode")
false
}
}
> Expose the kill Executor in Yarn Mode
> -
>
> Key: SPARK-18798
> URL: https://issues.apache.org/jira/browse/SPARK-18798
> Project: Spark
> Issue Type: Improvement
>Reporter: Narendra
>
> Expose the kill Executor in Yarn Mode
> I can see spark already has exposed the kill Executor method through spark
> context for Mesos , fi spark can expose the same method for Yarn it will
> good feature if some want to test application stability by killing randomly
> Executor
> I see spark have kill Executor in YarnAllocator, it won't we much time
> consuming to expose this , anyone can work I can also
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18798) Expose the kill Executor in Yarn Mode
[ https://issues.apache.org/jira/browse/SPARK-18798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736048#comment-15736048 ] Narendra commented on SPARK-18798: -- I can only see the API to kill the application not executors , can you please point to me the location or URL , it will be helpful and i will validate > Expose the kill Executor in Yarn Mode > - > > Key: SPARK-18798 > URL: https://issues.apache.org/jira/browse/SPARK-18798 > Project: Spark > Issue Type: Improvement >Reporter: Narendra > > Expose the kill Executor in Yarn Mode > I can see spark already has exposed the kill Executor method through spark > context for Mesos , fi spark can expose the same method for Yarn it will > good feature if some want to test application stability by killing randomly > Executor > I see spark have kill Executor in YarnAllocator, it won't we much time > consuming to expose this , anyone can work I can also -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18770) Current Spark Master branch missing yarn module in pom
[ https://issues.apache.org/jira/browse/SPARK-18770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733834#comment-15733834 ] Narendra commented on SPARK-18770: -- if even i have close this is available in main pom > Current Spark Master branch missing yarn module in pom > -- > > Key: SPARK-18770 > URL: https://issues.apache.org/jira/browse/SPARK-18770 > Project: Spark > Issue Type: Bug >Reporter: Narendra >Priority: Minor > > Current Spark Master branch missing yarn module in pom , because of if some > trying build is not able to build locally > I have added that module in pom because yarn -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18798) Expose the kill Executor in Yarn Mode
Narendra created SPARK-18798: Summary: Expose the kill Executor in Yarn Mode Key: SPARK-18798 URL: https://issues.apache.org/jira/browse/SPARK-18798 Project: Spark Issue Type: Improvement Reporter: Narendra Expose the kill Executor in Yarn Mode I can see spark already has exposed the kill Executor method through spark context for Mesos , fi spark can expose the same method for Yarn it will good feature if some want to test application stability by killing randomly Executor I see spark have kill Executor in YarnAllocator, it won't we much time consuming to expose this , anyone can work I can also -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-18770) Current Spark Master branch missing yarn module in pom
[ https://issues.apache.org/jira/browse/SPARK-18770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Narendra closed SPARK-18770. > Current Spark Master branch missing yarn module in pom > -- > > Key: SPARK-18770 > URL: https://issues.apache.org/jira/browse/SPARK-18770 > Project: Spark > Issue Type: Bug >Reporter: Narendra >Priority: Minor > > Current Spark Master branch missing yarn module in pom , because of if some > trying build is not able to build locally > I have added that module in pom because yarn -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18770) Current Spark Master branch missing yarn module in pom
[ https://issues.apache.org/jira/browse/SPARK-18770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15729668#comment-15729668 ] Narendra commented on SPARK-18770: -- SPARK-18770 i have pull request to resolve this > Current Spark Master branch missing yarn module in pom > -- > > Key: SPARK-18770 > URL: https://issues.apache.org/jira/browse/SPARK-18770 > Project: Spark > Issue Type: Bug >Reporter: Narendra >Priority: Minor > > Current Spark Master branch missing yarn module in pom , because of if some > trying build is not able to build locally > I have added that module in pom because yarn -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-18770) Current Spark Master branch missing yarn module in pom
Narendra created SPARK-18770: Summary: Current Spark Master branch missing yarn module in pom Key: SPARK-18770 URL: https://issues.apache.org/jira/browse/SPARK-18770 Project: Spark Issue Type: Bug Reporter: Narendra Priority: Minor Current Spark Master branch missing yarn module in pom , because of if some trying build is not able to build locally I have added that module in pom because yarn -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
