[jira] [Commented] (SPARK-8487) Update reduceByKeyAndWindow docs to highlight that filtering Function must be used
[ https://issues.apache.org/jira/browse/SPARK-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001737#comment-15001737 ] Chris Howard commented on SPARK-8487: - Hi [~tdas] - can you clarify this doc change? The def for reduceByKeyAndWindow shows filterFunc as optional - not sure how this relates to your comment "highlight that filtering Function must be used". The documentation is missing the overloaded version of reduceByKeyAndWindow that allows for a filtering function to be specified. > Update reduceByKeyAndWindow docs to highlight that filtering Function must be > used > -- > > Key: SPARK-8487 > URL: https://issues.apache.org/jira/browse/SPARK-8487 > Project: Spark > Issue Type: Improvement > Components: Documentation >Reporter: Tathagata Das >Assignee: Tathagata Das >Priority: Critical > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9597) Add Spark Streaming + MQTT Integration Guide
[ https://issues.apache.org/jira/browse/SPARK-9597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001751#comment-15001751 ] Chris Howard commented on SPARK-9597: - I have started working on this. Thanks > Add Spark Streaming + MQTT Integration Guide > > > Key: SPARK-9597 > URL: https://issues.apache.org/jira/browse/SPARK-9597 > Project: Spark > Issue Type: Documentation > Components: Documentation >Reporter: Prabeesh K > > Add Spark Streaming + MQTT Integration Guide like > [Spark Streaming + Flume Integration > Guide|http://spark.apache.org/docs/latest/streaming-flume-integration.html] > [Spark Streaming + Kinesis > Integration|http://spark.apache.org/docs/latest/streaming-kinesis-integration.html] > [Spark Streaming + Kafka Integration > Guide|http://spark.apache.org/docs/latest/streaming-kafka-integration.html] -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11154) make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options
[ https://issues.apache.org/jira/browse/SPARK-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001316#comment-15001316 ] Chris Howard commented on SPARK-11154: -- Hi [~tgraves] - this overlaps to a point with [SPARK-3374|https://issues.apache.org/jira/browse/SPARK-3374] and [SPARK-4408|https://issues.apache.org/jira/browse/SPARK-4408] I would agree with [~srowen] that 2.0 provides an opportunity for house keeping / consolidation and there is scope to clean up the config / args for cluster and client modes. I would prefer not to create new configs and would rather stick with the current naming and support k | m | g modifiers unless somebody has a strong view on what to rename the existing configs. > make specificaition spark.yarn.executor.memoryOverhead consistent with > typical JVM options > -- > > Key: SPARK-11154 > URL: https://issues.apache.org/jira/browse/SPARK-11154 > Project: Spark > Issue Type: Improvement > Components: Documentation, Spark Submit >Reporter: Dustin Cote >Priority: Minor > > spark.yarn.executor.memoryOverhead is currently specified in megabytes by > default, but it would be nice to allow users to specify the size as though it > were a typical -Xmx option to a JVM where you can have 'm' and 'g' appended > to the end to explicitly specify megabytes or gigabytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-11154) make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options
[ https://issues.apache.org/jira/browse/SPARK-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1544#comment-1544 ] Chris Howard edited comment on SPARK-11154 at 11/11/15 7:13 AM: On review it looks like the following conf values need changes - spark.yarn.driver.memoryOverhead - spark.yarn.am.memoryOverhead - spark.yarn.am.memory - spark.yarn.executor.memoryOverhead Changes required to - yarn/ClientArguments.scala - yarn/Client.scala - yarn/YarnAllocator.scala org.apache.spark.deploy.ClientArguments.scala uses org.apache.spark.util.MemoryParam to convert to MB so propose same. If we were to replicate the functionality of deploy.ClientArguments.scala we should also consider allowing setting via command line args. Propose we add new config values to ensure backward compatibility. Anyone issues with naming as follows? - spark.yarn.driver.memoryOverhead -> spark.yarn.driver.memOverhead - spark.yarn.am.memoryOverhead -> spark.yarn.am.memOverhead - spark.yarn.am.memory -> spark.yarn.am.mem - spark.yarn.executor.memoryOverhead -> spark.yarn.executor.memOverhead Docs will also need updating to reflect fact that original option is MB only and new options support k | m | g variants. Happy to run with this if no objections. was (Author: kitard): On review it looks like the following conf values need changes - spark.yarn.driver.memoryOverhead - spark.yarn.am.memoryOverhead - spark.yarn.am.memory - spark.yarn.executor.memoryOverhead Changes required to - yarn/ClientArguments.scala - yarn/Client.scala - yarn/YarnAllocator.scala org.apache.spark.deploy.ClientArguments.scala uses org.apache.spark.util.MemoryParam to convert to MB so propose same. If we were to replicate the functionality of deploy.ClientArguments.scala we should also consider allowing setting via command line args. Propose we add new config values to ensure backward compatibility. Anyone issues with naing as follows? - spark.yarn.driver.memoryOverhead -> spark.yarn.driver.memOverhead - spark.yarn.am.memoryOverhead -> spark.yarn.am.memOverhead - spark.yarn.am.memory -> spark.yarn.am.mem - spark.yarn.executor.memoryOverhead -> spark.yarn.executor.memOverhead Docs will also need updating to reflect fact that original option is MB only and new options support k | m | g variants. Happy to run with this if no objections. > make specificaition spark.yarn.executor.memoryOverhead consistent with > typical JVM options > -- > > Key: SPARK-11154 > URL: https://issues.apache.org/jira/browse/SPARK-11154 > Project: Spark > Issue Type: Improvement > Components: Documentation, Spark Submit >Reporter: Dustin Cote >Priority: Minor > > spark.yarn.executor.memoryOverhead is currently specified in megabytes by > default, but it would be nice to allow users to specify the size as though it > were a typical -Xmx option to a JVM where you can have 'm' and 'g' appended > to the end to explicitly specify megabytes or gigabytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11154) make specificaition spark.yarn.executor.memoryOverhead consistent with typical JVM options
[ https://issues.apache.org/jira/browse/SPARK-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1544#comment-1544 ] Chris Howard commented on SPARK-11154: -- On review it looks like the following conf values need changes - spark.yarn.driver.memoryOverhead - spark.yarn.am.memoryOverhead - spark.yarn.am.memory - spark.yarn.executor.memoryOverhead Changes required to - yarn/ClientArguments.scala - yarn/Client.scala - yarn/YarnAllocator.scala org.apache.spark.deploy.ClientArguments.scala uses org.apache.spark.util.MemoryParam to convert to MB so propose same. If we were to replicate the functionality of deploy.ClientArguments.scala we should also consider allowing setting via command line args. Propose we add new config values to ensure backward compatibility. Anyone issues with naing as follows? - spark.yarn.driver.memoryOverhead -> spark.yarn.driver.memOverhead - spark.yarn.am.memoryOverhead -> spark.yarn.am.memOverhead - spark.yarn.am.memory -> spark.yarn.am.mem - spark.yarn.executor.memoryOverhead -> spark.yarn.executor.memOverhead Docs will also need updating to reflect fact that original option is MB only and new options support k | m | g variants. Happy to run with this if no objections. > make specificaition spark.yarn.executor.memoryOverhead consistent with > typical JVM options > -- > > Key: SPARK-11154 > URL: https://issues.apache.org/jira/browse/SPARK-11154 > Project: Spark > Issue Type: Improvement > Components: Documentation, Spark Submit >Reporter: Dustin Cote >Priority: Minor > > spark.yarn.executor.memoryOverhead is currently specified in megabytes by > default, but it would be nice to allow users to specify the size as though it > were a typical -Xmx option to a JVM where you can have 'm' and 'g' appended > to the end to explicitly specify megabytes or gigabytes. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
