[jira] [Commented] (SPARK-19227) Typo in `org.apache.spark.internal.config.ConfigEntry`
[ https://issues.apache.org/jira/browse/SPARK-19227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15823366#comment-15823366 ] Biao Ma commented on SPARK-19227: - Its not outdated comment, simply for its sub-class. > Typo in `org.apache.spark.internal.config.ConfigEntry` > --- > > Key: SPARK-19227 > URL: https://issues.apache.org/jira/browse/SPARK-19227 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.0 >Reporter: Biao Ma >Priority: Minor > Labels: easyfix > Fix For: 2.1.1 > > > The parameter `defaultValue` is not exists in class > `org.apache.spark.internal.config.ConfigEntry` but `_defaultValue` in its sub > class `ConfigEntryWithDefault` and .Also there has some un used imports, > should we modify this class? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19227) Typo in `org.apache.spark.internal.config.ConfigEntry`
[ https://issues.apache.org/jira/browse/SPARK-19227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biao Ma updated SPARK-19227: Summary: Typo in `org.apache.spark.internal.config.ConfigEntry` (was: Modify some typo in `ConfigEntry`) > Typo in `org.apache.spark.internal.config.ConfigEntry` > --- > > Key: SPARK-19227 > URL: https://issues.apache.org/jira/browse/SPARK-19227 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.0 >Reporter: Biao Ma >Priority: Minor > Labels: easyfix > Fix For: 2.1.1 > > > The parameter `defaultValue` is not exists in class > `org.apache.spark.internal.config.ConfigEntry` but `_defaultValue` in its sub > class `ConfigEntryWithDefault` and .Also there has some un used imports, > should we modify this class? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19227) Modify some typo in `ConfigEntry`
[ https://issues.apache.org/jira/browse/SPARK-19227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biao Ma updated SPARK-19227: Description: The parameter `defaultValue` is not exists in class `org.apache.spark.internal.config.ConfigEntry` but `_defaultValue` in its sub class `ConfigEntryWithDefault` and .Also there has some un used imports, should we modify this class? (was: The parameter `defaultValue` is not exists in class `org.apache.spark.internal.config.ConfigEntry`, we should remove its annotation.) Issue Type: Bug (was: Documentation) Summary: Modify some typo in `ConfigEntry` (was: Remove non-existent parameter annotation in `ConfigEntry`) > Modify some typo in `ConfigEntry` > -- > > Key: SPARK-19227 > URL: https://issues.apache.org/jira/browse/SPARK-19227 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.0 >Reporter: Biao Ma >Priority: Minor > Labels: easyfix > Fix For: 2.1.1 > > > The parameter `defaultValue` is not exists in class > `org.apache.spark.internal.config.ConfigEntry` but `_defaultValue` in its sub > class `ConfigEntryWithDefault` and .Also there has some un used imports, > should we modify this class? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19227) Remove non-existent parameter annotation in `ConfigEntry`
Biao Ma created SPARK-19227: --- Summary: Remove non-existent parameter annotation in `ConfigEntry` Key: SPARK-19227 URL: https://issues.apache.org/jira/browse/SPARK-19227 Project: Spark Issue Type: Documentation Components: Spark Core Affects Versions: 2.1.0 Reporter: Biao Ma Priority: Minor Fix For: 2.1.1 The parameter `defaultValue` is not exists in class `org.apache.spark.internal.config.ConfigEntry`, we should remove its annotation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431070#comment-15431070 ] Biao Ma commented on SPARK-16593: - I had made new commits. > Provide a pre-fetch mechanism to accelerate shuffle stage. > -- > > Key: SPARK-16593 > URL: https://issues.apache.org/jira/browse/SPARK-16593 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Biao Ma >Priority: Minor > Labels: features > > Currently, the `NettyBlockRpcServer` will reading data through BlockManager, > while the block is not cached in memory, the data should be read from DISK > first, then into MEM. I wonder if we implement a mechanism add a message > contains the blockIds that the same as the openBlock message but one loop > ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer > to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387400#comment-15387400 ] Biao Ma commented on SPARK-16593: - There is some error in this PR, I'll create another later. > Provide a pre-fetch mechanism to accelerate shuffle stage. > -- > > Key: SPARK-16593 > URL: https://issues.apache.org/jira/browse/SPARK-16593 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Biao Ma >Priority: Minor > Labels: features > > Currently, the `NettyBlockRpcServer` will reading data through BlockManager, > while the block is not cached in memory, the data should be read from DISK > first, then into MEM. I wonder if we implement a mechanism add a message > contains the blockIds that the same as the openBlock message but one loop > ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer > to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biao Ma updated SPARK-16593: Target Version/s: (was: 2.1.0) > Provide a pre-fetch mechanism to accelerate shuffle stage. > -- > > Key: SPARK-16593 > URL: https://issues.apache.org/jira/browse/SPARK-16593 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Biao Ma >Priority: Minor > Labels: features > > Currently, the `NettyBlockRpcServer` will reading data through BlockManager, > while the block is not cached in memory, the data should be read from DISK > first, then into MEM. I wonder if we implement a mechanism add a message > contains the blockIds that the same as the openBlock message but one loop > ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer > to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biao Ma updated SPARK-16593: Summary: Provide a pre-fetch mechanism to accelerate shuffle stage. (was: a) > Provide a pre-fetch mechanism to accelerate shuffle stage. > -- > > Key: SPARK-16593 > URL: https://issues.apache.org/jira/browse/SPARK-16593 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Biao Ma >Priority: Minor > Labels: features > > Currently, the `NettyBlockRpcServer` will reading data through BlockManager, > while the block is not cached in memory, the data should be read from DISK > first, then into MEM. I wonder if we implement a mechanism add a message > contains the blockIds that the same as the openBlock message but one loop > ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer > to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-16593) a
[ https://issues.apache.org/jira/browse/SPARK-16593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Biao Ma updated SPARK-16593: Summary: a (was: Provide a pre-fetch mechanism to accelerate shuffle stage.) > a > - > > Key: SPARK-16593 > URL: https://issues.apache.org/jira/browse/SPARK-16593 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Biao Ma >Priority: Minor > Labels: features > > Currently, the `NettyBlockRpcServer` will reading data through BlockManager, > while the block is not cached in memory, the data should be read from DISK > first, then into MEM. I wonder if we implement a mechanism add a message > contains the blockIds that the same as the openBlock message but one loop > ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer > to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-16593) Provide a pre-fetch mechanism to accelerate shuffle stage.
Biao Ma created SPARK-16593: --- Summary: Provide a pre-fetch mechanism to accelerate shuffle stage. Key: SPARK-16593 URL: https://issues.apache.org/jira/browse/SPARK-16593 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Biao Ma Priority: Minor Currently, the `NettyBlockRpcServer` will reading data through BlockManager, while the block is not cached in memory, the data should be read from DISK first, then into MEM. I wonder if we implement a mechanism add a message contains the blockIds that the same as the openBlock message but one loop ahead, then the `NettyBlockRpcServer ` will load the block ready for transfer to the reduce side. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-16587) The annotation for the abstract Class 'BlockTransferMessage' missing several kind of messages.
Biao Ma created SPARK-16587: --- Summary: The annotation for the abstract Class 'BlockTransferMessage' missing several kind of messages. Key: SPARK-16587 URL: https://issues.apache.org/jira/browse/SPARK-16587 Project: Spark Issue Type: Documentation Reporter: Biao Ma Now, the Messages used by shuffle: OpenBlock, UploadBlock, RegisterExecutor has a simple introduction for the usage, while the other three (StreamHandle, RegisterDriver, HearBeat) haven' t. I'd like to add them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15477) HiveContext is private[hive] and not accessible to users.
[ https://issues.apache.org/jira/browse/SPARK-15477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15296318#comment-15296318 ] Biao Ma commented on SPARK-15477: - Thx Owen > HiveContext is private[hive] and not accessible to users. > -- > > Key: SPARK-15477 > URL: https://issues.apache.org/jira/browse/SPARK-15477 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.0.0 >Reporter: Doug Balog > > In 2.0 org.apache.spark.sql.hive.HiveContext was mark deprecated but should > still be accessible from user programs. It is not since, its marked at > `private[hive]` -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
