Github user yucai commented on the issue:
https://github.com/apache/spark/pull/10942
@gatorsmile, seems like getBuckets() is removed by below PR, which makes
this feture not work any more, could you kindly help check?
[SPARK-14535][SQL] Remove buildInternalScan from FileFormat
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14481
@maropu, I am doing some refactor recently, will update it soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/15199
Thanks all, it should be fixed in master only, my mistake.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/15199
[SPARK-17635][SQL] Remove hardcode "agg_plan" in HashAggregateExec
## What changes were proposed in this pull request?
"agg_plan" are hardcoded in HashAggregateExec,
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r77480256
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -50,33 +50,98 @@ private[spark] class DiskBlockManager(conf: SparkConf
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r77478689
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -50,33 +50,98 @@ private[spark] class DiskBlockManager(conf: SparkConf
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r77473963
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -50,33 +50,98 @@ private[spark] class DiskBlockManager(conf: SparkConf
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r76972643
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -50,35 +50,98 @@ private[spark] class DiskBlockManager(conf: SparkConf
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r76942560
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -50,35 +50,98 @@ private[spark] class DiskBlockManager(conf: SparkConf
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r76935911
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -50,35 +50,98 @@ private[spark] class DiskBlockManager(conf: SparkConf
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14481
@hvanhovell
**Benchmark Result**
**Summary**
We benchmark sortagg code gen with real custormers cases, and it improves
x6 when aggregating without keys, improves x1.18 when
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14481
@chenghao-intel Hao, kindly take a look at.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14481
Generated code example, **not for code view yet**.
```
scala> Seq(("a", "10"), ("b", "1"), ("b", "2"), ("c", &
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14481
Generated code example, **not for code review yet**
```
scala> Seq(("a", "3"), ("b", "20"), ("b", "2")).toDF("k",
&quo
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14481
@hvanhovell thanks very much for the advice, yes, I will post the benchmark
results first.
And it is WIP, I will post a generated codes, but kindly not review the
codes details at present, I am
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14481
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/14481
[WIP][SPARK-16844][SQL] Generate code for sort based aggregation
## This PR is WIP, kindly review it later.
You can merge this pull request into a Git repository by running:
$ git
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14366
@rxin this PR has passed tests, could you kindly take a look at? Much
thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user yucai commented on the issue:
https://github.com/apache/spark/pull/14366
@chenghao-intel @adrian-wang Kindly help review.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/14366
[SPARK-16732][SQL] Remove unused codes in
subexpressionEliminationForWholeStageCodegen
## What changes were proposed in this pull request?
Some codes in
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/13204#issuecomment-222431501
Dear Davies,
Sometimes, it could be hard for me to recognize the boundary of whole stage
codegen,
for example, in below SMJ, there are 3 separate whole
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/13204#issuecomment-220505548
Hi Davis,
In your comments "Hopefully we do need to pay much attention on this", do
you mean "we do not need to pay much attention..."?
--
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10225#issuecomment-208193697
@JoshRosen @rxin
Dear committers:
In China, most of companies are still using HDDs as external storage, the
IO bottleneck is quite obvious in shuffle, we make
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/11718
[MINOR][TEST][SQL] Remove useless "expected" parameter
## What changes were proposed in this pull request?
Remove useless "expected" parameter in MathFu
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/11212#issuecomment-186077126
@rxin I tried your suggestion like creating PowDecimal for Decimal
specially like below:
```
case class PowDecimal(left: Expression, right: Expression
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/11212#issuecomment-185717402
OK, let me try this implementation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/11212#discussion_r53275413
--- Diff:
sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter.java
---
@@ -170,6 +170,7 @@ public void write(int
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/11212#issuecomment-184495551
@adrian-wang could you help review? Much thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/11212
[WIP][SQL] Decimal datatype support for pow
Decimal datatype support for pow
- when base is Decimal and exponent is integer(Byte, Short, Int), return
Decimal
- otherwise, return Double
You
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10225#issuecomment-172698659
@rxin @JoshRosen, the PR has been ready, could you kindly help review?
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10526#issuecomment-170770940
@JoshRosen @andrewor14 could you kindly help retest?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10526#issuecomment-170767427
One failure is about flaky streaming test. I retest InputStreamsSuite
locally, no failure.
```
[info] - socket input stream *** FAILED *** (3 seconds, 978
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10526#issuecomment-170462047
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10526#issuecomment-170419313
Sorry, I miss the conversation! I will update it right now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r48709207
--- Diff: docs/configuration.md ---
@@ -926,6 +926,21 @@ Apart from these, the following properties are also
available, and may be useful
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r48709047
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -53,35 +53,98 @@ private[spark] class DiskBlockManager(blockManager
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r48709004
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -53,35 +53,98 @@ private[spark] class DiskBlockManager(blockManager
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r48708933
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -53,35 +53,98 @@ private[spark] class DiskBlockManager(blockManager
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r48708534
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -53,35 +53,98 @@ private[spark] class DiskBlockManager(blockManager
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10526#issuecomment-168449205
Sure, I will update the codes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/10526
[SPARK-12582][shuffle]IndexShuffleBlockResolverSuite fails in windows
[SPARK-12582][shuffle]IndexShuffleBlockResolverSuite fails in windows
* IndexShuffleBlockResolverSuite fails in windows
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10291#issuecomment-164600078
Merged already, so close this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user yucai closed the pull request at:
https://github.com/apache/spark/pull/10291
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10291#issuecomment-164403257
@yhuai could you help trigger testing again?
I'm from Intel Spark team(mgr: Hao/Grace), working on Spark SQL and x86
optimization etc., could you kindly add me
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/10291
[SPARK-12275][SQL] No plan for BroadcastHint in some condition - 1.5
backport
backport #10265 to branch 1.5
When SparkStrategies.BasicOperators's "case BroadcastHint(child) =>
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10265#issuecomment-164365408
@yhuai sure, I will do that ASAP.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10265#issuecomment-164339684
@andrewor14 , @yhuai could you kindly help re-trigger testing? Much thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10265#issuecomment-164325959
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10265#issuecomment-164233358
retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10265#issuecomment-164213095
Weird, I find no difference for
org.apache.spark.sql.sources.JsonHadoopFsRelationSuite#"test all data types -
TimestampType" in Jekins's test report.
Github user yucai commented on the pull request:
https://github.com/apache/spark/pull/10265#issuecomment-163856742
@zzcclp Good to know :).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/10265
[SPARK-12275][SQL] No plan for BroadcastHint in some condition
https://issues.apache.org/jira/browse/SPARK-12275
You can merge this pull request into a Git repository by running:
$ git pull
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r47182102
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -80,6 +77,65 @@ private[spark] class DiskBlockManager(blockManager
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r47182068
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -80,6 +77,65 @@ private[spark] class DiskBlockManager(blockManager
Github user yucai commented on a diff in the pull request:
https://github.com/apache/spark/pull/10225#discussion_r47181555
--- Diff:
core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala ---
@@ -80,6 +77,65 @@ private[spark] class DiskBlockManager(blockManager
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/10225
[SPARK-12196][Core] Store blocks in storage devices with hierarchy way
https://issues.apache.org/jira/browse/SPARK-12196
You can merge this pull request into a Git repository by running:
$ git
Github user yucai closed the pull request at:
https://github.com/apache/spark/pull/10192
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user yucai opened a pull request:
https://github.com/apache/spark/pull/10192
[SPARK-12196][Core]Store blocks in storage devices with hierarchy way
https://issues.apache.org/jira/browse/SPARK-12196
You can merge this pull request into a Git repository by running:
$ git
201 - 258 of 258 matches
Mail list logo
