[GitHub] spark issue #19168: [SPARK-21956][CORE] Fetch up to max bytes when buf reall...

2017-09-27 Thread caneGuy 
Github user caneGuy commented on the issue:

https://github.com/apache/spark/pull/19168
  
Thanks @cloud-fan i will close this pr.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19168: [SPARK-21956][CORE] Fetch up to max bytes when buf reall...

2017-09-27 Thread cloud-fan 
Github user cloud-fan commented on the issue:

https://github.com/apache/spark/pull/19168
  
Technically speaking, `maxBytesInFlight` controls how many bytes are being 
transferred in network, so we should decrease `bytesInFlight` once we get data 
from the result queue. The actual memory usage usually beyond 
`maxBytesInFlight`, because the network transfer needs memory, and the fetched 
data which is being processed also needs memory. This is not a big problem 
because the worst case is we consume `maxBytesInFlight * 2` memory.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19168: [SPARK-21956][CORE] Fetch up to max bytes when buf reall...

2017-09-26 Thread caneGuy 
Github user caneGuy commented on the issue:

https://github.com/apache/spark/pull/19168
  
@cloud-fan  Sorry for bothering you.Could you help review this?Thanks


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19168: [SPARK-21956][CORE] Fetch up to max bytes when buf reall...

2017-09-21 Thread caneGuy 
Github user caneGuy commented on the issue:

https://github.com/apache/spark/pull/19168
  
Sorry for replying so late.
I add some benchmark testing for this pr @kiszk .
And @jerryshao could you help review this pr?Thanks
```
Running benchmark: Benchmark fetch before vs after releasing buffer
  Running case: Testing fetch before releasing!
  Stopped after 10 iterations, 2423 ms
  Running case: Testing fetch after releasing!
  Stopped after 18 iterations, 2036 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17 on Linux 4.4.0-64-generic
Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Benchmark fetch before vs after releasing buffer: Best/Avg Time(ms)
Rate(M/s)   Per Row(ns)   Relative


Testing fetch before releasing! 46 /  242345.0  
 2.9   1.0X
Testing fetch after releasing!  73 /  113215.7  
 4.6   0.6X
```
```
Running benchmark: Benchmark fetch before vs after releasing buffer
  Running case: Testing fetch before releasing!
  Stopped after 10 iterations, 3888 ms
  Running case: Testing fetch after releasing!
  Stopped after 10 iterations, 3970 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17 on Linux 4.4.0-64-generic
Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Benchmark fetch before vs after releasing buffer: Best/Avg Time(ms)
Rate(M/s)   Per Row(ns)   Relative


Testing fetch before releasing!100 /  389157.8  
 6.3   1.0X
Testing fetch after releasing! 151 /  397104.3  
 9.6   0.7X
```
```
Running benchmark: Benchmark fetch before vs after releasing buffer
  Running case: Testing fetch before releasing!
  Stopped after 15 iterations, 2016 ms
  Running case: Testing fetch after releasing!
  Stopped after 14 iterations, 2110 ms

Java HotSpot(TM) 64-Bit Server VM 1.8.0_25-b17 on Linux 4.4.0-64-generic
Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz
Benchmark fetch before vs after releasing buffer: Best/Avg Time(ms)
Rate(M/s)   Per Row(ns)   Relative


Testing fetch before releasing! 43 /  134363.8  
 2.7   1.0X
Testing fetch after releasing!  99 /  151158.1  
 6.3   0.4X
```


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19168: [SPARK-21956][CORE] Fetch up to max bytes when buf reall...

2017-09-08 Thread caneGuy 
Github user caneGuy commented on the issue:

https://github.com/apache/spark/pull/19168
  
Yes you are right @kiszk Benchmark on the way.I will put here later.Thanks 
for your time @kiszk 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19168: [SPARK-21956][CORE] Fetch up to max bytes when buf reall...

2017-09-08 Thread kiszk 
Github user kiszk commented on the issue:

https://github.com/apache/spark/pull/19168
  
IIUC, this change defers sending requests to remote notes until 
`releaseCurrentResultBuffer()` will be called.
Do you have any benchmark results to confirm no performance degradation?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19168: [SPARK-21956][CORE] Fetch up to max bytes when buf reall...

2017-09-08 Thread AmplabJenkins 
Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19168
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org