[GitHub] spark pull request: The script copy_api_dirs.rb should be run anyw...

[KaiXinXiaoLei]

Github user KaiXinXiaoLei commented on the pull request:

https://github.com/apache/spark/pull/6820#issuecomment-112313096
  
@JoshRosen  I have create a JIRA :SPARK-8388 
(https://issues.apache.org/jira/browse/SPARK-8388).Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Scala version switching build enhancements

[ScrapCodes]

Github user ScrapCodes commented on the pull request:

https://github.com/apache/spark/pull/6832#issuecomment-112308999
  
I am Ok with 1.

Is there a reason we should restrict ourselves to GNU SED ? I suppose our 
scripts were posix complaint. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: spark ssc.textFileStream returns empty

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/6837#issuecomment-112309026
  
Could you close this PR ? It has a bunch of problems, and you need to 
review https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: spark ssc.textFileStream returns empty

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6837#issuecomment-112308311
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: spark ssc.textFileStream returns empty

[sduchh]

GitHub user sduchh opened a pull request:

https://github.com/apache/spark/pull/6837

spark ssc.textFileStream returns empty

I'm trying to stream from an hdfs folder (sparkstreamining) I tried:
JavaDStream sourceInfors 
=jssc.textFileStream("hdfs://172.19.23.24:8020/user/input");  
sourceInfors.print();
As the documentation for textFileStream said:Create an input stream that 
monitors a Hadoop-compatible filesystem for new files and reads them as text 
files (using key as LongWritable, value as Text and input format as 
TextInputFormat). Files must be written to the monitored directory by "moving" 
them from another location within the same file system. File names starting 
with . are ignored.

First, I run this application in local mode, and  there is no files in this 
directory.Second,I move one file from the same file system to the diectory 
"input", however, nothing returned.  Can anyone help solve this problem? Thanks.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/spark branch-1.3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/6837.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6837


commit d9e141cb7b973e1fa21f0144e08d7ffa976c0b39
Author: Volodymyr Lyubinets 
Date:   2015-03-12T07:55:26Z

[SPARK-6296] [SQL] Added equals to Column

Author: Volodymyr Lyubinets 

Closes #4988 from vlyubin/columncomp and squashes the following commits:

92d7c8f [Volodymyr Lyubinets] Added equals to Column

(cherry picked from commit 25b71d8c15572f0f2b951c827c169f8c65f726ad)
Signed-off-by: Reynold Xin 

commit 850e69451e3ea501a527c81d10fd4b34e2979a2a
Author: Davies Liu 
Date:   2015-03-12T08:34:38Z

[SPARK-6294] fix hang when call take() in JVM on PythonRDD

The Thread.interrupt() can not terminate the thread in some cases, so we 
should not wait for the writerThread of PythonRDD.

This PR also ignore some exception during clean up.

cc JoshRosen mengxr

Author: Davies Liu 

Closes #4987 from davies/fix_take and squashes the following commits:

4488f1a [Davies Liu] fix hang when call take() in JVM on PythonRDD

(cherry picked from commit 712679a7b447346a365b38574d7a86d56a93f767)
Signed-off-by: Xiangrui Meng 

commit 23069bd02af342668a6eadecb545f529acb843b6
Author: Joseph K. Bradley 
Date:   2015-03-12T23:46:29Z

[mllib] [python] Add LassoModel to __all__ in regression.py

Add LassoModel to __all__ in regression.py

LassoModel does not show up in Python docs

This should be merged into branch-1.3 and master.

Author: Joseph K. Bradley 

Closes #4970 from jkbradley/SPARK-6253 and squashes the following commits:

c2cb533 [Joseph K. Bradley] Add LassoModel to __all__ in regression.py

(cherry picked from commit 17c309c87e78da145dc358514150ec5700eed8f0)
Signed-off-by: Xiangrui Meng 

commit dc287f38f1cc192b7fa6ec0e83b36254f1cfec10
Author: Cheng Lian 
Date:   2015-03-13T13:34:50Z

[SPARK-5310] [SQL] [DOC] Parquet section for the SQL programming guide

Also fixed a bunch of minor styling issues.


[https://reviewable.io/review_button.png"; height=40 alt="Review 
on Reviewable"/>](https://reviewable.io/reviews/apache/spark/5001)


Author: Cheng Lian 

Closes #5001 from liancheng/parquet-doc and squashes the following commits:

89ad3db [Cheng Lian] Addresses @rxin's comments
7eb6955 [Cheng Lian] Docs for the new Parquet data source
415eefb [Cheng Lian] Some minor formatting improvements

(cherry picked from commit 69ff8e8cfbecd81fd54100c4dab332c3bc992316)
Signed-off-by: Cheng Lian 

commit 214f68103219317416e2278e80b8fc0fb5a616f4
Author: Xiangrui Meng 
Date:   2015-03-13T17:27:28Z

[SPARK-6278][MLLIB] Mention the change of objective in linear regression

As discussed in the RC3 vote thread, we should mention the change of 
objective in linear regression in the migration guide. srowen

Author: Xiangrui Meng 

Closes #4978 from mengxr/SPARK-6278 and squashes the following commits:

fb3bbe6 [Xiangrui Meng] mention regularization parameter
bfd6cff [Xiangrui Meng] Merge remote-tracking branch 'apache/master' into 
SPARK-6278
375fd09 [Xiangrui Meng] address Sean's comments
f87ae71 [Xiangrui Meng] mention step size change

(cherry picked from commit 7f13434a5c52b815c584ec773ab0e5df1a35ea86)
Signed-off-by: Xiangrui Meng 

commit dbee7e16c7434326cce6f6d5ab494093c60ee097
Author: Sean Owen 
Date:   2015-02-26T20:56:54Z

SPARK-4704 [CORE] SparkSubmitDriverBootstrap doesn't flush output

Join on output threads to make sure any lingering output from process 
reaches stdout

[GitHub] spark pull request: [SPARK-8129] [CORE] [Sec] Pass auth secrets to...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/6774


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8367][Streaming]Add a limit for 'spark....

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/6818


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Scala version switching build enhancements

[srowen]

Github user srowen commented on the pull request:

https://github.com/apache/spark/pull/6832#issuecomment-112306270
  
@mallman Have a look at 
https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark please

CC @ScrapCodes 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8309] [CORE] Support for more than 12M ...

[srowen]

Github user srowen commented on a diff in the pull request:

https://github.com/apache/spark/pull/6763#discussion_r32493399
  
--- Diff: 
core/src/main/scala/org/apache/spark/util/collection/OpenHashSet.scala ---
@@ -223,6 +224,8 @@ class OpenHashSet[@specialized(Long, Int) T: ClassTag](
*/
   private def rehash(k: T, allocateFunc: (Int) => Unit, moveFunc: (Int, 
Int) => Unit) {
 val newCapacity = _capacity * 2
+require(newCapacity <= OpenHashSet.MAX_CAPACITY,
--- End diff --

I agree this is the theoretically largest number of elements that can be in 
the set. The failure will occur any time that twice the grow threshold exceeds 
`MAX_CAPACITY`, which can happen when the collection is less full than this. So 
I am actually not sure what's clearer here. Up to you.

I think we still have a little problem here, because when capacity reaches 
2^30, twice that number becomes negative and `newCapacity <= 
OpenHashSet.MAX_CAPACITY` is true, still, because of overflow. Check whether 
the existing capacity is `<= OpenHashSet.MAX_CAPACITY / 2` first?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7184] [SQL] enable codegen by default

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/6726


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7184] [SQL] enable codegen by default

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/6726#issuecomment-112304969
  
Going to merge this one.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8387][WEBUI] Only show 4096 bytes conte...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6834#issuecomment-112304147
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8387][WEBUI] Only show 4096 bytes conte...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6834#issuecomment-112304047
  
  [Test build #34970 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34970/console)
 for   PR 6834 at commit 
[`744212f`](https://github.com/apache/spark/commit/744212fdab1bbe249e56f30f9065920fa6a558c9).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8221][SQL]Add pmod function

[zhichao-li]

Github user zhichao-li commented on the pull request:

https://github.com/apache/spark/pull/6783#issuecomment-112303755
  
@adrian-wang @chenghao-intel , wondering what type of api we should reserve 
regarding to this kind of api ` def pmod(leftName: String, r: Integer): Column` 
?   I guess it would be a good thing that enable such input: ` select 
pmod(columName, 10) .. from ...` 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8206][SQL]Add function round [WIP]

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6836#issuecomment-112303723
  
  [Test build #34972 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34972/consoleFull)
 for   PR 6836 at commit 
[`4069bba`](https://github.com/apache/spark/commit/4069bba95e18729498e9f97d2c23ed579b5b1b83).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8206][SQL]Add function round [WIP]

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6836#issuecomment-112303491
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8206][SQL]Add function round [WIP]

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6836#issuecomment-112303476
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8213][SQL]Add function factorial[WIP]

[zhichao-li]

Github user zhichao-li commented on the pull request:

https://github.com/apache/spark/pull/6822#issuecomment-112303316
  
cc @chenghao-intel 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8206][SQL]Add function round [WIP]

[zhichao-li]

GitHub user zhichao-li opened a pull request:

https://github.com/apache/spark/pull/6836

[SPARK-8206][SQL]Add function round [WIP]



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/zhichao-li/spark round

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/6836.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6836


commit 4069bba95e18729498e9f97d2c23ed579b5b1b83
Author: zhichao.li 
Date:   2015-06-16T05:29:15Z

add function round




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8206][SQL]Add function round [WIP]

[zhichao-li]

Github user zhichao-li commented on the pull request:

https://github.com/apache/spark/pull/6836#issuecomment-112303194
  
cc @chenghao-intel 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8379][SQL]avoid speculative tasks write...

[jeanlyn]

Github user jeanlyn commented on a diff in the pull request:

https://github.com/apache/spark/pull/6833#discussion_r32492419
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
 ---
@@ -197,7 +197,6 @@ case class InsertIntoHiveTable(
   table.hiveQlTable.getPartCols().foreach { entry =>
 orderedPartitionSpec.put(entry.getName, 
partitionSpec.get(entry.getName).getOrElse(""))
   }
-  val partVals = 
MetaStoreUtils.getPvals(table.hiveQlTable.getPartCols, partitionSpec)
 
--- End diff --

I think 
https://github.com/apache/spark/pull/5876/files#diff-d579db9a8f27e0bbef37720ab14ec3f6L203
 should remove this code. @marmbrus. Right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8379][SQL]avoid speculative tasks write...

[jeanlyn]

Github user jeanlyn commented on a diff in the pull request:

https://github.com/apache/spark/pull/6833#discussion_r32491951
  
--- Diff: 
sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala
 ---
@@ -197,7 +197,6 @@ case class InsertIntoHiveTable(
   table.hiveQlTable.getPartCols().foreach { entry =>
 orderedPartitionSpec.put(entry.getName, 
partitionSpec.get(entry.getName).getOrElse(""))
   }
-  val partVals = 
MetaStoreUtils.getPvals(table.hiveQlTable.getPartCols, partitionSpec)
 
--- End diff --

This code seems never use,so remove it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7916] [MLlib] MLlib Python doc parity c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6460#issuecomment-112291866
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7916] [MLlib] MLlib Python doc parity c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6460#issuecomment-112291848
  
  [Test build #34968 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34968/console)
 for   PR 6460 at commit 
[`f8deda4`](https://github.com/apache/spark/commit/f8deda47a087fce8ed313865f9b6b3288f3b75e9).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8247][SPARK-8249][SPARK-8252][SPARK-825...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6762#issuecomment-112291731
  
  [Test build #34971 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34971/consoleFull)
 for   PR 6762 at commit 
[`14be1f4`](https://github.com/apache/spark/commit/14be1f425f4e06567d2dd8fd723147d13af84469).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8247][SPARK-8249][SPARK-8252][SPARK-825...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6762#issuecomment-112291662
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8247][SPARK-8249][SPARK-8252][SPARK-825...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6762#issuecomment-112291651
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8161] Set externalBlockStoreInitialized...

[shimingfei]

Github user shimingfei commented on the pull request:

https://github.com/apache/spark/pull/6702#issuecomment-112291544
  
@andrewor14  @aarondav 
Can you take a look? Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8280][SPARK-8281][SQL]Handle NaN, null ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6835#issuecomment-112286482
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8280][SPARK-8281][SQL]Handle NaN, null ...

[yijieshen]

GitHub user yijieshen opened a pull request:

https://github.com/apache/spark/pull/6835

[SPARK-8280][SPARK-8281][SQL]Handle NaN, null and Infinity in math



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/yijieshen/spark nan_infinity_null

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/6835.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6835


commit a06c924e50c390cafdea4feb9a7290658a6749a9
Author: Yijie Shen 
Date:   2015-06-16T04:21:27Z

Enable NaN for acos & asin, handle -Infinity for log

commit 2d1dfc16d1e251cca432b6ef839620ad3e8a9fd5
Author: Yijie Shen 
Date:   2015-06-16T04:34:02Z

Enable udf7, udf_acos, udf_asin




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8247][SPARK-8249][SPARK-8252][SPARK-825...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6762#issuecomment-112284174
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8247][SPARK-8249][SPARK-8252][SPARK-825...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6762#issuecomment-112284158
  
  [Test build #34966 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34966/console)
 for   PR 6762 at commit 
[`852ca7c`](https://github.com/apache/spark/commit/852ca7c802f60d294c49296b98bd29e982696435).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `trait String2StringExpression extends ExpectsInputTypes `
  * `case class Upper(child: Expression) extends UnaryExpression with 
String2StringExpression `
  * `case class Lower(child: Expression) extends UnaryExpression with 
String2StringExpression `
  * `case class StringTrim(child: Expression)`
  * `case class StringTrimLeft(child: Expression)`
  * `case class StringTrimRight(child: Expression)`
  * `case class StringInstr(str: Expression, substr: Expression)`
  * `case class StringLocate(substr: Expression, str: Expression, start: 
Expression)`
  * `case class StringLPad(str: Expression, len: Expression, pad: 
Expression)`
  * `case class StringRPad(str: Expression, len: Expression, pad: 
Expression)`
  * `case class StringFormat(children: Expression*) extends Expression `
  * `case class StringRepeat(str: Expression, times: Expression)`
  * `case class StringReverse(child: Expression) extends UnaryExpression 
with String2StringExpression `
  * `case class StringSpace(child: Expression) extends UnaryExpression with 
ExpectsInputTypes `
  * `case class StringSplit(str: Expression, pattern: Expression)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8387][WEBUI] Only show 4096 bytes conte...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6834#issuecomment-112283191
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8301][SQL] Improve UTF8String substring...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6804#issuecomment-112282692
  
  [Test build #916 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/916/console)
 for   PR 6804 at commit 
[`9f17cc8`](https://github.com/apache/spark/commit/9f17cc80370c612b1cc40801a0d15c1b8f7e283b).
 * This patch **fails Spark unit tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8387][WEBUI] Only show 4096 bytes conte...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6834#issuecomment-112279356
  
  [Test build #34970 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34970/consoleFull)
 for   PR 6834 at commit 
[`744212f`](https://github.com/apache/spark/commit/744212fdab1bbe249e56f30f9065920fa6a558c9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8387][WEBUI] Only show 4096 bytes conte...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6834#issuecomment-112278720
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8387][WEBUI] Only show 4096 bytes conte...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6834#issuecomment-112278709
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112278245
  
  [Test build #34967 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34967/console)
 for   PR 6827 at commit 
[`cdbb089`](https://github.com/apache/spark/commit/cdbb0890f4286650d4a78d9d30b1f3f2457465f1).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112278349
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Only show 4096 bytes content for executor log ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6834#issuecomment-112277949
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Only show 4096 bytes content for executor log ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6834#issuecomment-112277854
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Only show 4096 bytes content for executor log ...

[suyanNone]

GitHub user suyanNone opened a pull request:

https://github.com/apache/spark/pull/6834

Only show 4096 bytes content for executor log instead all



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/suyanNone/spark small-display

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/6834.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6834


commit ed432573d30ef535fca84d1436a034d6a06dfaff
Author: hushan[胡珊] 
Date:   2015-06-02T13:47:24Z

[SPARK][CORE] Avoid to use directbuffer while reading or writing disk_level 
block

commit 539c3e80cc9a3853ed181948b1da698c2a1d1863
Author: hushan[胡珊] 
Date:   2015-06-04T08:35:56Z

Slice in channel read or write

commit 379098b3326fa5a84bae58c138563edc15a028a9
Author: hushan[胡珊] 
Date:   2015-06-04T09:21:05Z

Refine code style

commit e8a7dd6c5135ed258c89df256035c279f0bbdd65
Author: hushan[胡珊] 
Date:   2015-06-11T12:56:28Z

Merge branch 'master' of https://github.com/apache/spark

commit 46d78e1ede7e5e2221730c4a931b83b40fb5d5ed
Author: hushan[胡珊] 
Date:   2015-06-16T03:42:28Z

Merge branch 'master' of https://github.com/apache/spark

commit 1fc28a43396cc5519de3078f50b308b4ab8e278f
Author: hushan[胡珊] 
Date:   2015-06-16T03:44:15Z

[SPARK][Web-UI] Only show 4096 bytes content for executor log




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7555][docs] Add doc for elastic net in ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6504#issuecomment-112273021
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7555][docs] Add doc for elastic net in ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6504#issuecomment-112272958
  
  [Test build #34965 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34965/console)
 for   PR 6504 at commit 
[`706d3f7`](https://github.com/apache/spark/commit/706d3f778a5b6b9ecdc1a316349186c86c8dd896).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `public class LogisticRegressionWithElasticNetExample `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8129] [CORE] [Sec] Pass auth secrets to...

[kanzhang]

Github user kanzhang commented on the pull request:

https://github.com/apache/spark/pull/6774#issuecomment-112272884
  
And I get what you are saying all along. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8234][SQL] misc function: md5

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6779#issuecomment-112272645
  
  [Test build #34964 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34964/console)
 for   PR 6779 at commit 
[`c75bb09`](https://github.com/apache/spark/commit/c75bb0957fca5788a8c3c0dc0f59dedb9c392649).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Md5(child: Expression)`



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8234][SQL] misc function: md5

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6779#issuecomment-112272667
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7555][docs] Add doc for elastic net in ...

[coderxiang]

Github user coderxiang commented on the pull request:

https://github.com/apache/spark/pull/6504#issuecomment-112271606
  
@dbtsai @jkbradley finished the Java and Python sample code.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7916] [MLlib] MLlib Python doc parity c...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6460#issuecomment-112270239
  
  [Test build #34968 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34968/consoleFull)
 for   PR 6460 at commit 
[`f8deda4`](https://github.com/apache/spark/commit/f8deda47a087fce8ed313865f9b6b3288f3b75e9).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7916] [MLlib] MLlib Python doc parity c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6460#issuecomment-112270153
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7916] [MLlib] MLlib Python doc parity c...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6460#issuecomment-112270163
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8221][SQL]Add pmod function

[adrian-wang]

Github user adrian-wang commented on the pull request:

https://github.com/apache/spark/pull/6783#issuecomment-112269008
  
LGTM except the full type APIs


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8221][SQL]Add pmod function

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6783#issuecomment-112267010
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8221][SQL]Add pmod function

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6783#issuecomment-112266986
  
  [Test build #34963 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34963/console)
 for   PR 6783 at commit 
[`88c63ab`](https://github.com/apache/spark/commit/88c63abadb39ff0313d2b1ed09712f3a05120e8c).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class Pmod(left: Expression, right: Expression) extends 
BinaryArithmetic `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8240] [SPARK-8241] [SQL] string functio...

[adrian-wang]

Github user adrian-wang commented on a diff in the pull request:

https://github.com/apache/spark/pull/6775#discussion_r32487610
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala ---
@@ -914,6 +914,42 @@ object functions {
   def ceil(columnName: String): Column = ceil(Column(columnName))
 
   /**
+   * Computes the concat of the given values.
+   *
+   * @group normal_funcs
+   * @since 1.5.0
+   */
+  @scala.annotation.varargs
+  def concat(exprs: Column*): Column = Concat(exprs.map(_.expr): _*)
+
+  /**
+   * Computes the concat of the given column names.
+   *
+   * @group normal_funcs
+   * @since 1.5.0
+   */
+  @scala.annotation.varargs
+  def concat(columnName: String, columnNames: String*): Column =
+concat((columnName +: columnNames).map(Column.apply): _*)
+
+  /**
+   * Computes the concat_ws of the given columns with given separator.
+   *
+   * @group normal_funcs
+   * @since 1.5.0
+   */
+  @scala.annotation.varargs
+  def concat_ws(sep: Column, exprs: Column*): Column = concat_ws(sep, 
array(exprs: _*))
+
+  /**
+   * Computes the concat_ws of the given columns with given separator.
+   *
+   * @group normal_funcs
+   * @since 1.5.0
+   */
+  def concat_ws(sep: Column, exprs: Column): Column = ConcatWS(sep.expr, 
exprs.expr)
--- End diff --

So Hive is not doing exactly the same with what it specifies in its 
language manual then. Let's keep align with the actual Hive.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-7446][MLLIB] Add inverse transform for ...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/6339#issuecomment-112266368
  
@mengxr - should we put this in the parent class or maybe just add a trait 
for transformers which are invertible?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8371][SQL] improve unit test for MaxOf ...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/6825#issuecomment-112266153
  
@cloud-fan There are some changes related to Max/Min in #6726 , is it 
useful for you? It will be good if you could merge them here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-7781][MLLIB] gradient boosted trees.tra...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/6331#issuecomment-112265955
  
cc @jkbradley 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-7780][MLLIB] Intercept in logisticregre...

[holdenk]

Github user holdenk closed the pull request at:

https://github.com/apache/spark/pull/6771


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [Spark-7780][MLLIB] Intercept in logisticregre...

[holdenk]

Github user holdenk commented on the pull request:

https://github.com/apache/spark/pull/6386#issuecomment-112265729
  
So from running with a slightly larger scaling factor than I was initially 
it seems all the approaches are pretty similar in terms of time usage.

origin:
glm-regression, glm-regression --num-trials=10 --inter-trial-wait=3 
--num-partitions=6 --random-seed=5 --num-examples=5 --num-features=1 
--num-iterations=20 --step-size=0.001 --reg-type=l2 --reg-param=0.1 
--optimizer=lbfgs --intercept=0.0 --epsilon=0.1 --loss=l2
Training time: 4.5985, 0.175, 4.259, 4.452, 4.579
Test time: 0.1255, 0.007, 0.117, 0.125, 0.117
Training Set Metric: 33.2738700623, 0.133, 33.0214743257, 33.4757304367, 
33.2006471123
Test Set Metric: 33.1470085163, 0.274, 32.8030691787, 33.372759872, 
33.0449774996
glm-regression, glm-regression --num-trials=10 --inter-trial-wait=3 
--num-partitions=6 --random-seed=5 --num-examples=5 --num-features=1 
--num-iterations=20 --step-size=0.001 --reg-type=l2 --reg-param=0.1 
--optimizer=lbfgs --intercept=0.0 --epsilon=0.1 --loss=elastic-net
Training time: 4.5985, 0.175, 4.259, 4.452, 4.579
Test time: 0.1255, 0.007, 0.117, 0.125, 0.117
Training Set Metric: 33.2738700623, 0.133, 33.0214743257, 33.4757304367, 
33.2006471123
Test Set Metric: 33.1470085163, 0.274, 32.8030691787, 33.372759872, 
33.0449774996

current pr (rt through dataframes):

glm-regression, glm-regression --num-trials=10 --inter-trial-wait=3 
--num-partitions=6 --random-seed=5 --num-examples=5 --num-features=1 
--num-iterations=20 --step-size=0.001 --reg-type=l2 --reg-param=0.1 
--optimizer=lbfgs --intercept=0.0 --epsilon=0.1 --loss=l2
Training time: 4.382, 0.486, 3.937, 5.056, 3.937
Test time: 0.1255, 0.012, 0.114, 0.124, 0.115
Training Set Metric: 33.2738700623, 0.133, 33.0214743257, 33.4757304367, 
33.2006471123
Test Set Metric: 33.1470085163, 0.274, 32.8030691787, 33.372759872, 
33.0449774996
glm-regression, glm-regression --num-trials=10 --inter-trial-wait=3 
--num-partitions=6 --random-seed=5 --num-examples=5 --num-features=1 
--num-iterations=20 --step-size=0.001 --reg-type=l2 --reg-param=0.1 
--optimizer=lbfgs --intercept=0.0 --epsilon=0.1 --loss=elastic-net
Training time: 4.382, 0.486, 3.937, 5.056, 3.937
Test time: 0.1255, 0.012, 0.114, 0.124, 0.115
Training Set Metric: 33.2738700623, 0.133, 33.0214743257, 33.4757304367, 
33.2006471123
Test Set Metric: 33.1470085163, 0.274, 32.8030691787, 33.372759872, 
33.0449774996

pr without the rt through data frames:

glm-regression, glm-regression --num-trials=10 --inter-trial-wait=3 
--num-partitions=6 --random-seed=5 --num-examples=5 --num-features=1 
--num-iterations=20 --step-size=0.001 --reg-type=l2 --reg-param=0.1 
--optimizer=lbfgs --intercept=0.0 --epsilon=0.1 --loss=l2
Training time: 4.3305, 0.374, 4.049, 5.034, 4.049
Test time: 0.1225, 0.011, 0.119, 0.149, 0.119
Training Set Metric: 33.2738700623, 0.133, 33.0214743257, 33.4757304367, 
33.2006471123
Test Set Metric: 33.1470085163, 0.274, 32.8030691787, 33.372759872, 
33.0449774996
glm-regression, glm-regression --num-trials=10 --inter-trial-wait=3 
--num-partitions=6 --random-seed=5 --num-examples=5 --num-features=1 
--num-iterations=20 --step-size=0.001 --reg-type=l2 --reg-param=0.1 
--optimizer=lbfgs --intercept=0.0 --epsilon=0.1 --loss=elastic-net
Training time: 4.3305, 0.374, 4.049, 5.034, 4.049
Test time: 0.1225, 0.011, 0.119, 0.149, 0.119
Training Set Metric: 33.2738700623, 0.133, 33.0214743257, 33.4757304367, 
33.2006471123
Test Set Metric: 33.1470085163, 0.274, 32.8030691787, 33.372759872, 
33.0449774996


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[carsonwang]

Github user carsonwang commented on a diff in the pull request:

https://github.com/apache/spark/pull/6827#discussion_r32487191
  
--- Diff: 
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
 ---
@@ -218,6 +223,18 @@ class FsHistoryProviderSuite extends SparkFunSuite 
with BeforeAndAfter with Matc
 }
   }
 
+  test("Parse logs that application is not started") {
+val provider = new FsHistoryProvider((createTestConf()))
+
+val logFile1 = newLogFile("app1", None, inProgress = true)
+writeFile(logFile1, true, None,
+  SparkListenerLogStart("1.4")
+)
+updateAndCheck(provider) { list =>
+  list.size should be (0)
--- End diff --

Yes, that's correct.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8301][SQL] Improve UTF8String substring...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/6804#discussion_r32487048
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala
 ---
@@ -437,17 +437,17 @@ case class Cast(child: Expression, dataType: 
DataType) extends UnaryExpression w
 
   case (BinaryType, StringType) =>
 defineCodeGen (ctx, ev, c =>
-  s"new ${ctx.stringType}().set($c)")
+  s"new ${ctx.stringType}().fromBytes($c)")
--- End diff --

This should be `${ctx.stringType}.fromBytes($c)`, also others


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8301][SQL] Improve UTF8String substring...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6804#issuecomment-112263921
  
  [Test build #916 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/916/consoleFull)
 for   PR 6804 at commit 
[`9f17cc8`](https://github.com/apache/spark/commit/9f17cc80370c612b1cc40801a0d15c1b8f7e283b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-8135. Don't load defaults when reconstit...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6679#issuecomment-112263687
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-8135. Don't load defaults when reconstit...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6679#issuecomment-112263664
  
  [Test build #34962 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34962/console)
 for   PR 6679 at commit 
[`3f1b865`](https://github.com/apache/spark/commit/3f1b865544049265a91215f2edf8dfe0c986b919).
 * This patch **passes all tests**.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `class SerializableConfiguration(@transient var value: Configuration) 
extends Serializable `
  * `class SerializableJobConf(@transient var value: JobConf) extends 
Serializable `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[vanzin]

Github user vanzin commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112261337
  
Ok, now I get what the patch is doing. LGTM with latest changes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[vanzin]

Github user vanzin commented on a diff in the pull request:

https://github.com/apache/spark/pull/6827#discussion_r32486202
  
--- Diff: 
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
 ---
@@ -218,6 +223,18 @@ class FsHistoryProviderSuite extends SparkFunSuite 
with BeforeAndAfter with Matc
 }
   }
 
+  test("Parse logs that application is not started") {
+val provider = new FsHistoryProvider((createTestConf()))
+
+val logFile1 = newLogFile("app1", None, inProgress = true)
+writeFile(logFile1, true, None,
+  SparkListenerLogStart("1.4")
+)
+updateAndCheck(provider) { list =>
+  list.size should be (0)
--- End diff --

Oh I see. The filtering of active / inactive is actually done in 
HistoryServer, not in FsHistoryProvider. My bad. So basically the new behavior 
is that if the app doesn't have an app id, it's not even listed, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112260668
  
  [Test build #34967 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34967/consoleFull)
 for   PR 6827 at commit 
[`cdbb089`](https://github.com/apache/spark/commit/cdbb0890f4286650d4a78d9d30b1f3f2457465f1).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-2213] [SQL] sort merge join for spark s...

[adrian-wang]

Github user adrian-wang commented on the pull request:

https://github.com/apache/spark/pull/5208#issuecomment-112260579
  
@justmytwospence yes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112260564
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112260574
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8247][SPARK-8249][SPARK-8252][SPARK-825...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6762#issuecomment-112260360
  
  [Test build #34966 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34966/consoleFull)
 for   PR 6762 at commit 
[`852ca7c`](https://github.com/apache/spark/commit/852ca7c802f60d294c49296b98bd29e982696435).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[carsonwang]

Github user carsonwang commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112260070
  
Thanks for reviewing this, @vanzin . The code is updated.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8247][SPARK-8249][SPARK-8252][SPARK-825...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6762#issuecomment-112259725
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8247][SPARK-8249][SPARK-8252][SPARK-825...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6762#issuecomment-112259713
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8379][SQL]avoid speculative tasks write...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6833#issuecomment-112259097
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8379][SQL]avoid speculative tasks write...

[jeanlyn]

GitHub user jeanlyn opened a pull request:

https://github.com/apache/spark/pull/6833

[SPARK-8379][SQL]avoid speculative tasks write to the same file

The issue link 
[SPARK-8379](https://issues.apache.org/jira/browse/SPARK-8379)
Currently,when we insert data to the dynamic partition with speculative 
tasks we will get the Exception
```

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException):
 
Lease mismatch on 
/tmp/hive-jeanlyn/hive_2015-06-15_15-20-44_734_8801220787219172413-1/-ext-1/ds=2015-06-15/type=2/part-00301.lzo
 
owned by DFSClient_attempt_201506031520_0011_m_000189_0_-1513487243_53 
but is accessed by 
DFSClient_attempt_201506031520_0011_m_42_0_-1275047721_57
```
This pr try to write the data to temporary dir when using dynamic parition  
avoid the speculative tasks writing the same file

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeanlyn/spark speculation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/6833.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6833


commit e19a3bd77b6b9f44479e51659e244e9809b2963d
Author: jeanlyn 
Date:   2015-06-15T16:38:16Z

avoid speculative tasks write same file




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7555][docs] Add doc for elastic net in ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6504#issuecomment-112258739
  
  [Test build #34965 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34965/consoleFull)
 for   PR 6504 at commit 
[`706d3f7`](https://github.com/apache/spark/commit/706d3f778a5b6b9ecdc1a316349186c86c8dd896).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7555][docs] Add doc for elastic net in ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6504#issuecomment-112258642
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7555][docs] Add doc for elastic net in ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6504#issuecomment-112258631
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[carsonwang]

Github user carsonwang commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112258599
  
@vanzin SPARK-8275 is a different issue. What I wanted to fix in this PR is 
to avoid showing an App ID like application_1432793609805_009_1.inprogress. 
This is not a valid App ID. The cause of this is in `replay` method, it uses 
`appListener.appId.getOrElse(logPath.getName()),`  to get he App ID. If the 
`SparkListenerApplicationStart` event is not logged in the event log, the log 
path is used as the App ID. So I fixed this issue by make sure we only return a 
`FsApplicationAttemptInfo` when there is a valid App ID.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8234][SQL] misc function: md5

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6779#issuecomment-112258426
  
  [Test build #34964 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34964/consoleFull)
 for   PR 6779 at commit 
[`c75bb09`](https://github.com/apache/spark/commit/c75bb0957fca5788a8c3c0dc0f59dedb9c392649).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-7916] [MLlib] MLlib Python doc parity c...

[yanboliang]

Github user yanboliang commented on the pull request:

https://github.com/apache/spark/pull/6460#issuecomment-112258204
  
retest


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8234][SQL] misc function: md5

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6779#issuecomment-112258180
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8234][SQL] misc function: md5

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6779#issuecomment-112258188
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[carsonwang]

Github user carsonwang commented on a diff in the pull request:

https://github.com/apache/spark/pull/6827#discussion_r32485127
  
--- Diff: 
core/src/test/scala/org/apache/spark/deploy/history/FsHistoryProviderSuite.scala
 ---
@@ -218,6 +223,18 @@ class FsHistoryProviderSuite extends SparkFunSuite 
with BeforeAndAfter with Matc
 }
   }
 
+  test("Parse logs that application is not started") {
+val provider = new FsHistoryProvider((createTestConf()))
+
+val logFile1 = newLogFile("app1", None, inProgress = true)
+writeFile(logFile1, true, None,
+  SparkListenerLogStart("1.4")
+)
+updateAndCheck(provider) { list =>
+  list.size should be (0)
--- End diff --

@vanzin , it did fail without my changes and the list.size is 1. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8366] When tasks failed and append new ...

[XuTingjun]

Github user XuTingjun commented on a diff in the pull request:

https://github.com/apache/spark/pull/6817#discussion_r32484601
  
--- Diff: 
core/src/main/scala/org/apache/spark/ExecutorAllocationManager.scala ---
@@ -537,10 +537,19 @@ private[spark] class ExecutorAllocationManager(
   }
 }
 
+override def onTaskResubmit(taskResubmit: SparkListenerTaskResubmit): 
Unit = {
+  val stageId = taskResubmit.stageId
+  allocationManager.synchronized {
+val num = stageIdToNumTasks.getOrElse(stageId, 0)
+stageIdToNumTasks.update(stageId, num + 1)
+  }
+}
+
--- End diff --

@squito, I think when an executor goes down, the stages won't be 
resubmitted. Here I means when a task fails, it will retry, and so a new task 
will append. And in order to let the **ExecutorAllocationManager** knows there 
are new tasks submit, I add **SparkListenerTaskResubmit** event.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[markhamstra]

Github user markhamstra commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112254025
  
Yes, @jerryshao, the idiom is a little clearer when the contents of the 
Option are actually be used to produce a result other than Unit, but it's also 
a little odd to use a different idiom just to handle the Unit result type.  A 
lot of this is just style preference differences that often stem from how 
familiar and comfortable a developer is with other functional programming 
languages, libraries and idioms.  And don't get me started on whether `fold` 
over an Option is a better approach than `map getOrElse` -- we've already had 
that discussion, and I lost :(.

Anyway, the upshot of this is that `map` with a Unit result looks weird to 
some of us, unnecessary pattern matching over Option looks ugly to others of 
us, and `Option#fold` is so offensive to some that I can't use it in Spark!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8381][SQL]reuse typeConvert when conver...

[lianhuiwang]

Github user lianhuiwang commented on the pull request:

https://github.com/apache/spark/pull/6831#issuecomment-112249235
  
@JoshRosen yes, I have removed all uses of 
CatalystTypeConverters.convertToCatalyst and also removed this method. 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8221][SQL]Add pmod function

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6783#issuecomment-112249174
  
  [Test build #34963 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34963/consoleFull)
 for   PR 6783 at commit 
[`88c63ab`](https://github.com/apache/spark/commit/88c63abadb39ff0313d2b1ed09712f3a05120e8c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8221][SQL]Add pmod function

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6783#issuecomment-112248944
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8221][SQL]Add pmod function

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6783#issuecomment-112248937
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-8372] History server shows incorrect in...

[jerryshao]

Github user jerryshao commented on the pull request:

https://github.com/apache/spark/pull/6827#issuecomment-112246206
  
Hi @markhamstra , I'm pretty sure about this style `o.map{ ... }.getOrElse{ 
... }`, but here we transfer nothing but only print some different logs, so 
here using `map` is a little weird from my perspective, just my suggestion :).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Scala version switching build enhancements

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6832#issuecomment-112243794
  
Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: Scala version switching build enhancements

[mallman]

GitHub user mallman opened a pull request:

https://github.com/apache/spark/pull/6832

Scala version switching build enhancements

These commits address a few minor issues in the Scala cross-version support 
in the build:

  1. Correct two missing `${scala.binary.version}` pom file substitutions.
  2. Don't update `scala.binary.version` in parent POM. This property is 
set through profiles.
  3. Update the source of the generated scaladocs in 
`docs/_plugins/copy_api_dirs.rb`.
  4. Factor common code out of `dev/change-version-to-*.sh` and add some 
validation. We also test `sed` to see if it's GNU sed and try `gsed` as an 
alternative if not. This prevents the script from running with a non-GNU sed.

This is my first contribution to Spark. After reading the contributor 
guidelines I wasn't sure if this qualified for a Jira ticket, or if I should 
split these commits into separate contributions. I'm happy to amend my pull 
request or file a Jira ticket as requested.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/VideoAmp/spark-1 scala-versions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/6832.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #6832


commit 475088e7423b59dd7a9ef4b0ab426ebcc3e555ba
Author: Michael Allman 
Date:   2015-06-14T19:17:56Z

Replace jackson-module-scala_2.10 with
jackson-module-scala_${scala.binary.version}

commit e5a675e3411bdfa3e6e387f6688f330fb8e5067b
Author: Michael Allman 
Date:   2015-06-14T19:20:36Z

Don't update scala.binary.version in parent POM. This property is set
through profiles

commit a896dfdff26e56f8c6f0a4769b1d576e3494a524
Author: Michael Allman 
Date:   2015-06-14T19:34:57Z

Update source of scaladocs when changing Scala version

commit 2b926205ff863c1132f3e1d923ff4715ee566bbc
Author: Michael Allman 
Date:   2015-06-14T19:36:10Z

Factor change-scala-version.sh out of change-version-to-*.sh, adding
command line argument validation and testing for GNU sed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-8135. Don't load defaults when reconstit...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/6679#issuecomment-112243026
  
  [Test build #34962 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/34962/consoleFull)
 for   PR 6679 at commit 
[`3f1b865`](https://github.com/apache/spark/commit/3f1b865544049265a91215f2edf8dfe0c986b919).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-8135. Don't load defaults when reconstit...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6679#issuecomment-112242737
  
 Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-8135. Don't load defaults when reconstit...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/6679#issuecomment-112242743
  
Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-8336 Fix NullPointerException with funct...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/6793


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: SPARK-8336 Fix NullPointerException with funct...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/6793#issuecomment-112240939
  
Thanks. Merging in master & branch-1.4.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org