[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-09 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-91331541 @vlyubin Would you mind to add `[SQL]` to the PR title? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-91077336 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-91077326 [Test build #29897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29897/consoleFull) for PR 5279 at commit [`c327bc9`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-91076205 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-91076192 [Test build #29896 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29896/consoleFull) for PR 5279 at commit [`dec6802`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-91055718 [Test build #29897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29897/consoleFull) for PR 5279 at commit [`c327bc9`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r28020466 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeConversions.scala --- @@ -20,24 +20,21 @@ package org.apache.spark.sql.types im

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r28020458 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala --- @@ -0,0 +1,286 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-91054471 [Test build #29896 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29896/consoleFull) for PR 5279 at commit [`dec6802`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r28016805 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeConversions.scala --- @@ -20,24 +20,21 @@ package org.apache.spark.sql.types i

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r28000283 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScan.scala --- @@ -32,9 +32,20 @@ case class LocalTableScan(output: Seq[Attribute],

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r28000217 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeConversions.scala --- @@ -20,24 +20,21 @@ package org.apache.spark.sql.types im

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27989521 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/LocalTableScan.scala --- @@ -32,9 +32,20 @@ case class LocalTableScan(output: Seq[Attribute]

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27989287 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala --- @@ -0,0 +1,286 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27988958 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeConversions.scala --- @@ -20,24 +20,21 @@ package org.apache.spark.sql.types i

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r2790 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala --- @@ -0,0 +1,286 @@ +/* + * Licensed to the Apach

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27988790 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala --- @@ -459,7 +462,7 @@ class SQLContext(@transient val sparkContext: SparkContext)

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27986331 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala --- @@ -0,0 +1,286 @@ +/* + * Licensed to the Apac

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27985787 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala --- @@ -0,0 +1,286 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-08 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27971231 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/CatalystTypeConverters.scala --- @@ -0,0 +1,286 @@ +/* + * Licensed to the Apac

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90819071 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90819065 [Test build #29832 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29832/consoleFull) for PR 5279 at commit [`74301fa`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90806378 [Test build #29832 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29832/consoleFull) for PR 5279 at commit [`74301fa`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27943719 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27942699 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27942693 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27941970 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeConversions.scala --- @@ -20,24 +20,21 @@ package org.apache.spark.sql.types im

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27941762 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache S

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90792746 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90792740 [Test build #29823 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29823/consoleFull) for PR 5279 at commit [`afa3aa5`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939515 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeConversions.scala --- @@ -20,24 +20,21 @@ package org.apache.spark.sql.types i

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939459 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939451 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939422 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939398 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939366 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939310 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939235 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939093 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27939075 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27938939 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27938814 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27938786 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27938778 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27938767 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ReflectionConverters.scala --- @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90778442 [Test build #29823 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29823/consoleFull) for PR 5279 at commit [`afa3aa5`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90778182 Jenkins retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have t

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90778065 [Test build #29822 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29822/consoleFull) for PR 5279 at commit [`939c225`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90778059 [Test build #29822 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29822/consoleFull) for PR 5279 at commit [`939c225`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90778067 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90772886 [Test build #29821 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29821/consoleFull) for PR 5279 at commit [`881dc60`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90772896 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90772894 [Test build #29821 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29821/consoleFull) for PR 5279 at commit [`881dc60`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27935952 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/types/DataTypeConversions.scala --- @@ -67,11 +64,4 @@ private[sql] object DataTypeConversions {

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90719699 That brings up a good point - should ArrayTypes be converted into properly primitive Java Arrays if they began as Seqs? --- If your project is set up for it, you can re

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90711579 @marmbrus In ```case arrayType: ArrayType => val elementConverter = createScalaConverter(arrayType.elementType) (item: Any) => item match { case s: Seq[_]

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90708411 What is the java beans error? Maybe we can fix that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27910399 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -94,11 +178,95 @@ trait ScalaReflection { case (oth

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90704529 @marmbrus Not having `case other => other` in `createScalaConverter` makes java beans fail. As of now I also need it to handle nulls. As for `createCatalystConverter`,

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27910341 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -94,11 +178,95 @@ trait ScalaReflection { case (oth

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-07 Thread vlyubin
Github user vlyubin commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27910331 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -80,6 +80,90 @@ trait ScalaReflection { case (other

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90313848 This looks great. Thanks for working on it! Two high level pieces of feedback: - Can you move all this conversion logic to a separate object? Its my fault this i

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27839068 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -80,6 +80,90 @@ trait ScalaReflection { case (othe

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90269142 @vlyubin This optimization seems good to me, we also did something similar in Python API. Another king of optimization we could add is that we only call converter

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27837485 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -80,6 +80,90 @@ trait ScalaReflection { case (other,

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread marmbrus
Github user marmbrus commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27836865 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -94,11 +178,95 @@ trait ScalaReflection { case (ot

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27836835 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -94,11 +178,95 @@ trait ScalaReflection { case (othe

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread davies
Github user davies commented on a diff in the pull request: https://github.com/apache/spark/pull/5279#discussion_r27836566 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala --- @@ -94,11 +178,95 @@ trait ScalaReflection { case (othe

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90253853 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90253835 [Test build #29752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29752/consoleFull) for PR 5279 at commit [`8cad6e2`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90246124 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90246098 [Test build #29751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29751/consoleFull) for PR 5279 at commit [`8cad6e2`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90228322 [Test build #29752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29752/consoleFull) for PR 5279 at commit [`8cad6e2`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread vlyubin
Github user vlyubin commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90227470 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90223702 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90218702 [Test build #29751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29751/consoleFull) for PR 5279 at commit [`8cad6e2`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90204440 [Test build #29749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29749/consoleFull) for PR 5279 at commit [`3425446`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90204449 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-06 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-90204297 [Test build #29749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29749/consoleFull) for PR 5279 at commit [`3425446`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-89391261 [Test build #29688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29688/consoleFull) for PR 5279 at commit [`99d333a`](https://gith

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-03 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-89391265 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/29

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-03 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-89391131 [Test build #29688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/29688/consoleFull) for PR 5279 at commit [`99d333a`](https://githu

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-03 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-89390749 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark pull request: [SPARK-6620] Speed up toDF() and rdd() functio...

2015-04-03 Thread vlyubin
Github user vlyubin commented on the pull request: https://github.com/apache/spark/pull/5279#issuecomment-89389811 @rxin Sorry, I just removed the WIP tag. There isn't anything more to add, as it turned out that there are no places here where we could use SpecificMutableRow to speed t