[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/13269
  
https://github.com/apache/spark/pull/13402 is merged, and I have one more 
PR to send.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[yhuai]

Github user yhuai commented on the pull request:

https://github.com/apache/spark/pull/13269
  
Are we going to break this PR to multiple smaller PRs?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-52647
  
As we discussed offline, this PR also enables case insensitive encoder 
resolution. Would be nice to add a test case for it. Basically something like 
this:

```scala
case class A(a: String)

val data = Seq(
  "{ 'A': 'foo' }",
  "{ 'A': 'bar' }"
)

val df1 = spark.read.json(sc.parallelize(data))
df1.printSchema()
// root
//  |-- A: string (nullable = true)

val ds1 = df1.as[A]
ds1.printSchema()
// root
//  |-- a: string (nullable = true)
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[yhuai]

Github user yhuai commented on a diff in the pull request:

https://github.com/apache/spark/pull/13269#discussion_r64940498
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/encoders/ExpressionEncoder.scala
 ---
@@ -191,6 +189,26 @@ case class ExpressionEncoder[T](
 
   if (flat) require(serializer.size == 1)
 
+  /**
+   * Returns a new copy of this encoder, where the `deserializer` is 
resolved and bound to the
+   * given schema.
+   *
+   * Note that, ideally encoder is used as a container of serde 
expressions, the resolution and
+   * binding stuff should happen inside query framework.  However, in some 
cases we need to
+   * use encoder as a function to do serialization directly(e.g. 
Dataset.collect), then we can use
+   * this method to do resolution and binding outside of query framework.
+   */
+  def resolveAndBind(
+  attrs: Seq[Attribute] = schema.toAttributes,
+  analyzer: Analyzer = SimpleAnalyzer): ExpressionEncoder[T] = {
--- End diff --

Where do we pass in an existing analyzer?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-222041209
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-222041212
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59435/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-222041073
  
**[Test build #59435 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59435/consoleFull)**
 for PR 13269 at commit 
[`13fed35`](https://github.com/apache/spark/commit/13fed3591c9486dadc725c6d3fb95407f3c4f868).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds the following public classes _(experimental)_:
  * `case class GetColumnByOrdinal(ordinal: Int, dataType: DataType) 
extends LeafExpression`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/13269#discussion_r64842044
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/ScalaReflection.scala 
---
@@ -449,152 +449,151 @@ object ScalaReflection extends ScalaReflection {
   }
 }
 
-if (!inputObject.dataType.isInstanceOf[ObjectType]) {
--- End diff --

put this `if` into pattern match, to reduce one ident level


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-222030468
  
**[Test build #59435 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59435/consoleFull)**
 for PR 13269 at commit 
[`13fed35`](https://github.com/apache/spark/commit/13fed3591c9486dadc725c6d3fb95407f3c4f868).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[clockfly]

Github user clockfly commented on a diff in the pull request:

https://github.com/apache/spark/pull/13269#discussion_r64617059
  
--- Diff: 
sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala ---
@@ -42,17 +42,9 @@ class KeyValueGroupedDataset[K, V] private[sql](
 private val dataAttributes: Seq[Attribute],
 private val groupingAttributes: Seq[Attribute]) extends Serializable {
 
-  // Similar to [[Dataset]], we use unresolved encoders for later 
composition and resolved encoders
-  // when constructing new logical plans that will operate on the output 
of the current
-  // queryexecution.
-
-  private implicit val unresolvedKEncoder = encoderFor(kEncoder)
-  private implicit val unresolvedVEncoder = encoderFor(vEncoder)
-
-  private val resolvedKEncoder =
-unresolvedKEncoder.resolve(groupingAttributes, OuterScopes.outerScopes)
-  private val resolvedVEncoder =
-unresolvedVEncoder.resolve(dataAttributes, OuterScopes.outerScopes)
+  // Similar to [[Dataset]], we turn the passed in encoder to 
`ExpressionEncoder` explicitly.
+  private implicit val kEnc = encoderFor(kEncoder)
--- End diff --

Is it better to use the full name like keyEncoder?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221158097
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221158098
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59167/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221157983
  
**[Test build #59167 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59167/consoleFull)**
 for PR 13269 at commit 
[`3f51e4d`](https://github.com/apache/spark/commit/3f51e4db7a90dfd6ddea855b04a67bb7ca414beb).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221132593
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221132582
  
**[Test build #59165 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59165/consoleFull)**
 for PR 13269 at commit 
[`73e9c1a`](https://github.com/apache/spark/commit/73e9c1abec9ac22bb6e1370b0dcd44714b0acf71).
 * This patch **fails MiMa tests**.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221132596
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59165/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221132316
  
**[Test build #59167 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59167/consoleFull)**
 for PR 13269 at commit 
[`3f51e4d`](https://github.com/apache/spark/commit/3f51e4db7a90dfd6ddea855b04a67bb7ca414beb).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[cloud-fan]

Github user cloud-fan commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221131554
  
cc @marmbrus @liancheng @yhuai @clockfly 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/13269#issuecomment-221131551
  
**[Test build #59165 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59165/consoleFull)**
 for PR 13269 at commit 
[`73e9c1a`](https://github.com/apache/spark/commit/73e9c1abec9ac22bb6e1370b0dcd44714b0acf71).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15494][SQL] encoder code cleanup

[cloud-fan]

GitHub user cloud-fan opened a pull request:

https://github.com/apache/spark/pull/13269

[SPARK-15494][SQL] encoder code cleanup

## What changes were proposed in this pull request?

Our encoder framework has been evolved a lot, this PR tries to clean up the 
code to make it more readable and emphasise the concept that encoder should be 
used as a container of serde expressions.

1. move validation logic to analyzer instead of encoder
2. only have a `resolveAndBind` method in encoder instead of `resolve` and 
`bind`, as we don't have the encoder life cycle concept anymore.
3. `Dataset` don't need to keep a resolved encoder, as is no such concept 
anymore. bound encoder is still needed to do serialization outside of query 
framework. 


## How was this patch tested?

existing test


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloud-fan/spark clean-encoder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/13269.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #13269


commit 73e9c1abec9ac22bb6e1370b0dcd44714b0acf71
Author: Wenchen Fan 
Date:   2016-05-23T23:38:42Z

encoder code cleanup




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org