GitHub user gengliangwang opened a pull request:
https://github.com/apache/spark/pull/21838
[SPARK-24811][SQL]Avro: add new function from_avro and to_avro
## What changes were proposed in this pull request?
1. Add a new function from_avro for parsing a binary column of avro format
and converting it into its corresponding catalyst value.
2. Add a new function to_avro for converting a column into binary of avro
format with the specified schema.
I created #21774 for this, but it failed the build
https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Compile/job/spark-master-compile-maven-hadoop-2.6/7902/
Additional changes In this PR:
1. Add `scalacheck` dependency in pom.xml to resolve the failure.
2. Update the `log4j.properties` to make it consistent with other modules.
## How was this patch tested?
Unit test
Compile with different commands:
```
./build/mvn --force -DzincPort=3643 -DskipTests -Phadoop-2.6
-Phive-thriftserver -Pkinesis-asl -Pspark-ganglia-lgpl -Pmesos -Pyarn compile
test-compile
./build/mvn --force -DzincPort=3643 -DskipTests -Phadoop-2.7
-Phive-thriftserver -Pkinesis-asl -Pspark-ganglia-lgpl -Pmesos -Pyarn compile
test-compile
./build/mvn --force -DzincPort=3643 -DskipTests -Phadoop-3.1
-Phive-thriftserver -Pkinesis-asl -Pspark-ganglia-lgpl -Pmesos -Pyarn compile
test-compile
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gengliangwang/spark from_and_to_avro
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21838.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21838
----
commit b02d7edbf39f9ad4c4a2fdae6eeec1104097057e
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-15T18:47:29Z
add from_avro and to_avro
commit 8aa2b31fcfa7d3a36e47f5c0939c575971e95df3
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-17T14:35:27Z
add test suite
commit 54861323bec556c66e94ca9919760ac9b3a3c11e
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-18T09:43:27Z
add AvroFunctionsSuite
commit 76763da105ae6b6d05802c7ce846981ee9b29d3b
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-19T07:28:01Z
address some comments
commit 81614c23de0b1b7717b5b61dc81ab15e51671f3e
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-19T16:37:44Z
codegen
commit e5c170281429e0a3b2981eef9c08a2a757c89d0d
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-19T19:01:54Z
address comments
commit 8421f445160eac368a90f21ef091bdf1101eb390
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-20T07:45:31Z
address comments
commit 47caecee4efdfae4b74c05974f36254d2974c278
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-20T08:24:45Z
override sql
commit f54adc9251936abecf1cd8403c7df6b0d9b5a71e
Author: Gengliang Wang <gengliang.wang@...>
Date: 2018-07-22T12:31:13Z
add scalacheck and fix build failure
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
