Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r145271690
--- Diff: python/pyspark/serializers.py ---
@@ -259,11 +261,13 @@ def load_stream(self, stream):
"""
Deserialize
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r145245871
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
---
@@ -55,6 +55,12 @@ object ArrowWriter {
case
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r145215343
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,38 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r145037361
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowWriter.scala
---
@@ -55,6 +55,12 @@ object ArrowWriter {
case
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r145036404
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,38 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r145024991
--- Diff: python/pyspark/sql/tests.py ---
@@ -3383,6 +3403,42 @@ def test_vectorized_udf_varargs(self):
res = df.select(f(col('id')))
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144994750
--- Diff: python/pyspark/sql/tests.py ---
@@ -3086,18 +3086,35 @@ class ArrowTests(ReusedPySparkTestCase):
@classmethod
def
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144993515
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,47 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user icexelloss commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144300577
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -31,7 +31,8 @@ object ArrowUtils {
// todo:
Github user icexelloss commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144300232
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -31,7 +31,8 @@ object ArrowUtils {
// todo:
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144271872
--- Diff: python/pyspark/sql/tests.py ---
@@ -3086,18 +3086,35 @@ class ArrowTests(ReusedPySparkTestCase):
@classmethod
def
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144277391
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,47 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144250165
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,47 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144248880
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,47 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144168563
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -31,7 +31,8 @@ object ArrowUtils {
//
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144168006
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,47 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144167906
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,47 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144167503
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,47 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144167095
--- Diff: python/pyspark/sql/types.py ---
@@ -1619,11 +1619,47 @@ def to_arrow_type(dt):
arrow_type = pa.decimal(dt.precision, dt.scale)
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144166783
--- Diff: python/pyspark/sql/tests.py ---
@@ -3383,6 +3400,43 @@ def test_vectorized_udf_varargs(self):
res = df.select(f(col('id')))
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r14414
--- Diff: python/pyspark/sql/tests.py ---
@@ -3383,6 +3400,43 @@ def test_vectorized_udf_varargs(self):
res = df.select(f(col('id')))
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144166470
--- Diff: python/pyspark/serializers.py ---
@@ -223,12 +224,13 @@ def _create_batch(series):
# If a nullable integer series has been promoted
Github user icexelloss commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144148579
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -31,7 +31,8 @@ object ArrowUtils {
// todo:
Github user icexelloss commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144148043
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -31,7 +31,8 @@ object ArrowUtils {
// todo:
Github user icexelloss commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144147353
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -31,7 +31,8 @@ object ArrowUtils {
// todo:
Github user icexelloss commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r144146290
--- Diff: python/pyspark/serializers.py ---
@@ -213,6 +213,7 @@ def __repr__(self):
def _create_batch(series):
+from
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r143885245
--- Diff: python/pyspark/sql/types.py ---
@@ -1624,6 +1624,40 @@ def to_arrow_type(dt):
return arrow_type
+def
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r133261665
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r133058855
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r132870799
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131723483
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131532115
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131506100
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131503699
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131496971
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131487406
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131452100
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131308903
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user icexelloss commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131227296
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r131050585
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r130942914
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r130940849
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r130925333
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r130795748
--- Diff: python/pyspark/sql/tests.py ---
@@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self):
pdf = df.toPandas()
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r13079
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -42,6 +43,9 @@ object ArrowUtils {
case
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r130792754
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
@@ -3092,7 +3092,8 @@ class Dataset[T] private[sql](
val
Github user icexelloss commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r130201966
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -42,6 +43,9 @@ object ArrowUtils {
case
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r130018225
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
---
@@ -792,6 +793,104 @@ class ArrowConvertersSuite
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r128107798
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
---
@@ -792,6 +793,76 @@ class ArrowConvertersSuite
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r127879502
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
---
@@ -792,6 +793,76 @@ class ArrowConvertersSuite
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r127864419
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
---
@@ -792,6 +793,76 @@ class ArrowConvertersSuite
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r127861741
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
---
@@ -792,6 +793,76 @@ class ArrowConvertersSuite
GitHub user BryanCutler opened a pull request:
https://github.com/apache/spark/pull/18664
[SPARK-21375][PYSPARK][SQL][WIP] Add Date and Timestamp support to
ArrowConverters for toPandas() Conversion
## What changes were proposed in this pull request?
WIP started with
53 matches
Mail list logo
