Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/14830
Hi, are you still working on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/14963
Hi, are you still working on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/17096
@HyukjinKwon @holdenk Hi, are you still working on this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ueshin closed the pull request at:
https://github.com/apache/spark/pull/16056
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/16056
I'd close this for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and w
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r124465362
--- Diff: python/pyspark/sql/types.py ---
@@ -958,12 +968,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type(value
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r124464895
--- Diff: python/pyspark/sql/tests.py ---
@@ -2259,6 +2261,60 @@ def test_BinaryType_serialization(self):
df = self.spark.createDataFrame(data
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
Btw, seems like `array` module in Python 2 doesn't have `array.typecodes`
attribute.
```
Traceback (most recent call last):
File
"/home/jenkins/workspace/SparkPullReques
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
In my local environment, this patch for floatarray looks working well:
```python
df = spark.createDataFrame([Row(floatarray=array('f',[1,2,3]),
doublearray=array(
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/13873
Hmm, I guess we need #16056 to fix nullability of `StaticInvoke`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/16056
Yes, I'll reopen, and update if needed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fe
GitHub user ueshin reopened a pull request:
https://github.com/apache/spark/pull/16056
[SPARK-18623][SQL] Add `returnNullable` to `StaticInvoke` and modify it to
handle properly.
## What changes were proposed in this pull request?
Add `returnNullable` to `StaticInvoke` the
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17865#discussion_r125201828
--- Diff: python/pyspark/sql/functions.py ---
@@ -1073,12 +1108,17 @@ def last_day(date):
return Column(sc._jvm.functions.last_day(_to_java_column
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/17227
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18484#discussion_r125212648
--- Diff: python/pyspark/sql/tests.py ---
@@ -2021,6 +2021,11 @@ def test_toDF_with_schema_string(self):
self.assertEqual(df.schema.simpleString
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/17865
Looks like there is a merge conflict. Can you fix it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/17865
Yes, or you can merge master and fix the conflict.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18484
Thanks! merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
I guess you can define source code encoding to the top of the file like:
```
# -*- coding: utf-8 -*-
```
---
If your project is set up for it, you can reply to this email and have
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/18524
[SPARK-21300][SQL] ExternalMapToCatalyst should null-check map key prior to
converting to internal value.
## What changes were proposed in this pull request?
`ExternalMapToCatalyst` should
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18524
cc @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/16056
#18524 could fix the test failure.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18529#discussion_r125542708
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -588,12 +591,14 @@ case class MapObjects
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/16056
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18529#discussion_r12516
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala
---
@@ -588,12 +591,14 @@ case class MapObjects
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18529
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/17865
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/13873
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/17865
How about https://github.com/apache/spark/pull/17865/files#r123964695 ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
Hmm, seems like `net.razorvine.pickle.objects.ArrayConstructor` handles
`'l'` as `int` instead of `long` in Python 2.
https://github.com/irmen/Pyrolite/blob/master/java/src/mai
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/18553
[SPARK-21327][SQL][PYSPARK] ArrayConstructor should handle an array of
typecode 'l' as long rather than int in Python 2.
## What changes were proposed in this pull request?
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18553
cc @holdenk, @HyukjinKwon, @zasdfgbnm
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
I believe #18553 will fix the problem.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r125855962
--- Diff: python/pyspark/sql/types.py ---
@@ -958,12 +1043,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18553
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18553
Thanks for reviewing! merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
LGTM, pending Jenkins.
@HyukjinKwon, @holdenk, Do you have any other concerns?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/17865
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
@zasdfgbnm Hmm, I agree that this is a bug of `ArrayConstructor`.
I guess we should use `'c' -> 19` for [line
60](https://github.com/apache/spark/blob/master/core/src/main/scala/org
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18576#discussion_r126342208
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/UpdateNullabilityInAttributeReferencesSuite.scala
---
@@ -0,0 +1,53
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18597
How about surrounding with SQLConf `spark.sql.session.timeZone` as
`America/Los_Angeles` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18597
LGTM.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18597
Thanks! merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/17150
I guess we can also remove thread safety tests at the bottom of
`org.apache.spark.sql.catalyst.ScalaReflectionSuite`
([ScalaReflectionSuite.scala#L341-L376](https://github.com/srowen/spark/blob
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
@HyukjinKwon, @holdenk, Could you double-check the way to handle an array
of typecode `'c'` please?
I'm not exactly sure this can cover all cases.
---
If your project is set up
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/18655
[SPARK-21440][SQL][PYSPARK] Refactor ArrowConverters and add DecimalType,
ArrayType and StructType support.
## What changes were proposed in this pull request?
This is a refactoring of
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
cc @cloud-fan @BryanCutler
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18664#discussion_r127879502
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
---
@@ -792,6 +793,76 @@ class ArrowConvertersSuite
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
Thank you for your comments.
I agree that we should split this into smaller PRs. I'll push another
commit to remove `ArrowColumnVector` from this as soon as possible.
---
If your project i
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
@BryanCutler I'd like to share the motivation of refactoring
`ArrowConverters` and `ColumnWriter`.
For `ColumnWriter`, at first I'd like to support complex types like
`Arra
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128142573
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18655#discussion_r128146856
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/vectorized/ArrowWriter.scala
---
@@ -0,0 +1,405 @@
+/*
+ * Licensed to the Apache
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18655#discussion_r128146843
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
---
@@ -55,145 +51,55 @@ private[sql] class ArrowPayload
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18655#discussion_r128146875
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/arrow/ArrowConvertersSuite.scala
---
@@ -391,6 +392,85 @@ class ArrowConvertersSuite
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18655#discussion_r128146851
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowConverters.scala
---
@@ -55,145 +51,55 @@ private[sql] class ArrowPayload
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
I see, I'll move files back to `arrow` package.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
LGTM, too. pending Jenkins.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
Yea, it's related to typecode `'L'` in pypy environment.
We might need to handle it as the same as we did for typecode `'c'` or
override `constructLongArrayFromUInt64()`.
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
Btw, now I'm wondering Python 3 can handle array like `array('L',
[9223372036854775807])`.
---
If your project is set up for it, you can reply to this email and have your
reply appe
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18655
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
@HyukjinKwon Oh, I was missing his test for `'L'`. Thanks for testing!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If yo
GitHub user ueshin opened a pull request:
https://github.com/apache/spark/pull/18680
[SPARK-21472][SQL] Introduce ArrowColumnVector as a reader for Arrow
vectors.
## What changes were proposed in this pull request?
Introducing `ArrowColumnVector` as a reader for Arrow
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18680
cc @BryanCutler @kiszk @cloud-fan
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18680#discussion_r128217817
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java
---
@@ -0,0 +1,510 @@
+/*
+ * Licensed to the
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18444
Thanks! merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18680#discussion_r128422124
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ReadOnlyColumnVector.java
---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18680#discussion_r128425637
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ReadOnlyColumnVector.java
---
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18680#discussion_r128425617
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ArrowColumnVector.java
---
@@ -0,0 +1,545 @@
+/*
+ * Licensed to the
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/18680#discussion_r128425605
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/arrow/ArrowUtils.scala
---
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/18680
@BryanCutler Thank you for reviewing!
As for scope, yes, I'd like these APIs to be public. Do you have any
concerns about it?
---
If your project is set up for it, you can reply to this
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
The remaining discussions in this pr are:
- [ ] Do we need the config
`"spark.sql.execution.pandas.respectSessionTimeZone"`?
- @cloud-fan raised that we don't need the
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19726
LGTM for now, but I'd like you to add some comments to explain this is a
workaround for Sphinx <`1.6.6`.
---
-
To unsubs
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
I'll ask dev list if we can drop support old Pandas (<0.19.2), or what
version we should support if we can't.
---
---
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19630#discussion_r151042155
--- Diff: python/pyspark/sql/functions.py ---
@@ -2247,16 +2142,20 @@ def pandas_udf(f=None, returnType=StringType()):
| 8| JOHN DOE
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19762
Thanks! merging to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19630#discussion_r151331041
--- Diff: python/pyspark/worker.py ---
@@ -89,6 +90,26 @@ def verify_result_length(*a):
return lambda *a: (verify_result_length(*a
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17436#discussion_r151333181
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/VectorizedHashMapGenerator.scala
---
@@ -75,9 +77,14 @@ class
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19630#discussion_r151466298
--- Diff: python/pyspark/worker.py ---
@@ -89,6 +90,26 @@ def verify_result_length(*a):
return lambda *a: (verify_result_length(*a
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19769#discussion_r151597861
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetReadSupport.scala
---
@@ -30,6 +30,7 @@ import
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19769#discussion_r151597632
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
---
@@ -337,6 +341,8 @@ class
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19769#discussion_r151597335
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedParquetRecordReader.java
---
@@ -105,13 +108,23
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19769#discussion_r151596892
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
---
@@ -93,13 +94,18 @@
private
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19769#discussion_r151598171
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala
---
@@ -87,4 +96,113 @@ class
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19782
LGTM.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19769#discussion_r151908974
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetInteroperabilitySuite.scala
---
@@ -87,4 +95,109 @@ class
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19769#discussion_r151909614
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java
---
@@ -298,7 +304,10 @@ private void
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Jenkins, retest this please.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19607#discussion_r152174813
--- Diff: python/pyspark/sql/dataframe.py ---
@@ -1913,7 +1920,16 @@ def toPandas(self):
for f, t in dtype.items
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19607#discussion_r152174935
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala
---
@@ -58,6 +59,11 @@ class ArrowPythonRunner
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19607#discussion_r152174929
--- Diff: python/pyspark/worker.py ---
@@ -150,7 +150,8 @@ def read_udfs(pickleSer, infile, eval_type):
if eval_type
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19607
Well, I'd leave the config
`spark.sql.execution.pandas.respectSessionTimeZone` as it is for now to be safe
as @gatorsmile mentioned before.
And as we discussed in dev list, I'll upda
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/17436#discussion_r152189670
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -140,6 +140,13 @@ object SQLConf {
.booleanConf
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19607#discussion_r152201452
--- Diff: python/pyspark/sql/types.py ---
@@ -1678,37 +1678,105 @@ def from_arrow_schema(arrow_schema):
for field in arrow_schema
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19468#discussion_r152217994
--- Diff:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/scheduler/cluster/k8s/KubernetesClusterSchedulerBackendSuite.scala
---
@@ -0,0
701 - 800 of 1149 matches
Mail list logo
