Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
fixed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
We just change the dirty hacking to something like:
```
if python2.7 and not pypy:
do the dirty hacking
fool the unit test
```
_Sent from my OnePlus ONEPLUS
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
@ueshin Maybe we don't have to do the same thing. If this is the problem,
then I think in pypy environment, 'L' was originally unsupported and there is
no need to support it now.
_Sent
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
Hmm, how was 'L' supported in master python 2? Why there was no such an
error there?
_Sent from my OnePlus ONEPLUS A3000 using
[FastHub](https://play.google.com/store/apps/details?id
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
Looks to be an unrelated error.
_Sent from my OnePlus ONEPLUS A3000 using
[FastHub](https://play.google.com/store/apps/details?id=com.fastaccess.github)_
---
If your project is set up
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
@HyukjinKwon Take a look at my newest commit. I think I find a better way
to solve the problem that keeps all the hacking code for `SPARK-21465` in a
single place, making it easier to be removed
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128142511
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128141018
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128141384
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128138629
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128132778
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128132584
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128132120
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128131993
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128131853
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128130893
--- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
---
@@ -55,13 +55,12 @@ private[spark] object SerDeUtil extends Logging
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
By the way, I'm traveling tomorrow and will be back next Tuesday. During
traveling, I may not be able to answer any comments, questions, etc.
---
If your project is set up for it, you can reply
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128129716
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128129280
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128128561
--- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
---
@@ -55,13 +55,12 @@ private[spark] object SerDeUtil extends Logging
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r128127304
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1016,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
@HyukjinKwon all done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r127814477
--- Diff: python/pyspark/sql/types.py ---
@@ -938,12 +1023,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r127555013
--- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
---
@@ -57,11 +57,11 @@ private[spark] object SerDeUtil extends Logging
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
I updated my code according to @HyukjinKwon's suggestion
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r127521525
--- Diff: python/pyspark/sql/types.py ---
@@ -915,6 +916,90 @@ def _parse_datatype_json_value(json_value):
long: LongType
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r127513810
--- Diff: python/pyspark/sql/types.py ---
@@ -935,6 +936,90 @@ def _parse_datatype_json_value(json_value):
long: LongType
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
Finally, all tests pass...
I'm happy that my test case helps find two more bugs and thank @ueshin for
fixing these two bugs :)
---
If your project is set up for it, you can reply
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r126809517
--- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
---
@@ -72,7 +72,11 @@ private[spark] object SerDeUtil extends Logging
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r126806942
--- Diff: core/src/main/scala/org/apache/spark/api/python/SerDeUtil.scala
---
@@ -72,7 +72,11 @@ private[spark] object SerDeUtil extends Logging
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
For some reason, I can not reproduce the error on my machine. I run the
test using the following command:
```bash
PYSPARK_PYTHON=$(which python2) ./bin/spark-submit
python/pyspark/sql
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
```
net.razorvine.pickle.PickleException: for c/u type must be 18/19/20/21
at
net.razorvine.pickle.objects.ArrayConstructor.construct(ArrayConstructor.java:154
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r125910692
--- Diff: python/pyspark/sql/types.py ---
@@ -958,12 +1043,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
Maybe the bug is in py4j
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
There seems to be another bug in spark on somewhere:
```python
rdd = sc.parallelize([array('l',[9223372036854775807])])
print(rdd._to_java_object_rdd().collect()[0][0
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
it is strange...
The test pass at python3, but at python2 with typecode `l`,
`9223372036854775807` becomes `-1` after creating `DataFrame` and
---
If your project is set up for it, you can
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r125642042
--- Diff: python/pyspark/sql/types.py ---
@@ -935,6 +936,86 @@ def _parse_datatype_json_value(json_value):
long: LongType
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
test fails due to a `§` in comment...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
I'm not sure if Jenkins would listen to me :(
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r125173526
--- Diff: python/pyspark/sql/tests.py ---
@@ -2250,6 +2256,67 @@ def test_BinaryType_serialization(self):
df = self.spark.createDataFrame
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r125173517
--- Diff: python/pyspark/sql/tests.py ---
@@ -2250,6 +2256,67 @@ def test_BinaryType_serialization(self):
df = self.spark.createDataFrame
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r125173508
--- Diff: python/pyspark/sql/types.py ---
@@ -935,6 +936,86 @@ def _parse_datatype_json_value(json_value):
long: LongType
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r125173496
--- Diff: python/pyspark/sql/types.py ---
@@ -935,6 +936,86 @@ def _parse_datatype_json_value(json_value):
long: LongType
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r125173449
--- Diff: python/pyspark/sql/types.py ---
@@ -935,6 +936,86 @@ def _parse_datatype_json_value(json_value):
long: LongType
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
jenkins retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
@ueshin @HyukjinKwon I think I'm done now. There are still fails in tests,
but it doesn't looks to be something related to my change
---
If your project is set up for it, you can reply
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r124572922
--- Diff: python/pyspark/sql/types.py ---
@@ -958,12 +968,17 @@ def _infer_type(obj):
return MapType(_infer_type(key), _infer_type
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/18444
@HyukjinKwon Thanks for figuring that out. I will fix those issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r124536696
--- Diff: python/pyspark/sql/tests.py ---
@@ -2259,6 +2261,69 @@ def test_BinaryType_serialization(self):
df = self.spark.createDataFrame
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/18444#discussion_r124531036
--- Diff: python/pyspark/sql/tests.py ---
@@ -2259,6 +2261,60 @@ def test_BinaryType_serialization(self):
df = self.spark.createDataFrame
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14198
reopened at https://github.com/apache/spark/pull/18444
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
GitHub user zasdfgbnm opened a pull request:
https://github.com/apache/spark/pull/18444
[SPARK-16542][SQL][PYSPARK] Fix bugs about types that result an array of
null when creating DataFrame using python
## What changes were proposed in this pull request?
This is the reopen
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14198
@ueshin @gatorsmile I'm happy to resolve the conflicts IF AND ONLY IF there
will be a developer work on the code review for this. This PR was opened more
than a year ago and I keep waiting
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14198
Hi @holdenk , I think I'm done. I create a test for this issue and I do
find from the test that spark has the same issue not only for float but also
for byte and short. After several commits
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14198
Something to mention is, there is still one problem that I'm not sure
whether I solve it correctly: in python's array, unsigned types are supported,
but unsigned types are not supported in JVM
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14198
I'd love to help
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user zasdfgbnm closed the pull request at:
https://github.com/apache/spark/pull/14231
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14231
Yes this patch looks clearer
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72803075
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72562015
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72460695
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72458729
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72457989
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72410689
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72364403
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72364159
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72276770
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72167441
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r72167136
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r71063901
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
Github user zasdfgbnm commented on a diff in the pull request:
https://github.com/apache/spark/pull/14231#discussion_r71063792
--- Diff: bin/spark-class ---
@@ -65,24 +65,25 @@ fi
# characters that would be otherwise interpreted by the shell. Read that
in a while loop
GitHub user zasdfgbnm opened a pull request:
https://github.com/apache/spark/pull/14231
[SPARK-16586] Change the way the exit code of launcher is handled to avoid
problem when launcher fails
## What changes were proposed in this pull request?
In the spark-class shell
GitHub user zasdfgbnm opened a pull request:
https://github.com/apache/spark/pull/14198
Fix bugs about types that result an array of null when creating dataframe
using python
## What changes were proposed in this pull request?
Fix bugs about types that result an array
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14027
OK, but will the status of the original JIRA keep being "won't fix", if
someone decide to reopen the discussion on it?
Can anyone give a comment on whether the running on foregrou
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/3881
Can anyone explain what happened to this PR? Why people close this PR
without adding any support for running in foreground?
---
If your project is set up for it, you can reply to this email
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14027
I followed that PR (https://github.com/apache/spark/pull/3881)
"I was under the impression all the shell scripts were getting refactored
and that this patch had become obsolete. I agree
Github user zasdfgbnm commented on the issue:
https://github.com/apache/spark/pull/14027
I see similar pull request here:
https://github.com/apache/spark/pull/3881
But I didn't get what happened to that and why it was closed without adding
any support for running
GitHub user zasdfgbnm opened a pull request:
https://github.com/apache/spark/pull/14027
Spark daemon run.sh
## What changes were proposed in this pull request?
I add some shell scripts to support running master and slave foreground,
which makes it more convenient to write
80 matches
Mail list logo
