Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19540
**[Test build #82924 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82924/testReport)**
for PR 19540 at commit
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19540
I think branch 2.2 also has similar issue when fetching resources from
remote secure HDFS.
---
-
To unsubscribe, e-mail:
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/10949
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19540
ok to test.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19540
Thanks for the fix! I didn't test on secure cluster when did glob path
support, so I didn't realize such issue.
---
-
To
Github user ueshin closed the pull request at:
https://github.com/apache/spark/pull/19505
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user ueshin commented on the issue:
https://github.com/apache/spark/pull/19505
Sure, I'd close this.
@icexelloss Of course you can open a separate JIRA and another PR. Thanks!
---
-
To unsubscribe,
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19485
This is the API link you refer
`https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.DataFrameReader@csv(paths:String*):org.apache.spark.sql.DataFrame`
I just
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19523
ok to test
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19523
**[Test build #82923 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82923/testReport)**
for PR 19523 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19523
@mgaido91 Could you update the PR title?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user kiszk commented on the issue:
https://github.com/apache/spark/pull/19508
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19517
**[Test build #82922 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82922/testReport)**
for PR 19517 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19540
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19517#discussion_r145878293
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/UserDefinedPythonFunction.scala
---
@@ -22,17 +22,26 @@ import
GitHub user sjrand opened a pull request:
https://github.com/apache/spark/pull/19540
[SPARK-22319][Core] call loginUserFromKeytab before accessing hdfs
## What changes were proposed in this pull request?
In `SparkSubmit`, call `loginUserFromKeytab` before attempting to make
Github user ueshin commented on a diff in the pull request:
https://github.com/apache/spark/pull/19517#discussion_r145878275
--- Diff: python/pyspark/sql/functions.py ---
@@ -2038,13 +2038,22 @@ def _wrap_function(sc, func, returnType):
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19272
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82919/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19272
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19272
**[Test build #82919 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82919/testReport)**
for PR 19272 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19485
I meant adding a new chapter describing options, removing duplication, for
example here
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19505
@ueshin Maybe close this PR?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands,
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19517
LGTM.
@ueshin Could you remove `[WIP]` from the title of this PR?
---
-
To unsubscribe, e-mail:
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19517#discussion_r145877519
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/UserDefinedPythonFunction.scala
---
@@ -22,17 +22,26 @@ import
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19517#discussion_r145877442
--- Diff: python/pyspark/sql/functions.py ---
@@ -2038,13 +2038,22 @@ def _wrap_function(sc, func, returnType):
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19517
**[Test build #82921 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82921/testReport)**
for PR 19517 at commit
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19517
retest this please
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19485
@HyukjinKwon I did not understand what is your suggestion.
@jomach Any reason you closed this PR or you plan to open a new one?
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19539
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82920/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19539
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19539
**[Test build #82920 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82920/testReport)**
for PR 19539 at commit
Github user icexelloss commented on the issue:
https://github.com/apache/spark/pull/19505
@viirya @cloud-fan I updated my original summary. I think it answers
`group_transform` question. I also added more example to each type.
@HyukjinKwon @viirya I agree we can move this to
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19524
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82918/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19524
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19524
**[Test build #82918 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82918/testReport)**
for PR 19524 at commit
Github user ambauma commented on the issue:
https://github.com/apache/spark/pull/19528
Believed fixed. Hard to say for sure without knowing the precise python
and numpy versions the build is using.
---
-
To
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19469
Ah, I didn't realize there is a change in that PR.
I agree we need a better solution
---
-
To unsubscribe,
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19471
> No behavior change if there is no overlapped columns in data and
partition schema.
> The schema changed(partition columns go to the end) when reading file
format data source with
Github user ambauma commented on the issue:
https://github.com/apache/spark/pull/19528
Able to duplicate. Working theory is that this is related to numpy 1.12.1.
Here is my conda env:
(spark-1.6) andrew@andrew-Inspiron-7559:~/git/spark$ conda list
# packages in environment
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19269
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19269
Thanks! Merged to master.
This is just the first commit of the data source v2 write protocol. More
PRs are coming to further improve it.
---
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/19269
It sounds like the initialization stages are missing in the current
protocol API design. We can do it later.
LGTM
---
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19505
The group_transform udfs looks a bit weird to me. @icexelloss Can you
explain the use case of it?
---
-
To unsubscribe, e-mail:
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/19269#discussion_r145870061
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/WriteToDataSourceV2.scala
---
@@ -0,0 +1,133 @@
+/*
+ *
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19534
@sitalkedia I have a very old similar PR #11205 , maybe you can refer to it.
---
-
To unsubscribe, e-mail:
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19459#discussion_r145865969
--- Diff: python/pyspark/sql/session.py ---
@@ -510,6 +578,12 @@ def createDataFrame(self, data, schema=None,
samplingRatio=None, verifySchema=Tr
Github user ambauma commented on the issue:
https://github.com/apache/spark/pull/19528
Working on duplicating PySpark failures...
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19459#discussion_r145863796
--- Diff: python/pyspark/sql/session.py ---
@@ -414,6 +415,73 @@ def _createFromLocal(self, data, schema):
data = [schema.toInternal(row) for
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19527
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82917/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19527
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19527
**[Test build #82917 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82917/testReport)**
for PR 19527 at commit
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19459#discussion_r145862488
--- Diff: python/pyspark/sql/session.py ---
@@ -414,6 +415,73 @@ def _createFromLocal(self, data, schema):
data = [schema.toInternal(row) for
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19539
**[Test build #82920 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82920/testReport)**
for PR 19539 at commit
GitHub user wzhfy opened a pull request:
https://github.com/apache/spark/pull/19539
[WIP] [SQL] Remove unnecessary methods
## What changes were proposed in this pull request?
Remove unnecessary methods.
## How was this patch tested?
Existing tests.
You
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19272
**[Test build #82919 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82919/testReport)**
for PR 19272 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19505
+1 for separate JIRA to clarify the proposal and +0 for 3. out of those
three, too.
---
-
To unsubscribe, e-mail:
Github user ArtRand commented on a diff in the pull request:
https://github.com/apache/spark/pull/19272#discussion_r145861062
--- Diff:
resource-managers/mesos/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosCoarseGrainedSchedulerBackend.scala
---
@@ -194,6 +198,27
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19469
@felixcheung As you can see there's bunch of configurations needs to be
added here in https://github.com/apache-spark-on-k8s/spark/pull/516, that's why
I'm asking a general solutions for such
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19459#discussion_r145859471
--- Diff: python/pyspark/sql/session.py ---
@@ -510,6 +578,12 @@ def createDataFrame(self, data, schema=None,
samplingRatio=None, verifySchema=Tr
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19505
@icexelloss The summary and the proposal 3 looks great. To prevent
confusing, can you also put the usage of each function type in proposal 3?
E.g., group_map is for `groupby().apply()`, transform is
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19505
Btw, I think the scope of this change is more than just a follow-up. Should
we create another JIRA for it?
---
-
To unsubscribe,
Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19459#discussion_r145858362
--- Diff: python/pyspark/sql/session.py ---
@@ -414,6 +415,73 @@ def _createFromLocal(self, data, schema):
data = [schema.toInternal(row)
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19524
**[Test build #82918 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82918/testReport)**
for PR 19524 at commit
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19524
Thanks for your review @shaneknapp.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/19486
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19486
Merged to master.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19527
**[Test build #82917 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82917/testReport)**
for PR 19527 at commit
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r145856074
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19486
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19486
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82916/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19486
**[Test build #82916 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82916/testReport)**
for PR 19486 at commit
Github user MrBago commented on the issue:
https://github.com/apache/spark/pull/19439
@imatiach-msft just a few more comments. When I was looking over this I
realized that the python and Scala name spaces are going to be a little
different, eg `pyspark.ml.image.readImages` vs
Github user MrBago commented on a diff in the pull request:
https://github.com/apache/spark/pull/19439#discussion_r145843289
--- Diff: python/pyspark/ml/image.py ---
@@ -0,0 +1,122 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user MrBago commented on a diff in the pull request:
https://github.com/apache/spark/pull/19439#discussion_r145842379
--- Diff: python/pyspark/ml/image.py ---
@@ -0,0 +1,122 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user MrBago commented on a diff in the pull request:
https://github.com/apache/spark/pull/19439#discussion_r145845879
--- Diff: python/pyspark/ml/image.py ---
@@ -0,0 +1,122 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+#
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r145848280
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache
Github user viirya commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r145847823
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache
Github user viirya commented on the issue:
https://github.com/apache/spark/pull/19527
Benchmark against existing one hot encoder.
Because existing encoder only needs to run `transform`, there is no fitting
time.
Transforming:
numColums | Existing one
Github user yssharma commented on the issue:
https://github.com/apache/spark/pull/18029
@brkyvz Please have a look once you have time. Thanks.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For
Github user ron8hu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19479#discussion_r145840719
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala
---
@@ -216,65 +218,61 @@ object ColumnStat extends
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r145839008
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/19527#discussion_r145834490
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/OneHotEncoderEstimator.scala
---
@@ -0,0 +1,439 @@
+/*
+ * Licensed to the Apache
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18664
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18664
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82915/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18664
**[Test build #82915 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82915/testReport)**
for PR 18664 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19534
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82914/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19534
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19534
**[Test build #82914 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82914/testReport)**
for PR 19534 at commit
Github user ron8hu commented on a diff in the pull request:
https://github.com/apache/spark/pull/19479#discussion_r145828713
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/Statistics.scala
---
@@ -216,65 +218,61 @@ object ColumnStat extends
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19530
fyi it looks like this is cleanup from removing a broadcast in #18152
---
-
To unsubscribe, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19486
**[Test build #82916 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82916/testReport)**
for PR 19486 at commit
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/19530
LGTM
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user ash211 commented on the issue:
https://github.com/apache/spark/pull/19486
Updated
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail:
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19528
**[Test build #3955 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3955/consoleFull)**
for PR 19528 at commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19537
**[Test build #3953 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3953/testReport)**
for PR 19537 at commit
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Merged build finished. Test PASSed.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18805
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/82911/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18805
**[Test build #82911 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82911/testReport)**
for PR 18805 at commit
Github user bdrillard commented on the issue:
https://github.com/apache/spark/pull/19518
@kiszk Ah, thanks for the link back to that discussion. I'll make
modifications to the trials for better data.
---
-
To
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/19439
**[Test build #82913 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/82913/testReport)**
for PR 19439 at commit
1 - 100 of 245 matches
Mail list logo
