Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
This was superceded by #19643
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands
Github user mariusvniekerk closed the pull request at:
https://github.com/apache/spark/pull/15666
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/19643#discussion_r149085525
--- Diff: R/pkg/R/context.R ---
@@ -319,6 +319,27 @@ spark.addFile <- function(path, recursive = FALSE) {
invisible(callJMethod(sc, "
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r122578282
--- Diff: R/pkg/R/context.R ---
@@ -319,6 +319,34 @@ spark.addFile <- function(path, recursive = FALSE) {
invisible(callJMethod(sc, "
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r122578275
--- Diff: R/pkg/R/context.R ---
@@ -319,6 +319,34 @@ spark.addFile <- function(path, recursive = FALSE) {
invisible(callJMethod(sc, "
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
@HyukjinKwon Any hints what's needed to get the R stuff passing? I don't
really have a windows testbed that I can use.
---
If your project is set up for it, you can reply to this
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/16766
Let me rebase this. I don't currently have a clean way of testing this on
Windows
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r106781948
--- Diff: R/pkg/R/context.R ---
@@ -319,6 +319,34 @@ spark.addFile <- function(path, recursive = FALSE) {
invisible(callJMethod(sc, "
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r104908469
--- Diff: R/pkg/R/context.R ---
@@ -319,6 +319,34 @@ spark.addFile <- function(path, recursive = FALSE) {
invisible(callJMethod(sc, "
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r104486665
--- Diff: R/pkg/inst/tests/testthat/test_context.R ---
@@ -167,6 +167,18 @@ test_that("spark.lapply should perform simple
trans
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
Ah thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
whoops.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
Seems to be something in pyspark.SparkContext.addJar:10: ERROR: Unexpected
indentation. ?
what exactly does it want in that docstring?
---
If your project is set up for it, you can
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
@holdenk done
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
I'll see if I can rebase it tomorrow
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r100177099
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1802,19 +1802,34 @@ class SparkContext(config: SparkConf) extends
Logging
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r100176188
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1802,19 +1802,34 @@ class SparkContext(config: SparkConf) extends
Logging
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15821
Probably a good thing to look at is the R pieces since that is effectively
constrained to InternalRow
---
If your project is set up for it, you can reply to this email and have your
reply
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/16766
@felixcheung This does not touch any of the coalesce internals. Only
allows setting a partitionCoalescer similar to what is already available in
rdd.coalesce
---
If your project is set
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16766#discussion_r99369813
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
---
@@ -497,7 +496,9 @@ case class UnionExec(children
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16766#discussion_r99366754
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -117,6 +134,34 @@ class DatasetSuite extends QueryTest with
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16766#discussion_r99366143
--- Diff: sql/core/src/test/scala/org/apache/spark/sql/DatasetSuite.scala
---
@@ -17,24 +17,41 @@
package org.apache.spark.sql
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16766#discussion_r99363149
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -823,6 +825,17 @@ case class
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/16766#discussion_r99132600
--- Diff:
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala
---
@@ -823,6 +825,17 @@ case class
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
Yeah I'll be there
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enable
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
@holdenk Anything i can do from my side to help this guy along?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user mariusvniekerk opened a pull request:
https://github.com/apache/spark/pull/16766
[SPARK-19426][SQL] Custom coalesce for Dataset
## What changes were proposed in this pull request?
This adds support for using the PartitionCoalescer features added in #11865
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r93730314
--- Diff: core/src/main/scala/org/apache/spark/TestUtils.scala ---
@@ -164,6 +164,27 @@ private[spark] object TestUtils {
createCompiledClass
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r93729928
--- Diff: core/src/main/scala/org/apache/spark/TestUtils.scala ---
@@ -164,6 +164,27 @@ private[spark] object TestUtils {
createCompiledClass
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
Rebased.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or
Github user mariusvniekerk closed the pull request at:
https://github.com/apache/spark/pull/15666
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the
GitHub user mariusvniekerk reopened a pull request:
https://github.com/apache/spark/pull/15666
[SPARK-11421] [Core][Python][R] Added ability for addJar to augment the
current classloader
## What changes were proposed in this pull request?
Adds a flag to sc.addJar to add
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15821
So this is very cool stuff.
Would it be reasonable to add some api pieces so that on the python side
things like DataFrame.mapPartitions makes use of Apache Arrow to lower the
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/15666
@HyukjinKwon there seems to be something weird with the appveyor checks?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r85865112
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1700,19 +1700,34 @@ class SparkContext(config: SparkConf) extends
Logging
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/15666#discussion_r85833766
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -1700,19 +1700,34 @@ class SparkContext(config: SparkConf) extends
Logging
GitHub user mariusvniekerk opened a pull request:
https://github.com/apache/spark/pull/15666
[SPARK-11421] [Core][Python][R] Added ability for addJar to augment the
current classloader
## What changes were proposed in this pull request?
Adds a flag to sc.addJar to add the
Github user mariusvniekerk commented on the issue:
https://github.com/apache/spark/pull/9313
So since py4j now uses the context classloader, we can remove the python
pieces about loading a class by name.
@holdenk If you want I can revisit this PR.
This case
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/9861#discussion_r45620973
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRDD.scala
---
@@ -489,6 +492,13 @@ private[sql] class JDBCRDD
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9861#issuecomment-158701845
Not entirely sure why this causes NPE exceptions in some of the unit
tests...
---
If your project is set up for it, you can reply to this email and have your
GitHub user mariusvniekerk opened a pull request:
https://github.com/apache/spark/pull/9861
[SPARK-11881][SQL] Fix for postgresql fetchsize > 0
Reference:
https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor
In order for PostgreSQL to honor the fetchS
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9662#issuecomment-156285098
These test failures don't seem to be related?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as wel
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/9662#discussion_r44664911
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -121,6 +145,12 @@ object JdbcUtils extends
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9662#issuecomment-156121867
I've added write support in #9137 as well if you want to just use it from
there.
---
If your project is set up for it, you can reply to this email and
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9137#issuecomment-154415116
@JoshRosen Guess its refactor time due to SPARK-11541.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9137#issuecomment-152561158
Is the best approach to rebase or just merge master into this and resolve
conflicts?
---
If your project is set up for it, you can reply to this email and have
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9137#issuecomment-152309023
I also need to rebase this thing against master again it seems
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/9137#discussion_r43439253
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -121,6 +122,21 @@ object JdbcUtils extends
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/9137#discussion_r43437880
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
---
@@ -171,21 +187,9 @@ object JdbcUtils extends
Github user mariusvniekerk commented on a diff in the pull request:
https://github.com/apache/spark/pull/9137#discussion_r43437555
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/jdbc/JdbcDialects.scala ---
@@ -207,6 +225,25 @@ case object PostgresDialect extends
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9137#issuecomment-149907705
I'll add tests once #8101 is merged in
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9137#issuecomment-149857945
Sure. Had to refactor a little to work around type erasure warnings
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/4549#issuecomment-149619987
I've given this a shot in https://github.com/apache/spark/pull/9137
---
If your project is set up for it, you can reply to this email and have your
reply a
Github user mariusvniekerk commented on the pull request:
https://github.com/apache/spark/pull/9137#issuecomment-149393173
Still need to add some additional types from
https://github.com/pgjdbc/pgjdbc/blob/master/org/postgresql/jdbc2/TypeInfoCache.java#L70
---
If your
GitHub user mariusvniekerk opened a pull request:
https://github.com/apache/spark/pull/9137
[SPARK-10186][SQL] Array types using JDBCRDD and postgres
This change allows reading from jdbc array column types for the postgresql
dialect.
This also opens up some implementation
55 matches
Mail list logo
