[GitHub] [spark] zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r406614726
##
File path:
mllib/src/test/scala/org/apache/spark/ml/stat/ChiSquareTestSuite.scala
##
@@ -117,15 +117,15 @@ class ChiSquareTestSuite
withClue("ChiSquare should throw an exception when given a
continuous-valued label") {
intercept[SparkException] {
val df = spark.createDataFrame(continuousLabel)
-ChiSquareTest.test(df, "features", "label")
+ChiSquareTest.test(df, "features", "label").count()
Review comment:
need to add an action to trigger the computation
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r406614762
##
File path:
mllib/src/test/scala/org/apache/spark/ml/stat/ChiSquareTestSuite.scala
##
@@ -117,15 +117,15 @@ class ChiSquareTestSuite
withClue("ChiSquare should throw an exception when given a
continuous-valued label") {
intercept[SparkException] {
val df = spark.createDataFrame(continuousLabel)
-ChiSquareTest.test(df, "features", "label")
+ChiSquareTest.test(df, "features", "label").count()
}
}
val continuousFeature = Seq.fill(tooManyCategories)(
LabeledPoint(random.nextInt(2), Vectors.dense(random.nextDouble(
withClue("ChiSquare should throw an exception when given continuous-valued
features") {
intercept[SparkException] {
val df = spark.createDataFrame(continuousFeature)
-ChiSquareTest.test(df, "features", "label")
+ChiSquareTest.test(df, "features", "label").count()
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
zhengruifeng commented on a change in pull request #28176: [SPARK-31301][ML]
Flatten the result dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#discussion_r406614269
##
File path: mllib/src/main/scala/org/apache/spark/ml/stat/ChiSquareTest.scala
##
@@ -63,40 +55,47 @@ object ChiSquareTest {
*/
@Since("2.2.0")
def test(dataset: DataFrame, featuresCol: String, labelCol: String):
DataFrame = {
-val spark = dataset.sparkSession
-import spark.implicits._
-
-SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
-SchemaUtils.checkNumericType(dataset.schema, labelCol)
-val rdd = dataset.select(col(labelCol).cast("double"),
col(featuresCol)).as[(Double, Vector)]
- .rdd.map { case (label, features) => OldLabeledPoint(label,
OldVectors.fromML(features)) }
-val testResults = OldStatistics.chiSqTest(rdd)
-val pValues = Vectors.dense(testResults.map(_.pValue))
-val degreesOfFreedom = testResults.map(_.degreesOfFreedom)
-val statistics = Vectors.dense(testResults.map(_.statistic))
-spark.createDataFrame(Seq(ChiSquareResult(pValues, degreesOfFreedom,
statistics)))
+test(dataset, featuresCol, labelCol, false)
}
/**
* @param dataset DataFrame of categorical labels and categorical features.
* Real-valued features will be treated as categorical for
each distinct value.
* @param featuresCol Name of features column in dataset, of type `Vector`
(`VectorUDT`)
* @param labelCol Name of label column in dataset, of any numerical type
- * @return Array containing the SelectionTestResult for every feature
against the label.
+ * @param flatten If false, the returned DataFrame contains only a single
Row, otherwise, one
+ * row per feature.
*/
@Since("3.1.0")
- def testChiSquare(
- dataset: Dataset[_],
+ def test(
+ dataset: DataFrame,
featuresCol: String,
- labelCol: String): Array[SelectionTestResult] = {
-
+ labelCol: String,
+ flatten: Boolean): DataFrame = {
SchemaUtils.checkColumnType(dataset.schema, featuresCol, new VectorUDT)
SchemaUtils.checkNumericType(dataset.schema, labelCol)
-val input = dataset.select(col(labelCol).cast(DoubleType),
col(featuresCol)).rdd
- .map { case Row(label: Double, features: Vector) =>
-OldLabeledPoint(label, OldVectors.fromML(features))
- }
-val chiTestResult = OldStatistics.chiSqTest(input)
-chiTestResult.map(r => new ChiSqTestResult(r.pValue, r.degreesOfFreedom,
r.statistic))
+
+val spark = dataset.sparkSession
+import spark.implicits._
+
+val data = dataset.select(col(labelCol).cast("double"),
col(featuresCol)).rdd
+ .map { case Row(label: Double, vec: Vector) => (label,
OldVectors.fromML(vec)) }
+val resultRDD = OldChiSqTest.computeChiSquared(data)
+
+if (flatten) {
+ resultRDD.map { case (col, (pValue, degreesOfFreedom, statistic, _)) =>
+(col, pValue, degreesOfFreedom, statistic)
+ }.toDF("featureIndex", "pValue", "degreesOfFreedom", "statistic")
+} else {
+ resultRDD.map { case (col, (pValue, degreesOfFreedom, statistic, _)) =>
+(0, (col, pValue, degreesOfFreedom, statistic))
+ }.groupByKey().map { case (_, seq) =>
Review comment:
collect results in some executor instead of the driver
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406612146
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
In the above, `options.readOnly=None` is added because we don't failure at
missing `options.readOnly`.
If possible, it would be great if we can use `mount.readOnly` only.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406612146
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
In the above, `options.readOnly=None` is added because we don't failure at
missing `options.readOnly`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611440
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
Do we need this option additionally? Then, could you add some comment about
all combinations, please?
1. mount.readOnly=true, options.readOnly=true
2. mount.readOnly=true, options.readOnly=false
3. mount.readOnly=true, options.readOnly=None
4. mount.readOnly=false, options.readOnly=true
5. mount.readOnly=false, options.readOnly=false
6. mount.readOnly=false, options.readOnly=None
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611440
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
Do we need this option additionally? Then, could you add some comment about
all combinations, please?
1. mount.readOnly=true, options.readOnly=true
2. mount.readOnly=true, options.readOnly=false
3. mount.readOnly=false, options.readOnly=true
4. mount.readOnly=false, options.readOnly=false
5. mount.readOnly=false, options.readOnly=None
6. mount.readOnly=false, options.readOnly=None
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611618
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtilsSuite.scala
##
@@ -117,4 +117,61 @@ class KubernetesVolumeUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("hostPath.volumeName.options.path"))
}
+
+ test("Parses nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
Review comment:
In this case, we don't need
`sparkConf.set("test.nfs.volumeName.options.readOnly", "true")`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611440
##
File path:
resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala
##
@@ -385,13 +385,16 @@ private[spark] object Config extends Logging {
val KUBERNETES_VOLUMES_HOSTPATH_TYPE = "hostPath"
val KUBERNETES_VOLUMES_PVC_TYPE = "persistentVolumeClaim"
val KUBERNETES_VOLUMES_EMPTYDIR_TYPE = "emptyDir"
+ val KUBERNETES_VOLUMES_NFS_TYPE = "nfs"
val KUBERNETES_VOLUMES_MOUNT_PATH_KEY = "mount.path"
val KUBERNETES_VOLUMES_MOUNT_SUBPATH_KEY = "mount.subPath"
val KUBERNETES_VOLUMES_MOUNT_READONLY_KEY = "mount.readOnly"
val KUBERNETES_VOLUMES_OPTIONS_PATH_KEY = "options.path"
val KUBERNETES_VOLUMES_OPTIONS_CLAIM_NAME_KEY = "options.claimName"
val KUBERNETES_VOLUMES_OPTIONS_MEDIUM_KEY = "options.medium"
val KUBERNETES_VOLUMES_OPTIONS_SIZE_LIMIT_KEY = "options.sizeLimit"
+ val KUBERNETES_VOLUMES_OPTIONS_SERVER_KEY = "options.server"
+ val KUBERNETES_VOLUMES_OPTIONS_READ_ONLY_KEY = "options.readOnly"
Review comment:
Do we need this option additionally? Then, could you add some comment about
all combinations, please?
1. mount.readOnly=true, options.readOnly=true
2. mount.readOnly=true, options.readOnly=false
3. mount.readOnly=false, options.readOnly=true
4. mount.readOnly=false, options.readOnly=false
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611075
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtilsSuite.scala
##
@@ -117,4 +117,61 @@ class KubernetesVolumeUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("hostPath.volumeName.options.path"))
}
+
+ test("Parses nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val volumeSpec = KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf,
"test.").head
+assert(volumeSpec.volumeName === "volumeName")
+assert(volumeSpec.mountPath === "/path")
+assert(volumeSpec.mountReadOnly === true)
+assert(volumeSpec.volumeConf.asInstanceOf[KubernetesNFSVolumeConf] ===
+ KubernetesNFSVolumeConf("/share", None, "nfs.example.com"))
+ }
+
+ test("Parses read/write nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "false")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.readOnly", "false")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val volumeSpec = KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf,
"test.").head
+assert(volumeSpec.volumeName === "volumeName")
+assert(volumeSpec.mountPath === "/path")
+assert(volumeSpec.mountReadOnly === false)
+assert(volumeSpec.volumeConf.asInstanceOf[KubernetesNFSVolumeConf] ===
+ KubernetesNFSVolumeConf("/share", Some(false), "nfs.example.com"))
+ }
+
+ test("Fails on missing path option") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.pth", "/share")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val e = intercept[NoSuchElementException] {
+ KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf, "test.")
+}
+assert(e.getMessage.contains("nfs.volumeName.options.path"))
+ }
+
+ test("Fails on missing server option") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.s", "nfs.example.com")
Review comment:
ditto.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406611010
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtilsSuite.scala
##
@@ -117,4 +117,61 @@ class KubernetesVolumeUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("hostPath.volumeName.options.path"))
}
+
+ test("Parses nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val volumeSpec = KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf,
"test.").head
+assert(volumeSpec.volumeName === "volumeName")
+assert(volumeSpec.mountPath === "/path")
+assert(volumeSpec.mountReadOnly === true)
+assert(volumeSpec.volumeConf.asInstanceOf[KubernetesNFSVolumeConf] ===
+ KubernetesNFSVolumeConf("/share", None, "nfs.example.com"))
+ }
+
+ test("Parses read/write nfs volumes correctly") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "false")
+sparkConf.set("test.nfs.volumeName.options.path", "/share")
+sparkConf.set("test.nfs.volumeName.options.readOnly", "false")
+sparkConf.set("test.nfs.volumeName.options.server", "nfs.example.com")
+
+val volumeSpec = KubernetesVolumeUtils.parseVolumesWithPrefix(sparkConf,
"test.").head
+assert(volumeSpec.volumeName === "volumeName")
+assert(volumeSpec.mountPath === "/path")
+assert(volumeSpec.mountReadOnly === false)
+assert(volumeSpec.volumeConf.asInstanceOf[KubernetesNFSVolumeConf] ===
+ KubernetesNFSVolumeConf("/share", Some(false), "nfs.example.com"))
+ }
+
+ test("Fails on missing path option") {
+val sparkConf = new SparkConf(false)
+sparkConf.set("test.nfs.volumeName.mount.path", "/path")
+sparkConf.set("test.nfs.volumeName.mount.readOnly", "true")
+sparkConf.set("test.nfs.volumeName.options.pth", "/share")
Review comment:
Shall we remove this line instead of using `pth`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611885344 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25749/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611885354 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611885358 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25749/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611885354 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException
AmplabJenkins removed a comment on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException URL: https://github.com/apache/spark/pull/28174#issuecomment-611885259 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException
AmplabJenkins removed a comment on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException URL: https://github.com/apache/spark/pull/28174#issuecomment-611885261 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121052/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611885358 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25749/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException
AmplabJenkins commented on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException URL: https://github.com/apache/spark/pull/28174#issuecomment-611885259 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException
AmplabJenkins commented on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException URL: https://github.com/apache/spark/pull/28174#issuecomment-611885261 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121052/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException
SparkQA removed a comment on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException URL: https://github.com/apache/spark/pull/28174#issuecomment-611847152 **[Test build #121052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121052/testReport)** for PR 28174 at commit [`23c425b`](https://github.com/apache/spark/commit/23c425b1b6e1f318e2398339adb69d20be660cc7). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException
SparkQA commented on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException URL: https://github.com/apache/spark/pull/28174#issuecomment-611884774 **[Test build #121052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121052/testReport)** for PR 28174 at commit [`23c425b`](https://github.com/apache/spark/commit/23c425b1b6e1f318e2398339adb69d20be660cc7). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on a change in pull request #27364: [SPARK-31394][K8S]
Adds support for Kubernetes NFS volume mounts
URL: https://github.com/apache/spark/pull/27364#discussion_r406608506
##
File path:
resource-managers/kubernetes/core/src/test/scala/org/apache/spark/deploy/k8s/KubernetesVolumeUtilsSuite.scala
##
@@ -117,4 +117,61 @@ class KubernetesVolumeUtilsSuite extends SparkFunSuite {
}
assert(e.getMessage.contains("hostPath.volumeName.options.path"))
}
+
+ test("Parses nfs volumes correctly") {
Review comment:
nit.
```
- test("Parses nfs volumes correctly") {
+ test("Parses read-only nfs volumes correctly") {
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
AmplabJenkins removed a comment on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec URL: https://github.com/apache/spark/pull/28175#issuecomment-611884308 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121054/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
AmplabJenkins commented on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec URL: https://github.com/apache/spark/pull/28175#issuecomment-611884302 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
SparkQA removed a comment on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec URL: https://github.com/apache/spark/pull/28175#issuecomment-611864155 **[Test build #121054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121054/testReport)** for PR 28175 at commit [`806a143`](https://github.com/apache/spark/commit/806a1433c893c939a00841f6a3bfd6d94899cae3). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
AmplabJenkins commented on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec URL: https://github.com/apache/spark/pull/28175#issuecomment-611884308 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121054/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
AmplabJenkins removed a comment on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec URL: https://github.com/apache/spark/pull/28175#issuecomment-611884302 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
SparkQA commented on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec URL: https://github.com/apache/spark/pull/28175#issuecomment-611884163 **[Test build #121054 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121054/testReport)** for PR 28175 at commit [`806a143`](https://github.com/apache/spark/commit/806a1433c893c939a00841f6a3bfd6d94899cae3). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611882927 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121055/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611882920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611882920 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611882927 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121055/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
SparkQA removed a comment on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611867286 **[Test build #121055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121055/testReport)** for PR 28176 at commit [`0107dc4`](https://github.com/apache/spark/commit/0107dc47ba432988f686e9019788e76ffc145753). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts
dongjoon-hyun commented on issue #27364: [SPARK-31394][K8S] Adds support for Kubernetes NFS volume mounts URL: https://github.com/apache/spark/pull/27364#issuecomment-611882673 Thank you for your detailed explanation. Please copy some of them into the PR description. The PR description will be the permanent commit log. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
SparkQA commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611882567 **[Test build #121055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121055/testReport)** for PR 28176 at commit [`0107dc4`](https://github.com/apache/spark/commit/0107dc47ba432988f686e9019788e76ffc145753). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611880986 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25749/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
AmplabJenkins removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611881027 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
AmplabJenkins commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611881027 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611880660 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
AmplabJenkins commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611881033 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121060/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
AmplabJenkins removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611881033 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121060/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
SparkQA removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611875980 **[Test build #121060 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121060/testReport)** for PR 28157 at commit [`36cbcb7`](https://github.com/apache/spark/commit/36cbcb7168fd62b7f9acee320ae3809ec63f7cce). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
SparkQA commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611880869 **[Test build #121060 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121060/testReport)** for PR 28157 at commit [`36cbcb7`](https://github.com/apache/spark/commit/36cbcb7168fd62b7f9acee320ae3809ec63f7cce). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611880665 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25747/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611880660 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611880650 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25747/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611880665 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25747/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] iRakson commented on issue #28167: [SPARK-31106][SQL] Support is_json function
iRakson commented on issue #28167: [SPARK-31106][SQL] Support is_json function URL: https://github.com/apache/spark/pull/28167#issuecomment-611879823 > @iRakson Could you provide the links? > > * MySQL > * SQL Server > * IBM Db2 > * Sqlite > * MariaDB > * Amazon Redshift [MySQL](https://dev.mysql.com/doc/refman/8.0/en/json-attribute-functions.html#function_json-valid) [SQL Server](https://docs.microsoft.com/en-us/sql/t-sql/functions/isjson-transact-sql?view=sql-server-ver15) [IBM Db2](https://developer.ibm.com/technologies/systems/articles/sql-json-publishing-functions/) [Sqlite](https://www.sqlite.org/json1.html) [MariaDB](https://mariadb.com/kb/en/json_valid/) [Amazon Redshift](https://docs.aws.amazon.com/redshift/latest/dg/IS_VALID_JSON.html) Names may differ across DBMSs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #28163: [SPARK-31359][SQL][3.0] Speed up timestamps rebasing
gatorsmile commented on issue #28163: [SPARK-31359][SQL][3.0] Speed up timestamps rebasing URL: https://github.com/apache/spark/pull/28163#issuecomment-611878265 We need explicitly say the reason why we need it is for fixing the regression instead of making it faster. Is my understanding right? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] MaxGekk commented on issue #28169: [SPARK-31398][SQL][test-hive1.2] Speed up reading dates in ORC
MaxGekk commented on issue #28169: [SPARK-31398][SQL][test-hive1.2] Speed up reading dates in ORC URL: https://github.com/apache/spark/pull/28169#issuecomment-611876712 > Can you check the benchmark numbers with Spark 2.4? Just want to see how much perf regression we have in 3.0 after this patch. @cloud-fan To have comparable results, need to port: 1. NoOp datasource 2. Changes in Benchmark framework to save results to files This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
AmplabJenkins commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611876205 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611876234 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25747/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
AmplabJenkins removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611876205 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
AmplabJenkins commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611876209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25751/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
AmplabJenkins removed a comment on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611876209 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25751/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function
SparkQA commented on issue #28157: [SPARK-31390][SQL][DOCS] Document Window Function URL: https://github.com/apache/spark/pull/28157#issuecomment-611875980 **[Test build #121060 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121060/testReport)** for PR 28157 at commit [`36cbcb7`](https://github.com/apache/spark/commit/36cbcb7168fd62b7f9acee320ae3809ec63f7cce). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #28176: [SPARK-31301][ML] Flatten the result dataframe of tests in testChiSquare
zhengruifeng commented on issue #28176: [SPARK-31301][ML] Flatten the result
dataframe of tests in testChiSquare
URL: https://github.com/apache/spark/pull/28176#issuecomment-611875587
bin/spark-shell --driver-memory=2G
testcodes:
```
import org.apache.spark.ml.feature.ChiSqSelector
import org.apache.spark.ml.stat._
val df =
spark.read.format("libsvm").load("/data1/Datasets/webspam/webspam_wc_normalized_trigram.svm.10k")
val chi = ChiSquareTest.test(df, "features", "label")
// val chi = ChiSquareTest.test(df, "features", "label", true) // added in
this PR
chi.show
val selector = new
ChiSqSelector().setNumTopFeatures(1000).setLabelCol("label").setFeaturesCol("features")
val model = selector.fit(df)
```
Existing `ChiSquareTest.test` and `selector.fit` will crash Spark-Shell due
to OOM; while new methods in this PR work fine:
Existing `ChiSquareTest.test`
```
Driver stacktrace:
at
org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2094)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2043)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2042)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2042)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1020)
at
org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1020)
at scala.Option.foreach(Option.scala:407)
at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1020)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2274)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2223)
at
org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2212)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:822)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2108)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2129)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2148)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:2173)
at org.apache.spark.rdd.RDD.$anonfun$collect$1(RDD.scala:1030)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
at
org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:414)
at org.apache.spark.rdd.RDD.collect(RDD.scala:1029)
at
org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredSparseFeatures(ChiSqTest.scala:148)
at
org.apache.spark.mllib.stat.test.ChiSqTest$.chiSquaredFeatures(ChiSqTest.scala:88)
at org.apache.spark.mllib.stat.Statistics$.chiSqTest(Statistics.scala:192)
at org.apache.spark.ml.stat.ChiSquareTest$.test(ChiSquareTest.scala:73)
... 49 elided
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
```
new `ChiSquareTest.test(df, "features", "label", true)`
```
scala> val chi = ChiSquareTest.test(df, "features", "label", true)
chi: org.apache.spark.sql.DataFrame = [featureIndex: int, pValue: double ...
2 more fields]
scala> chi.show
20/04/10 11:28:51 WARN Executor: Managed memory leak detected; size =
843044942 bytes, TID = 49
++---++--+
|featureIndex| pValue|degreesOfFreedom| statistic|
++---++--+
| 3184020|1.0| 0| 0.0|
| 6697512|0.21072480020843432| 2|3.1144045224283925|
| 3387408|1.0| 0| 0.0|
| 5907828|1.0| 0| 0.0|
| 6582516|1.0| 0| 0.0|
| 3490824|1.0| 0| 0.0|
| 5916408|1.0| 0| 0.0|
| 2292732|1.0| 0| 0.0|
| 7447896|1.0| 0| 0.0|
| 2157804|1.0| 0| 0.0|
| 1732608|1.0| 0| 0.0|
| 3427284|1.0| 0| 0.0|
| 5284836|
[GitHub] [spark] AmplabJenkins removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
AmplabJenkins removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611874586 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121058/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
SparkQA removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611871614 **[Test build #121058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121058/testReport)** for PR 28151 at commit [`945b28c`](https://github.com/apache/spark/commit/945b28c51193017fb93b1d5e7710e1f7b14ae014). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
AmplabJenkins removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611874580 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
AmplabJenkins commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611874580 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
AmplabJenkins commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611874586 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121058/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
SparkQA commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611874498 **[Test build #121058 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121058/testReport)** for PR 28151 at commit [`945b28c`](https://github.com/apache/spark/commit/945b28c51193017fb93b1d5e7710e1f7b14ae014). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
dongjoon-hyun closed pull request #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
dongjoon-hyun commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611873717 Thank you for review and approval, @HyukjinKwon . Since this PR is irrelevant to the Jenkins UT, I'll merge this. The JDK11 docker image generation is verified locally. Merged to master/3.0. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException
dongjoon-hyun commented on issue #28174: [SPARK-18886][CORE][TESTS][FOLLOWUP] Fix a test failure due to InvalidUseOfMatchersException URL: https://github.com/apache/spark/pull/28174#issuecomment-611873172 Thank you for reviewing and merging, @HyukjinKwon ! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611873059 **[Test build #121059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121059/testReport)** for PR 28171 at commit [`1d3fe79`](https://github.com/apache/spark/commit/1d3fe79324420c3033457e56ad3bef2a29cbb3a1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
cloud-fan commented on issue #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec URL: https://github.com/apache/spark/pull/28175#issuecomment-611872533 can you put the before/after screenshots? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
cloud-fan commented on a change in pull request #28175:
[SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in
CustomShuffleReaderExec
URL: https://github.com/apache/spark/pull/28175#discussion_r406597200
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CustomShuffleReaderExec.scala
##
@@ -124,6 +104,24 @@ case class CustomShuffleReaderExec private(
Map("numSkewedPartitions" -> metrics)
}
+ private def sendPartitionDataSizeMetrics(executionId: String): Unit = {
+val mapStats = shuffleStage.get.mapStats.bytesByPartitionId
+partitionSpecs.foreach {
+ case CoalescedPartitionSpec(startReducerIndex, endReducerIndex) =>
+val dataSize =
startReducerIndex.until(endReducerIndex).map(mapStats(_)).sum
+metrics("partitionDataSize").set(dataSize)
+SQLMetrics.postDriverMetricUpdates(
+ sparkContext, executionId,
+ metrics.filter(_._1 == "partitionDataSize").values.toSeq)
Review comment:
can we look up the `partitionDataSize` `SQLMetric` at the beginning of this
method? then here we can simply write `Seq(metric)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28175: [SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in CustomShuffleReaderExec
cloud-fan commented on a change in pull request #28175:
[SPARK-31253][SQL][FOLLOW-UP] Improve the partition data size metrics in
CustomShuffleReaderExec
URL: https://github.com/apache/spark/pull/28175#discussion_r406596906
##
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CustomShuffleReaderExec.scala
##
@@ -124,6 +104,24 @@ case class CustomShuffleReaderExec private(
Map("numSkewedPartitions" -> metrics)
}
+ private def sendPartitionDataSizeMetrics(executionId: String): Unit = {
+val mapStats = shuffleStage.get.mapStats.bytesByPartitionId
+partitionSpecs.foreach {
+ case CoalescedPartitionSpec(startReducerIndex, endReducerIndex) =>
+val dataSize =
startReducerIndex.until(endReducerIndex).map(mapStats(_)).sum
+metrics("partitionDataSize").set(dataSize)
Review comment:
we should call `add`, not `set`. The SQL metrics itself should contain the
sum value.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
AmplabJenkins removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611871881 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
AmplabJenkins commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611871881 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
AmplabJenkins commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611871882 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25750/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
AmplabJenkins removed a comment on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611871882 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25750/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
HyukjinKwon commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611871605 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] PerilousApricot commented on issue #28159: [WIP][SPARK-31363][SQL] Add DataSourceRegisterV2
PerilousApricot commented on issue #28159: [WIP][SPARK-31363][SQL] Add
DataSourceRegisterV2
URL: https://github.com/apache/spark/pull/28159#issuecomment-611871616
I did a POC with Spark2.4 as well, which can be found at
https://github.com/PerilousApricot/spark/tree/feature/registerv2-24
I compiled my datasource and both Spark patches and verified that the patch
correctly loads the right version of the plugin. You can verify it yourself
with the following jar
http://mirror.accre.vanderbilt.edu/spark/laurelin-both.jar and input file
https://github.com/spark-root/laurelin/raw/master/testdata/stdvector.root
```
scala> val df =
spark.read.format("root").option("tree","tvec").load("stdvector.root")
val df =
spark.read.format("root").option("tree","tvec").load("stdvector.root")
df: org.apache.spark.sql.DataFrame = [vpx: array, vpy: array
... 7 more fields]
scala> df.select("vpx").show()
++
| vpx|
++
| [-2.409915]|
|[-0.39048654, 0.3...|
|[-0.0517636, 0.06...|
|[-0.4088529, 0.23...|
|[0.22863834, -0.3...|
|[1.5184512, -1.10...|
|[-1.2615219, 1.17...|
|[0.26021498, -0.3...|
|[-1.2066879, 0.91...|
|[0.54583037, -0.4...|
++
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref*
SparkQA commented on issue #28151: [SPARK-31383][SQL][DOC] Clean up the SQL documents in docs/sql-ref* URL: https://github.com/apache/spark/pull/28151#issuecomment-611871614 **[Test build #121058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121058/testReport)** for PR 28151 at commit [`945b28c`](https://github.com/apache/spark/commit/945b28c51193017fb93b1d5e7710e1f7b14ae014). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] gatorsmile commented on issue #28167: [SPARK-31106][SQL] Support is_json function
gatorsmile commented on issue #28167: [SPARK-31106][SQL] Support is_json function URL: https://github.com/apache/spark/pull/28167#issuecomment-611870941 @iRakson Could you provide the links? - MySQL - SQL Server - IBM Db2 - Sqlite - MariaDB - Amazon Redshift This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611870440 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25744/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611870435 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611870422 Kubernetes integration test status success URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/25744/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611870435 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611870440 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25744/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28169: [SPARK-31398][SQL][test-hive1.2] Speed up reading dates in ORC
cloud-fan commented on issue #28169: [SPARK-31398][SQL][test-hive1.2] Speed up reading dates in ORC URL: https://github.com/apache/spark/pull/28169#issuecomment-611868922 LGTM. Can you check the benchmark numbers with Spark 2.4? Just want to see how much perf regression we have in 3.0 after this patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611868620 **[Test build #121057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121057/testReport)** for PR 28171 at commit [`1d3fe79`](https://github.com/apache/spark/commit/1d3fe79324420c3033457e56ad3bef2a29cbb3a1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611868133 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121051/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611868133 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/121051/ Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611868130 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA removed a comment on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611845235 **[Test build #121051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121051/testReport)** for PR 28171 at commit [`1d3fe79`](https://github.com/apache/spark/commit/1d3fe79324420c3033457e56ad3bef2a29cbb3a1). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
AmplabJenkins commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611868130 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh`
SparkQA commented on issue #28171: [SPARK-31401][K8S] Show JDK11 usage in `bin/docker-image-tool.sh` URL: https://github.com/apache/spark/pull/28171#issuecomment-611867969 **[Test build #121051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/121051/testReport)** for PR 28171 at commit [`1d3fe79`](https://github.com/apache/spark/commit/1d3fe79324420c3033457e56ad3bef2a29cbb3a1). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
AmplabJenkins removed a comment on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611867523 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
AmplabJenkins removed a comment on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611867529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25748/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611867467 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25746/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
AmplabJenkins commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611867529 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25748/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions
AmplabJenkins commented on issue #28158: [SPARK-25154][SQL] Support NOT IN sub-queries inside nested OR conditions URL: https://github.com/apache/spark/pull/28158#issuecomment-611867523 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611867466 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
AmplabJenkins commented on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611867466 Merged build finished. Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare
AmplabJenkins removed a comment on issue #28176: [SPARK-31301][ML] flatten the result dataframe of tests in testChiSquare URL: https://github.com/apache/spark/pull/28176#issuecomment-611867467 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25746/ Test PASSed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
