hvanhovell closed pull request #45701: [SPARK-47545][CONNECT] Dataset `observe`
support for the Scala client
URL: https://github.com/apache/spark/pull/45701
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
hvanhovell commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2101304531
Merging!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsub
xupefei commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2088626203
> > @xupefei there is a genuine test failure. Can you check what is going on?
>
> It seems the test is flaky, even after the previous attempt to fix it:
#45173
I re-ran the t
xupefei commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2088228423
> @xupefei there is a genuine test failure. Can you check what is going on?
It seems the test is flaky, even after the previous attempt to fix it:
https://github.com/apache/spark/pu
hvanhovell commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2085537023
@xupefei there is a genuine test failure. Can you check what is going on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitH
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1583392160
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -198,6 +206,29 @@ private[sql] class SparkResult[T](
non
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1583350392
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,23 @@ class SparkSession private[sql] (
* Set to false
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1583349498
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,23 @@ class SparkSession private[sql] (
* Set to false
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572265083
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -27,18 +27,22 @@ import org.apache.arrow.vector.ipc.message.{Arr
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572264833
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572263903
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572148735
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572148735
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1572146508
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false to
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1571122260
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -27,18 +27,22 @@ import org.apache.arrow.vector.ipc.message.{
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1571108161
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/ClientE2ETestSuite.scala:
##
@@ -1511,6 +1514,46 @@ class ClientE2ETestSuite extends RemoteSparkS
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1571105638
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1571102181
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/SparkSession.scala:
##
@@ -813,6 +823,28 @@ class SparkSession private[sql] (
* Set to false
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1570681872
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -198,6 +206,29 @@ private[sql] class SparkResult[T](
nonEmp
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1569265956
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -198,6 +206,29 @@ private[sql] class SparkResult[T](
non
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1567676904
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) unde
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1567644036
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3397,7 +3488,11 @@ class Dataset[T] private[sql] (
sparkSession.analyze(p
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1567173975
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -27,18 +27,21 @@ import org.apache.arrow.vector.ipc.message.{Arr
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566210772
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3397,7 +3488,11 @@ class Dataset[T] private[sql] (
sparkSession.analyz
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566170352
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566171078
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566169201
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566164128
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -79,6 +82,7 @@ private[sql] class SparkResult[T](
private[
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566163349
##
connector/connect/common/src/main/scala/org/apache/spark/sql/connect/client/SparkResult.scala:
##
@@ -27,18 +27,21 @@ import org.apache.arrow.vector.ipc.message.{
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1566146921
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1561082565
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -131,13 +131,25 @@ import org.apache.spark.util.SparkClassUtils
class Dataset[
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559266569
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3397,7 +3488,11 @@ class Dataset[T] private[sql] (
sparkSession.analyze(p
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559262517
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) unde
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559250767
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) unde
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559250767
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) unde
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1559249905
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/CheckConnectJvmClientCompatibility.scala:
##
@@ -363,6 +363,8 @@ object CheckConnectJ
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1558066618
##
connector/connect/client/jvm/src/test/scala/org/apache/spark/sql/connect/client/CheckConnectJvmClientCompatibility.scala:
##
@@ -363,6 +363,8 @@ object CheckConne
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1557936456
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Observation.scala:
##
@@ -0,0 +1,46 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) u
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1557935463
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3397,7 +3488,11 @@ class Dataset[T] private[sql] (
sparkSession.analyz
hvanhovell commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1557934162
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -131,13 +131,25 @@ import org.apache.spark.util.SparkClassUtils
class Datas
xupefei commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2031447755
> So `df.collectObservations()` seems to be a new API available only in
Spark Connect Scala client?
Yes, similar to `df.attrs["observed_metrics"]` which is only in the Python
clien
xupefei commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1547415263
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3338,7 +3358,25 @@ class Dataset[T] private[sql] (
}
def observe(name:
ueshin commented on code in PR #45701:
URL: https://github.com/apache/spark/pull/45701#discussion_r1543431051
##
connector/connect/client/jvm/src/main/scala/org/apache/spark/sql/Dataset.scala:
##
@@ -3338,7 +3358,25 @@ class Dataset[T] private[sql] (
}
def observe(name:
ueshin commented on PR #45701:
URL: https://github.com/apache/spark/pull/45701#issuecomment-2025839011
So `df.collectObservations()` seems to be a new API available only in Spark
Connect Scala client?
--
This is an automated message from the Apache Git Service.
To respond to the message,
44 matches
Mail list logo