HeartSaVioR closed pull request #45341: [SPARK-47272][SS] Add MapState
implementation for State API v2.
URL: https://github.com/apache/spark/pull/45341
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go
HeartSaVioR commented on PR #45341:
URL: https://github.com/apache/spark/pull/45341#issuecomment-1992674919
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
HeartSaVioR commented on PR #45341:
URL: https://github.com/apache/spark/pull/45341#issuecomment-1992673425
Yeah, doesn't look to be related. I'm ignoring the CI error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
jingz-db commented on PR #45341:
URL: https://github.com/apache/spark/pull/45341#issuecomment-1992613952
Test suite failure seems unrelated in pyspark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
jingz-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1521842235
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -73,24 +74,31 @@ class MapStateImpl[K, V](
}
/** Get the map
HeartSaVioR commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520711461
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -73,24 +74,31 @@ class MapStateImpl[K, V](
}
/** Get the map
HeartSaVioR commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520713399
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -73,24 +74,31 @@ class MapStateImpl[K, V](
}
/** Get the map
HeartSaVioR commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520711461
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -73,24 +74,31 @@ class MapStateImpl[K, V](
}
/** Get the map
jingz-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520699470
##
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/MapStateSuite.scala:
##
@@ -67,6 +67,7 @@ class MapStateSuite extends StateVariableSuiteBase
HeartSaVioR commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520688468
##
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/MapStateSuite.scala:
##
@@ -67,6 +67,7 @@ class MapStateSuite extends
HeartSaVioR commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520688468
##
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/MapStateSuite.scala:
##
@@ -67,6 +67,7 @@ class MapStateSuite extends
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520618069
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation
jingz-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520613982
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
jingz-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1520256936
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF)
HeartSaVioR commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1519035053
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation
anishshri-db commented on PR #45341:
URL: https://github.com/apache/spark/pull/45341#issuecomment-1982166978
@jingz-db - could you please fix this style error ?
```
[info] compiling 1 Scala source to
/home/runner/work/spark/spark/tools/target/scala-2.13/classes ...
[error]
jingz-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1515298710
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -86,3 +88,53 @@ object StateTypesEncoder {
new
jingz-db commented on PR #45341:
URL: https://github.com/apache/spark/pull/45341#issuecomment-1981833646
> @jingz-db - test failure seems related ?
Weirdly is passing locally. Let me resolve your comments and retrigger the
CI and see if it still fails. Thanks for the review!
--
anishshri-db commented on PR #45341:
URL: https://github.com/apache/spark/pull/45341#issuecomment-1981548558
@jingz-db - test failure seems related ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1514975454
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala:
##
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1514974624
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala:
##
@@ -0,0 +1,199 @@
+/*
+ * Licensed to the Apache Software
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1514973964
##
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/ValueStateSuite.scala:
##
@@ -218,3 +173,56 @@ class ValueStateSuite extends
jingz-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513621273
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -86,3 +88,53 @@ object StateTypesEncoder {
new
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513618200
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala:
##
@@ -0,0 +1,392 @@
+/*
+ * Licensed to the Apache Software
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513570476
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -86,3 +88,53 @@ object StateTypesEncoder {
new
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513570178
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -86,3 +88,53 @@ object StateTypesEncoder {
new
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513568839
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -86,3 +88,53 @@ object StateTypesEncoder {
new
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513559752
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513559241
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513544356
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala:
##
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513544356
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala:
##
@@ -0,0 +1,201 @@
+/*
+ * Licensed to the Apache Software
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513485844
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala:
##
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513485319
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala:
##
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513484718
##
sql/core/src/test/scala/org/apache/spark/sql/streaming/TransformWithMapStateSuite.scala:
##
@@ -0,0 +1,204 @@
+/*
+ * Licensed to the Apache Software
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513483233
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513480749
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -66,6 +68,38 @@ class StateTypesEncoder[GK](
keyRow
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1513480414
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/StateTypesEncoderUtils.scala:
##
@@ -66,6 +68,38 @@ class StateTypesEncoder[GK](
keyRow
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1512013529
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateEncoder.scala:
##
@@ -29,6 +29,14 @@ sealed trait RocksDBKeyStateEncoder {
anishshri-db commented on code in PR #45341:
URL: https://github.com/apache/spark/pull/45341#discussion_r1512002599
##
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MapStateImpl.scala:
##
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation
39 matches
Mail list logo
