[
https://issues.apache.org/jira/browse/SPARK-49092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17870369#comment-17870369
]
Anish Shrigondekar commented on SPARK-49092:
Actually seems like this is working as
Anish Shrigondekar created SPARK-49092:
--
Summary: Fix broken doc link for state data source reader
Key: SPARK-49092
URL: https://issues.apache.org/jira/browse/SPARK-49092
Project: Spark
Anish Shrigondekar created SPARK-49048:
--
Summary: Support reading operator metadata for passed batchId
Key: SPARK-49048
URL: https://issues.apache.org/jira/browse/SPARK-49048
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-49046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17869422#comment-17869422
]
Anish Shrigondekar commented on SPARK-49046:
With processing time, we always return true for
[
https://issues.apache.org/jira/browse/SPARK-49046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17869422#comment-17869422
]
Anish Shrigondekar edited comment on SPARK-49046 at 7/29/24 6:12 PM:
-
Anish Shrigondekar created SPARK-49046:
--
Summary: Run new batches only if timers are registered
Key: SPARK-49046
URL: https://issues.apache.org/jira/browse/SPARK-49046
Project: Spark
Anish Shrigondekar created SPARK-49032:
--
Summary: Add schema path to metadata table entry within operator
metadata
Key: SPARK-49032
URL: https://issues.apache.org/jira/browse/SPARK-49032
[
https://issues.apache.org/jira/browse/SPARK-49021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-49021:
---
Epic Link: SPARK-46815
> Add support for state data source reader with list/map
Anish Shrigondekar created SPARK-49021:
--
Summary: Add support for state data source reader with list/map
state
Key: SPARK-49021
URL: https://issues.apache.org/jira/browse/SPARK-49021
Project:
[
https://issues.apache.org/jira/browse/SPARK-45939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar resolved SPARK-45939.
Resolution: Fixed
> SPIP: Structured Streaming - Arbitrary State API v2
>
Anish Shrigondekar created SPARK-48957:
--
Summary: Return classified store load exception type on load
failure
Key: SPARK-48957
URL: https://issues.apache.org/jira/browse/SPARK-48957
Project:
Anish Shrigondekar created SPARK-48903:
--
Summary: Update latestSnapshot version correctly on remote load
Key: SPARK-48903
URL: https://issues.apache.org/jira/browse/SPARK-48903
Project: Spark
Anish Shrigondekar created SPARK-48891:
--
Summary: Refactor state schema compatibility checker to handle all
state schema formats
Key: SPARK-48891
URL: https://issues.apache.org/jira/browse/SPARK-48891
[
https://issues.apache.org/jira/browse/SPARK-48886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-48886:
---
Epic Link: SPARK-46815
> Add version info for changelog format v2
>
Anish Shrigondekar created SPARK-48886:
--
Summary: Add version info for changelog format v2
Key: SPARK-48886
URL: https://issues.apache.org/jira/browse/SPARK-48886
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-48799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17862904#comment-17862904
]
Anish Shrigondekar commented on SPARK-48799:
PR here -
Anish Shrigondekar created SPARK-48799:
--
Summary: Support versioning for operator metadata reader/writer
and update callers
Key: SPARK-48799
URL: https://issues.apache.org/jira/browse/SPARK-48799
[
https://issues.apache.org/jira/browse/SPARK-48770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17861252#comment-17861252
]
Anish Shrigondekar edited comment on SPARK-48770 at 7/1/24 7:13 PM:
Anish Shrigondekar created SPARK-48770:
--
Summary: Read operator metadata only once on driver for store read
Key: SPARK-48770
URL: https://issues.apache.org/jira/browse/SPARK-48770
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-48687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17856883#comment-17856883
]
Anish Shrigondekar commented on SPARK-48687:
PR here -
Anish Shrigondekar created SPARK-48687:
--
Summary: Add changes to implement state schema validation in
planning phase on driver for stateful streaming queries
Key: SPARK-48687
URL:
Anish Shrigondekar created SPARK-48543:
--
Summary: Track invalid unsafe row exception explicitly as error
class
Key: SPARK-48543
URL: https://issues.apache.org/jira/browse/SPARK-48543
Project:
Anish Shrigondekar created SPARK-48513:
--
Summary: Use NERF framework for state schema compatibility
exception
Key: SPARK-48513
URL: https://issues.apache.org/jira/browse/SPARK-48513
Project:
Anish Shrigondekar created SPARK-48470:
--
Summary: Rename listState functions for single and list type
appends
Key: SPARK-48470
URL: https://issues.apache.org/jira/browse/SPARK-48470
Project:
Anish Shrigondekar created SPARK-48208:
--
Summary: Skip reporting memory usage metrics if bounded memory
usage is enabled
Key: SPARK-48208
URL: https://issues.apache.org/jira/browse/SPARK-48208
[
https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843123#comment-17843123
]
Anish Shrigondekar commented on SPARK-48105:
Thanks [~huanli.wang]
Worth noting that we
Anish Shrigondekar created SPARK-48102:
--
Summary: Track time to acquire source progress metrics for
streaming triggers
Key: SPARK-48102
URL: https://issues.apache.org/jira/browse/SPARK-48102
Anish Shrigondekar created SPARK-47553:
--
Summary: Add Java support tests for transformWithState operator
Key: SPARK-47553
URL: https://issues.apache.org/jira/browse/SPARK-47553
Project: Spark
Anish Shrigondekar created SPARK-47524:
--
Summary: Add support for range scan encoder and state data source
reader
Key: SPARK-47524
URL: https://issues.apache.org/jira/browse/SPARK-47524
Project:
Anish Shrigondekar created SPARK-47512:
--
Summary: Tag operation type for RocksDB instance lock acquisition
Key: SPARK-47512
URL: https://issues.apache.org/jira/browse/SPARK-47512
Project: Spark
Anish Shrigondekar created SPARK-47490:
--
Summary: Fix RocksDB Logger deprecation warning
Key: SPARK-47490
URL: https://issues.apache.org/jira/browse/SPARK-47490
Project: Spark
Issue
Anish Shrigondekar created SPARK-47250:
--
Summary: Move additional RocksDB errors/exceptions to NERF
Key: SPARK-47250
URL: https://issues.apache.org/jira/browse/SPARK-47250
Project: Spark
Anish Shrigondekar created SPARK-47047:
--
Summary: Add support for transformWithState operator with state
data source reader
Key: SPARK-47047
URL: https://issues.apache.org/jira/browse/SPARK-47047
Anish Shrigondekar created SPARK-46979:
--
Summary: Add support for defining state encoder for key/value and
col family independently
Key: SPARK-46979
URL: https://issues.apache.org/jira/browse/SPARK-46979
[
https://issues.apache.org/jira/browse/SPARK-46815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-46815:
---
Description:
We are planning to introduce a new operator for [Spark Structured
[
https://issues.apache.org/jira/browse/SPARK-46816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810071#comment-17810071
]
Anish Shrigondekar commented on SPARK-46816:
PR here:
Anish Shrigondekar created SPARK-46816:
--
Summary: Base implementation for new operator and state API changes
Key: SPARK-46816
URL: https://issues.apache.org/jira/browse/SPARK-46816
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-45939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-45939:
---
Epic Link: SPARK-46815
> SPIP: Structured Streaming - Arbitrary State API v2
>
[
https://issues.apache.org/jira/browse/SPARK-46815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-46815:
---
Description:
We are planning to introduce a new operator for [Spark Structured
Anish Shrigondekar created SPARK-46815:
--
Summary: Structured Streaming - Arbitrary State API v2
Key: SPARK-46815
URL: https://issues.apache.org/jira/browse/SPARK-46815
Project: Spark
Anish Shrigondekar created SPARK-46712:
--
Summary: [Minor] Fix indentation for streaming aggregation operator
Key: SPARK-46712
URL: https://issues.apache.org/jira/browse/SPARK-46712
Project: Spark
Anish Shrigondekar created SPARK-46711:
--
Summary: Fix race condition during rollback with
RocksDBStateStoreProvider
Key: SPARK-46711
URL: https://issues.apache.org/jira/browse/SPARK-46711
[
https://issues.apache.org/jira/browse/SPARK-46547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801318#comment-17801318
]
Anish Shrigondekar commented on SPARK-46547:
[~kabhwan] - PTAL, thx ! -
Anish Shrigondekar created SPARK-46547:
--
Summary: Fix deadlock issue between maintenance thread and
streaming agg physical operators
Key: SPARK-46547
URL: https://issues.apache.org/jira/browse/SPARK-46547
[
https://issues.apache.org/jira/browse/SPARK-46249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793036#comment-17793036
]
Anish Shrigondekar commented on SPARK-46249:
PR here -
Anish Shrigondekar created SPARK-46249:
--
Summary: Fix state store metrics access after commit
Key: SPARK-46249
URL: https://issues.apache.org/jira/browse/SPARK-46249
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-45939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788894#comment-17788894
]
Anish Shrigondekar commented on SPARK-45939:
Added the docs here -
SPIP:
[
https://issues.apache.org/jira/browse/SPARK-45946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786629#comment-17786629
]
Anish Shrigondekar commented on SPARK-45946:
[~kabhwan] - PR here - [GitHub Pull Request
Anish Shrigondekar created SPARK-45946:
--
Summary: Fix use of deprecated FileUtils write in RocksDBSuite
Key: SPARK-45946
URL: https://issues.apache.org/jira/browse/SPARK-45946
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-45727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781093#comment-17781093
]
Anish Shrigondekar commented on SPARK-45727:
[https://github.com/apache/spark/pull/43588]
[
https://issues.apache.org/jira/browse/SPARK-45582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776442#comment-17776442
]
Anish Shrigondekar commented on SPARK-45582:
[https://github.com/apache/spark/pull/43413]
Anish Shrigondekar created SPARK-45582:
--
Summary: Streaming aggregation in complete mode should not refer
to store instance after commit
Key: SPARK-45582
URL:
[
https://issues.apache.org/jira/browse/SPARK-45539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775103#comment-17775103
]
Anish Shrigondekar edited comment on SPARK-45539 at 10/14/23 1:24 AM:
Anish Shrigondekar created SPARK-45539:
--
Summary: Add assert and log error if watermark predicate is
missing in append mode and streaming aggregation
Key: SPARK-45539
URL:
[
https://issues.apache.org/jira/browse/SPARK-45025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760677#comment-17760677
]
Anish Shrigondekar commented on SPARK-45025:
cc - [~kabhwan] - PR here -
Anish Shrigondekar created SPARK-45025:
--
Summary: Block manager write to memory store iterator should
process thread interrupt
Key: SPARK-45025
URL: https://issues.apache.org/jira/browse/SPARK-45025
[
https://issues.apache.org/jira/browse/SPARK-45002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-45002:
---
Affects Version/s: 4.0.0
(was: 3.5.1)
> Avoid uncaught
[
https://issues.apache.org/jira/browse/SPARK-45002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759829#comment-17759829
]
Anish Shrigondekar commented on SPARK-45002:
PR here:
Anish Shrigondekar created SPARK-45002:
--
Summary: Avoid uncaught exception from state store maintenance
task thread on error
Key: SPARK-45002
URL: https://issues.apache.org/jira/browse/SPARK-45002
[
https://issues.apache.org/jira/browse/SPARK-44878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756119#comment-17756119
]
Anish Shrigondekar commented on SPARK-44878:
[~kabhwan] - PR here -
Anish Shrigondekar created SPARK-44878:
--
Summary: Address LRU cache insertion failure for RocksDB with
strict limit
Key: SPARK-44878
URL: https://issues.apache.org/jira/browse/SPARK-44878
[
https://issues.apache.org/jira/browse/SPARK-44818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754729#comment-17754729
]
Anish Shrigondekar commented on SPARK-44818:
PR here -
Anish Shrigondekar created SPARK-44818:
--
Summary: Fix race for pending interrupt issued before taskThread
is initialized
Key: SPARK-44818
URL: https://issues.apache.org/jira/browse/SPARK-44818
Anish Shrigondekar created SPARK-44809:
--
Summary: Remove unused custom metrics for RocksDB state store
provider
Key: SPARK-44809
URL: https://issues.apache.org/jira/browse/SPARK-44809
Project:
[
https://issues.apache.org/jira/browse/SPARK-44504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745272#comment-17745272
]
Anish Shrigondekar commented on SPARK-44504:
Sent PR here:
[
https://issues.apache.org/jira/browse/SPARK-44504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745272#comment-17745272
]
Anish Shrigondekar edited comment on SPARK-44504 at 7/20/23 8:57 PM:
-
Anish Shrigondekar created SPARK-44504:
--
Summary: Maintenance task should clean up loaded providers on stop
error
Key: SPARK-44504
URL: https://issues.apache.org/jira/browse/SPARK-44504
Project:
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743301#comment-17743301
]
Anish Shrigondekar commented on SPARK-44438:
Sent the PR here:
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743272#comment-17743272
]
Anish Shrigondekar commented on SPARK-44438:
cc - [~kabhwan] - will send the PR soon. Thx
>
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-44438:
---
Description:
Shutdown maintenance task thread on error
Currently if we get an
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-44438:
---
Description:
Shutdown maintenance task thread on error
Currently if we get an
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-44438:
---
Description:
Shutdown maintenance task thread on error
Currently if we get an
Anish Shrigondekar created SPARK-44438:
--
Summary: Shutdown maintenance task thread on error
Key: SPARK-44438
URL: https://issues.apache.org/jira/browse/SPARK-44438
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-44288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739758#comment-17739758
]
Anish Shrigondekar edited comment on SPARK-44288 at 7/4/23 1:04 AM:
[
https://issues.apache.org/jira/browse/SPARK-44288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739758#comment-17739758
]
Anish Shrigondekar commented on SPARK-44288:
cc -
Anish Shrigondekar created SPARK-44288:
--
Summary: Pass colFamilyOptions correctly for rocksdb state store
Key: SPARK-44288
URL: https://issues.apache.org/jira/browse/SPARK-44288
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-43902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar resolved SPARK-43902.
Resolution: Won't Do
> Use keyMayExist to avoid gets if key is absent for
[
https://issues.apache.org/jira/browse/SPARK-43902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17728115#comment-17728115
]
Anish Shrigondekar commented on SPARK-43902:
PR here:
[
https://issues.apache.org/jira/browse/SPARK-43902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17728104#comment-17728104
]
Anish Shrigondekar commented on SPARK-43902:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-43902:
--
Summary: Use keyMayExist to avoid gets if key is absent for
tracking rocksdb state store updates
Key: SPARK-43902
URL:
Anish Shrigondekar created SPARK-43512:
--
Summary: Update stateStoreOperationsBenchmark to allow rocksdb jni
upgrade
Key: SPARK-43512
URL: https://issues.apache.org/jira/browse/SPARK-43512
[
https://issues.apache.org/jira/browse/SPARK-43404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720390#comment-17720390
]
Anish Shrigondekar commented on SPARK-43404:
PR here:
Anish Shrigondekar created SPARK-43404:
--
Summary: Filter current version while reusing sst files for
RocksDB state store provider while uploading to DFS to prevent id mismatch
Key: SPARK-43404
URL:
[
https://issues.apache.org/jira/browse/SPARK-43364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719103#comment-17719103
]
Anish Shrigondekar commented on SPARK-43364:
PR here:
Anish Shrigondekar created SPARK-43364:
--
Summary: Add docs for rocksdb state store memory mgmt to
structured streaming guide
Key: SPARK-43364
URL: https://issues.apache.org/jira/browse/SPARK-43364
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717407#comment-17717407
]
Anish Shrigondekar commented on SPARK-43244:
Yes correct. All the usage will be accounted
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717393#comment-17717393
]
Anish Shrigondekar commented on SPARK-43244:
[~kimahriman] - we did some investigation here
[
https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717359#comment-17717359
]
Anish Shrigondekar commented on SPARK-43311:
PR here:
[
https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-43311:
---
Description:
Today when RocksDB is used as a State Store provider, memory usage
[
https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717331#comment-17717331
]
Anish Shrigondekar commented on SPARK-43311:
[~kabhwan] - will send the PR for this soon.
Anish Shrigondekar created SPARK-43311:
--
Summary: RocksDB state store provider memory management
enhancements
Key: SPARK-43311
URL: https://issues.apache.org/jira/browse/SPARK-43311
Project:
[
https://issues.apache.org/jira/browse/SPARK-43120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711711#comment-17711711
]
Anish Shrigondekar commented on SPARK-43120:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-43120:
--
Summary: Add tracking for block cache pinned usage for RocksDB
state store
Key: SPARK-43120
URL: https://issues.apache.org/jira/browse/SPARK-43120
[
https://issues.apache.org/jira/browse/SPARK-43056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709541#comment-17709541
]
Anish Shrigondekar commented on SPARK-43056:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-43056:
--
Summary: RocksDB state store commit should continue background
work in finally only if its paused
Key: SPARK-43056
URL:
[
https://issues.apache.org/jira/browse/SPARK-42968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706698#comment-17706698
]
Anish Shrigondekar commented on SPARK-42968:
Sent PR here:
Anish Shrigondekar created SPARK-42968:
--
Summary: Add option to skip using commit coordinator in
StreamingWrite API
Key: SPARK-42968
URL: https://issues.apache.org/jira/browse/SPARK-42968
[
https://issues.apache.org/jira/browse/SPARK-42968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706600#comment-17706600
]
Anish Shrigondekar commented on SPARK-42968:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-42819:
--
Summary: Add support for setting max_write_buffer_number and
write_buffer_size for RocksDB used in streaming
Key: SPARK-42819
URL:
[
https://issues.apache.org/jira/browse/SPARK-42819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700915#comment-17700915
]
Anish Shrigondekar commented on SPARK-42819:
cc - [~kabhwan] - will send the PR soon. Thx
>
1 - 100 of 126 matches
Mail list logo