Anish Shrigondekar created SPARK-48208:
--
Summary: Skip reporting memory usage metrics if bounded memory
usage is enabled
Key: SPARK-48208
URL: https://issues.apache.org/jira/browse/SPARK-48208
[
https://issues.apache.org/jira/browse/SPARK-48105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17843123#comment-17843123
]
Anish Shrigondekar commented on SPARK-48105:
Thanks [~huanli.wang]
Worth noting that we
Anish Shrigondekar created SPARK-48102:
--
Summary: Track time to acquire source progress metrics for
streaming triggers
Key: SPARK-48102
URL: https://issues.apache.org/jira/browse/SPARK-48102
Anish Shrigondekar created SPARK-47553:
--
Summary: Add Java support tests for transformWithState operator
Key: SPARK-47553
URL: https://issues.apache.org/jira/browse/SPARK-47553
Project: Spark
Anish Shrigondekar created SPARK-47524:
--
Summary: Add support for range scan encoder and state data source
reader
Key: SPARK-47524
URL: https://issues.apache.org/jira/browse/SPARK-47524
Project:
Anish Shrigondekar created SPARK-47512:
--
Summary: Tag operation type for RocksDB instance lock acquisition
Key: SPARK-47512
URL: https://issues.apache.org/jira/browse/SPARK-47512
Project: Spark
Anish Shrigondekar created SPARK-47490:
--
Summary: Fix RocksDB Logger deprecation warning
Key: SPARK-47490
URL: https://issues.apache.org/jira/browse/SPARK-47490
Project: Spark
Issue
Anish Shrigondekar created SPARK-47250:
--
Summary: Move additional RocksDB errors/exceptions to NERF
Key: SPARK-47250
URL: https://issues.apache.org/jira/browse/SPARK-47250
Project: Spark
Anish Shrigondekar created SPARK-47047:
--
Summary: Add support for transformWithState operator with state
data source reader
Key: SPARK-47047
URL: https://issues.apache.org/jira/browse/SPARK-47047
Anish Shrigondekar created SPARK-46979:
--
Summary: Add support for defining state encoder for key/value and
col family independently
Key: SPARK-46979
URL: https://issues.apache.org/jira/browse/SPARK-46979
[
https://issues.apache.org/jira/browse/SPARK-46815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-46815:
---
Description:
We are planning to introduce a new operator for [Spark Structured
[
https://issues.apache.org/jira/browse/SPARK-46816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810071#comment-17810071
]
Anish Shrigondekar commented on SPARK-46816:
PR here:
Anish Shrigondekar created SPARK-46816:
--
Summary: Base implementation for new operator and state API changes
Key: SPARK-46816
URL: https://issues.apache.org/jira/browse/SPARK-46816
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-45939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-45939:
---
Epic Link: SPARK-46815
> SPIP: Structured Streaming - Arbitrary State API v2
>
[
https://issues.apache.org/jira/browse/SPARK-46815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-46815:
---
Description:
We are planning to introduce a new operator for [Spark Structured
Anish Shrigondekar created SPARK-46815:
--
Summary: Structured Streaming - Arbitrary State API v2
Key: SPARK-46815
URL: https://issues.apache.org/jira/browse/SPARK-46815
Project: Spark
Anish Shrigondekar created SPARK-46712:
--
Summary: [Minor] Fix indentation for streaming aggregation operator
Key: SPARK-46712
URL: https://issues.apache.org/jira/browse/SPARK-46712
Project: Spark
Anish Shrigondekar created SPARK-46711:
--
Summary: Fix race condition during rollback with
RocksDBStateStoreProvider
Key: SPARK-46711
URL: https://issues.apache.org/jira/browse/SPARK-46711
[
https://issues.apache.org/jira/browse/SPARK-46547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17801318#comment-17801318
]
Anish Shrigondekar commented on SPARK-46547:
[~kabhwan] - PTAL, thx ! -
Anish Shrigondekar created SPARK-46547:
--
Summary: Fix deadlock issue between maintenance thread and
streaming agg physical operators
Key: SPARK-46547
URL: https://issues.apache.org/jira/browse/SPARK-46547
[
https://issues.apache.org/jira/browse/SPARK-46249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793036#comment-17793036
]
Anish Shrigondekar commented on SPARK-46249:
PR here -
Anish Shrigondekar created SPARK-46249:
--
Summary: Fix state store metrics access after commit
Key: SPARK-46249
URL: https://issues.apache.org/jira/browse/SPARK-46249
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-45939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17788894#comment-17788894
]
Anish Shrigondekar commented on SPARK-45939:
Added the docs here -
SPIP:
[
https://issues.apache.org/jira/browse/SPARK-45946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786629#comment-17786629
]
Anish Shrigondekar commented on SPARK-45946:
[~kabhwan] - PR here - [GitHub Pull Request
Anish Shrigondekar created SPARK-45946:
--
Summary: Fix use of deprecated FileUtils write in RocksDBSuite
Key: SPARK-45946
URL: https://issues.apache.org/jira/browse/SPARK-45946
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-45727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17781093#comment-17781093
]
Anish Shrigondekar commented on SPARK-45727:
[https://github.com/apache/spark/pull/43588]
[
https://issues.apache.org/jira/browse/SPARK-45582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776442#comment-17776442
]
Anish Shrigondekar commented on SPARK-45582:
[https://github.com/apache/spark/pull/43413]
Anish Shrigondekar created SPARK-45582:
--
Summary: Streaming aggregation in complete mode should not refer
to store instance after commit
Key: SPARK-45582
URL:
[
https://issues.apache.org/jira/browse/SPARK-45539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775103#comment-17775103
]
Anish Shrigondekar edited comment on SPARK-45539 at 10/14/23 1:24 AM:
Anish Shrigondekar created SPARK-45539:
--
Summary: Add assert and log error if watermark predicate is
missing in append mode and streaming aggregation
Key: SPARK-45539
URL:
[
https://issues.apache.org/jira/browse/SPARK-45025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17760677#comment-17760677
]
Anish Shrigondekar commented on SPARK-45025:
cc - [~kabhwan] - PR here -
Anish Shrigondekar created SPARK-45025:
--
Summary: Block manager write to memory store iterator should
process thread interrupt
Key: SPARK-45025
URL: https://issues.apache.org/jira/browse/SPARK-45025
[
https://issues.apache.org/jira/browse/SPARK-45002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-45002:
---
Affects Version/s: 4.0.0
(was: 3.5.1)
> Avoid uncaught
[
https://issues.apache.org/jira/browse/SPARK-45002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17759829#comment-17759829
]
Anish Shrigondekar commented on SPARK-45002:
PR here:
Anish Shrigondekar created SPARK-45002:
--
Summary: Avoid uncaught exception from state store maintenance
task thread on error
Key: SPARK-45002
URL: https://issues.apache.org/jira/browse/SPARK-45002
[
https://issues.apache.org/jira/browse/SPARK-44878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756119#comment-17756119
]
Anish Shrigondekar commented on SPARK-44878:
[~kabhwan] - PR here -
Anish Shrigondekar created SPARK-44878:
--
Summary: Address LRU cache insertion failure for RocksDB with
strict limit
Key: SPARK-44878
URL: https://issues.apache.org/jira/browse/SPARK-44878
[
https://issues.apache.org/jira/browse/SPARK-44818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17754729#comment-17754729
]
Anish Shrigondekar commented on SPARK-44818:
PR here -
Anish Shrigondekar created SPARK-44818:
--
Summary: Fix race for pending interrupt issued before taskThread
is initialized
Key: SPARK-44818
URL: https://issues.apache.org/jira/browse/SPARK-44818
Anish Shrigondekar created SPARK-44809:
--
Summary: Remove unused custom metrics for RocksDB state store
provider
Key: SPARK-44809
URL: https://issues.apache.org/jira/browse/SPARK-44809
Project:
[
https://issues.apache.org/jira/browse/SPARK-44504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745272#comment-17745272
]
Anish Shrigondekar commented on SPARK-44504:
Sent PR here:
[
https://issues.apache.org/jira/browse/SPARK-44504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17745272#comment-17745272
]
Anish Shrigondekar edited comment on SPARK-44504 at 7/20/23 8:57 PM:
-
Anish Shrigondekar created SPARK-44504:
--
Summary: Maintenance task should clean up loaded providers on stop
error
Key: SPARK-44504
URL: https://issues.apache.org/jira/browse/SPARK-44504
Project:
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743301#comment-17743301
]
Anish Shrigondekar commented on SPARK-44438:
Sent the PR here:
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17743272#comment-17743272
]
Anish Shrigondekar commented on SPARK-44438:
cc - [~kabhwan] - will send the PR soon. Thx
>
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-44438:
---
Description:
Shutdown maintenance task thread on error
Currently if we get an
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-44438:
---
Description:
Shutdown maintenance task thread on error
Currently if we get an
[
https://issues.apache.org/jira/browse/SPARK-44438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-44438:
---
Description:
Shutdown maintenance task thread on error
Currently if we get an
Anish Shrigondekar created SPARK-44438:
--
Summary: Shutdown maintenance task thread on error
Key: SPARK-44438
URL: https://issues.apache.org/jira/browse/SPARK-44438
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-44288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739758#comment-17739758
]
Anish Shrigondekar edited comment on SPARK-44288 at 7/4/23 1:04 AM:
[
https://issues.apache.org/jira/browse/SPARK-44288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739758#comment-17739758
]
Anish Shrigondekar commented on SPARK-44288:
cc -
Anish Shrigondekar created SPARK-44288:
--
Summary: Pass colFamilyOptions correctly for rocksdb state store
Key: SPARK-44288
URL: https://issues.apache.org/jira/browse/SPARK-44288
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-43902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar resolved SPARK-43902.
Resolution: Won't Do
> Use keyMayExist to avoid gets if key is absent for
[
https://issues.apache.org/jira/browse/SPARK-43902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17728115#comment-17728115
]
Anish Shrigondekar commented on SPARK-43902:
PR here:
[
https://issues.apache.org/jira/browse/SPARK-43902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17728104#comment-17728104
]
Anish Shrigondekar commented on SPARK-43902:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-43902:
--
Summary: Use keyMayExist to avoid gets if key is absent for
tracking rocksdb state store updates
Key: SPARK-43902
URL:
Anish Shrigondekar created SPARK-43512:
--
Summary: Update stateStoreOperationsBenchmark to allow rocksdb jni
upgrade
Key: SPARK-43512
URL: https://issues.apache.org/jira/browse/SPARK-43512
[
https://issues.apache.org/jira/browse/SPARK-43404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17720390#comment-17720390
]
Anish Shrigondekar commented on SPARK-43404:
PR here:
Anish Shrigondekar created SPARK-43404:
--
Summary: Filter current version while reusing sst files for
RocksDB state store provider while uploading to DFS to prevent id mismatch
Key: SPARK-43404
URL:
[
https://issues.apache.org/jira/browse/SPARK-43364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17719103#comment-17719103
]
Anish Shrigondekar commented on SPARK-43364:
PR here:
Anish Shrigondekar created SPARK-43364:
--
Summary: Add docs for rocksdb state store memory mgmt to
structured streaming guide
Key: SPARK-43364
URL: https://issues.apache.org/jira/browse/SPARK-43364
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717407#comment-17717407
]
Anish Shrigondekar commented on SPARK-43244:
Yes correct. All the usage will be accounted
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717393#comment-17717393
]
Anish Shrigondekar commented on SPARK-43244:
[~kimahriman] - we did some investigation here
[
https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717359#comment-17717359
]
Anish Shrigondekar commented on SPARK-43311:
PR here:
[
https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anish Shrigondekar updated SPARK-43311:
---
Description:
Today when RocksDB is used as a State Store provider, memory usage
[
https://issues.apache.org/jira/browse/SPARK-43311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717331#comment-17717331
]
Anish Shrigondekar commented on SPARK-43311:
[~kabhwan] - will send the PR for this soon.
Anish Shrigondekar created SPARK-43311:
--
Summary: RocksDB state store provider memory management
enhancements
Key: SPARK-43311
URL: https://issues.apache.org/jira/browse/SPARK-43311
Project:
[
https://issues.apache.org/jira/browse/SPARK-43120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17711711#comment-17711711
]
Anish Shrigondekar commented on SPARK-43120:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-43120:
--
Summary: Add tracking for block cache pinned usage for RocksDB
state store
Key: SPARK-43120
URL: https://issues.apache.org/jira/browse/SPARK-43120
[
https://issues.apache.org/jira/browse/SPARK-43056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17709541#comment-17709541
]
Anish Shrigondekar commented on SPARK-43056:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-43056:
--
Summary: RocksDB state store commit should continue background
work in finally only if its paused
Key: SPARK-43056
URL:
[
https://issues.apache.org/jira/browse/SPARK-42968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706698#comment-17706698
]
Anish Shrigondekar commented on SPARK-42968:
Sent PR here:
Anish Shrigondekar created SPARK-42968:
--
Summary: Add option to skip using commit coordinator in
StreamingWrite API
Key: SPARK-42968
URL: https://issues.apache.org/jira/browse/SPARK-42968
[
https://issues.apache.org/jira/browse/SPARK-42968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17706600#comment-17706600
]
Anish Shrigondekar commented on SPARK-42968:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-42819:
--
Summary: Add support for setting max_write_buffer_number and
write_buffer_size for RocksDB used in streaming
Key: SPARK-42819
URL:
[
https://issues.apache.org/jira/browse/SPARK-42819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700915#comment-17700915
]
Anish Shrigondekar commented on SPARK-42819:
cc - [~kabhwan] - will send the PR soon. Thx
>
Anish Shrigondekar created SPARK-42792:
--
Summary: Add support to track FLUSH_WRITE_BYTES for RocksDB state
store for streaming
Key: SPARK-42792
URL: https://issues.apache.org/jira/browse/SPARK-42792
[
https://issues.apache.org/jira/browse/SPARK-42792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17700387#comment-17700387
]
Anish Shrigondekar commented on SPARK-42792:
Will send the PR soon - cc - [~kabhwan]
> Add
[
https://issues.apache.org/jira/browse/SPARK-42676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696672#comment-17696672
]
Anish Shrigondekar commented on SPARK-42676:
Will send the PR soon
cc - [~kabhwan]
>
Anish Shrigondekar created SPARK-42676:
--
Summary: Write temp checkpoints for streaming queries to local
filesystem even if default FS is set differently
Key: SPARK-42676
URL:
Anish Shrigondekar created SPARK-42668:
--
Summary: Catch exception while trying to close compressed stream
in HDFSStateStoreProvider abort
Key: SPARK-42668
URL:
[
https://issues.apache.org/jira/browse/SPARK-42668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17696368#comment-17696368
]
Anish Shrigondekar commented on SPARK-42668:
Will send out the fix soon
cc - [~kabhwan]
[
https://issues.apache.org/jira/browse/SPARK-42567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693355#comment-17693355
]
Anish Shrigondekar commented on SPARK-42567:
Sent the PR here:
Anish Shrigondekar created SPARK-42567:
--
Summary: Track state store provider load time and log warning if
it exceeds a threshold
Key: SPARK-42567
URL: https://issues.apache.org/jira/browse/SPARK-42567
[
https://issues.apache.org/jira/browse/SPARK-42567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17693337#comment-17693337
]
Anish Shrigondekar commented on SPARK-42567:
I will send the fix out soon
cc -
[
https://issues.apache.org/jira/browse/SPARK-41996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17675764#comment-17675764
]
Anish Shrigondekar commented on SPARK-41996:
Have the fix and will send the PR out soon
Anish Shrigondekar created SPARK-41996:
--
Summary: KafkaMicroBatchV2SourceSuite failed for topic partitions
unavailable test due to kafka operations taking longer
Key: SPARK-41996
URL:
[
https://issues.apache.org/jira/browse/SPARK-39781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17566965#comment-17566965
]
Anish Shrigondekar commented on SPARK-39781:
Sent change here:
[
https://issues.apache.org/jira/browse/SPARK-39781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17566940#comment-17566940
]
Anish Shrigondekar commented on SPARK-39781:
Working on the change and will send out for
Anish Shrigondekar created SPARK-39781:
--
Summary: Add support for configuring max_open_files through
RocksDB state store provider
Key: SPARK-39781
URL: https://issues.apache.org/jira/browse/SPARK-39781
[
https://issues.apache.org/jira/browse/SPARK-39264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541194#comment-17541194
]
Anish Shrigondekar commented on SPARK-39264:
Based on Alex's comments post merge. Will send
Anish Shrigondekar created SPARK-39264:
--
Summary: Add explicit type checks and casting for awaitOffset fix
Key: SPARK-39264
URL: https://issues.apache.org/jira/browse/SPARK-39264
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-39242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540252#comment-17540252
]
Anish Shrigondekar commented on SPARK-39242:
PR for the change submitted here:
[
https://issues.apache.org/jira/browse/SPARK-39242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17540231#comment-17540231
]
Anish Shrigondekar commented on SPARK-39242:
I have found the root cause for the issue and
Anish Shrigondekar created SPARK-39242:
--
Summary: AwaitOffset does not wait correctly for atleast expected
offset and RateStreamProvider test is flaky
Key: SPARK-39242
URL:
[
https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518344#comment-17518344
]
Anish Shrigondekar commented on SPARK-38809:
Working on this PR and will send the change out
[
https://issues.apache.org/jira/browse/SPARK-38809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17518344#comment-17518344
]
Anish Shrigondekar edited comment on SPARK-38809 at 4/6/22 6:38 PM:
Anish Shrigondekar created SPARK-38809:
--
Summary: Implement option to skip null values in symmetric hash
impl of stream-stream joins
Key: SPARK-38809
URL: https://issues.apache.org/jira/browse/SPARK-38809
[
https://issues.apache.org/jira/browse/SPARK-38787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517220#comment-17517220
]
Anish Shrigondekar commented on SPARK-38787:
Working on the PR for this fix. Will send it
[
https://issues.apache.org/jira/browse/SPARK-38787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17517217#comment-17517217
]
Anish Shrigondekar commented on SPARK-38787:
CC - [~kabhwan]
> Possible correctness issue
1 - 100 of 102 matches
Mail list logo
