[
https://issues.apache.org/jira/browse/SPARK-47952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17848408#comment-17848408
]
Adam Binford commented on SPARK-47952:
--
We have a similar setup, currently using Livy and Zeppelin
Adam Binford created SPARK-48314:
Summary: FileStreamSource shouldn't double cache files for
availableNow
Key: SPARK-48314
URL: https://issues.apache.org/jira/browse/SPARK-48314
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-33133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17793270#comment-17793270
]
Adam Binford commented on SPARK-33133:
--
I never got around to fixing the issue creating the event
[
https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756323#comment-17756323
]
Adam Binford commented on SPARK-22876:
--
Attempting to finally add support for this:
[
https://issues.apache.org/jira/browse/SPARK-22876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford reopened SPARK-22876:
--
> spark.yarn.am.attemptFailuresValidityInterval does not work correctly
>
Adam Binford created SPARK-44639:
Summary: Add option to use Java tmp dir for RocksDB state store
Key: SPARK-44639
URL: https://issues.apache.org/jira/browse/SPARK-44639
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-43951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford resolved SPARK-43951.
--
Resolution: Fixed
> RocksDB state store can become corrupt on task retries
>
[
https://issues.apache.org/jira/browse/SPARK-43951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17728736#comment-17728736
]
Adam Binford commented on SPARK-43951:
--
Of course as soon as I finish figuring all this out I found
Adam Binford created SPARK-43951:
Summary: RocksDB state store can become corrupt on task retries
Key: SPARK-43951
URL: https://issues.apache.org/jira/browse/SPARK-43951
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-43802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-43802:
-
Description:
to_binary with hex and base64 generate invalid codegen:
Adam Binford created SPARK-43802:
Summary: unbase64 and unhex codegen are invalid with failOnError
Key: SPARK-43802
URL: https://issues.apache.org/jira/browse/SPARK-43802
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford resolved SPARK-43244.
--
Resolution: Duplicate
Will be addressed by https://issues.apache.org/jira/browse/SPARK-43311
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717394#comment-17717394
]
Adam Binford edited comment on SPARK-43244 at 4/27/23 11:08 PM:
Yeah I
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17717394#comment-17717394
]
Adam Binford commented on SPARK-43244:
--
Yeah I just saw that PR and was looking through that, I can
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715525#comment-17715525
]
Adam Binford commented on SPARK-43244:
--
Read through [https://github.com/apache/spark/pull/30344]
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-43244:
-
Description:
We noticed in one of our production stateful streaming jobs using RocksDB that
an
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-43244:
-
Description:
We noticed in one of our production stateful streaming jobs using RocksDB that
an
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-43244:
-
Description:
We noticed in one of our production stateful streaming jobs using RocksDB that
an
[
https://issues.apache.org/jira/browse/SPARK-43244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17715495#comment-17715495
]
Adam Binford commented on SPARK-43244:
--
[~kabhwan] curious your thoughts on this. Seems somewhat
Adam Binford created SPARK-43244:
Summary: RocksDB State Store can accumulate unbounded native memory
Key: SPARK-43244
URL: https://issues.apache.org/jira/browse/SPARK-43244
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-42492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-42492:
-
Description:
Doing data validation in Spark can lead to a lot of extra evaluations of
Adam Binford created SPARK-42492:
Summary: Add new function filter_value
Key: SPARK-42492
URL: https://issues.apache.org/jira/browse/SPARK-42492
Project: Spark
Issue Type: New Feature
[
https://issues.apache.org/jira/browse/SPARK-42347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17684212#comment-17684212
]
Adam Binford commented on SPARK-42347:
--
[https://github.com/apache/spark/pull/39572] is a potential
Adam Binford created SPARK-42347:
Summary: Arrow string and binary vectors only support 1 GiB
Key: SPARK-42347
URL: https://issues.apache.org/jira/browse/SPARK-42347
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-39979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17676610#comment-17676610
]
Adam Binford commented on SPARK-39979:
--
We have encountered this as well and I'm am working on a
[
https://issues.apache.org/jira/browse/SPARK-41339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-41339:
-
Summary: RocksDB state store WriteBatch doesn't clean up native memory
(was: RocksDB State
[
https://issues.apache.org/jira/browse/SPARK-41340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17641442#comment-17641442
]
Adam Binford commented on SPARK-41340:
--
Got added twice
> RocksDB State Store WriteBatch doesn't
[
https://issues.apache.org/jira/browse/SPARK-41340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford resolved SPARK-41340.
--
Resolution: Duplicate
> RocksDB State Store WriteBatch doesn't cleanup native memory
>
Adam Binford created SPARK-41340:
Summary: RocksDB State Store WriteBatch doesn't cleanup native
memory
Key: SPARK-41340
URL: https://issues.apache.org/jira/browse/SPARK-41340
Project: Spark
Adam Binford created SPARK-41339:
Summary: RocksDB State Store WriteBatch doesn't cleanup native
memory
Key: SPARK-41339
URL: https://issues.apache.org/jira/browse/SPARK-41339
Project: Spark
Adam Binford created SPARK-40775:
Summary: V2 file scans have duplicative descriptions
Key: SPARK-40775
URL: https://issues.apache.org/jira/browse/SPARK-40775
Project: Spark
Issue Type: Bug
Adam Binford created SPARK-40565:
Summary: Non-deterministic filters shouldn't get pushed to V2 file
sources
Key: SPARK-40565
URL: https://issues.apache.org/jira/browse/SPARK-40565
Project: Spark
Adam Binford created SPARK-40314:
Summary: Add inline Scala and Python bindings
Key: SPARK-40314
URL: https://issues.apache.org/jira/browse/SPARK-40314
Project: Spark
Issue Type: Bug
[
https://issues.apache.org/jira/browse/SPARK-40246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-40246:
-
Component/s: Build
(was: Spark Core)
> Logging isn't configurable via
Adam Binford created SPARK-40246:
Summary: Logging isn't configurable via log4j2 with
hadoop-provided profile
Key: SPARK-40246
URL: https://issues.apache.org/jira/browse/SPARK-40246
Project: Spark
Adam Binford created SPARK-39951:
Summary: Support columnar batches with nested fields in Parquet V2
Key: SPARK-39951
URL: https://issues.apache.org/jira/browse/SPARK-39951
Project: Spark
Adam Binford created SPARK-38640:
Summary: NPE with unpersisting memory-only RDD with RDD fetching
from shuffle service enabled
Key: SPARK-38640
URL: https://issues.apache.org/jira/browse/SPARK-38640
[
https://issues.apache.org/jira/browse/SPARK-38640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-38640:
-
Description:
If you have RDD fetching from shuffle service enabled, memory-only cached RDDs
Adam Binford created SPARK-38194:
Summary: Make Yarn memory overhead factor configurable
Key: SPARK-38194
URL: https://issues.apache.org/jira/browse/SPARK-38194
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-35570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17463386#comment-17463386
]
Adam Binford commented on SPARK-35570:
--
Just found this after making
Adam Binford created SPARK-37618:
Summary: Support cleaning up shuffle blocks from external shuffle
service
Key: SPARK-37618
URL: https://issues.apache.org/jira/browse/SPARK-37618
Project: Spark
Adam Binford created SPARK-37467:
Summary: Consolidate whole stage and non-whole stage subexpression
elimination
Key: SPARK-37467
URL: https://issues.apache.org/jira/browse/SPARK-37467
Project: Spark
Adam Binford created SPARK-37466:
Summary: Support subexpression elimination in lambda functions
Key: SPARK-37466
URL: https://issues.apache.org/jira/browse/SPARK-37466
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-37019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-37019:
-
Description:
Currently all of the higher order functions use CodegenFallback. We can improve
Adam Binford created SPARK-37019:
Summary: Add Codegen support to ArrayTransform
Key: SPARK-37019
URL: https://issues.apache.org/jira/browse/SPARK-37019
Project: Spark
Issue Type:
Adam Binford created SPARK-36918:
Summary: unionByName shouldn't consider types when comparing
structs
Key: SPARK-36918
URL: https://issues.apache.org/jira/browse/SPARK-36918
Project: Spark
Adam Binford created SPARK-35863:
Summary: Upgrade Ivy to 2.5.0
Key: SPARK-35863
URL: https://issues.apache.org/jira/browse/SPARK-35863
Project: Spark
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/SPARK-35564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358286#comment-17358286
]
Adam Binford commented on SPARK-35564:
--
Is that documented somewhere? I know Boolean expressions
[
https://issues.apache.org/jira/browse/SPARK-35564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358275#comment-17358275
]
Adam Binford edited comment on SPARK-35564 at 6/6/21, 10:50 PM:
No the
[
https://issues.apache.org/jira/browse/SPARK-35564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358275#comment-17358275
]
Adam Binford commented on SPARK-35564:
--
No the values are fine, it's the condition that cause the
[
https://issues.apache.org/jira/browse/SPARK-35564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358210#comment-17358210
]
Adam Binford commented on SPARK-35564:
--
You can construct a similar CaseWhen that could lead to a
[
https://issues.apache.org/jira/browse/SPARK-35564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358206#comment-17358206
]
Adam Binford commented on SPARK-35564:
--
Yes that was an example of "will run at least once and
[
https://issues.apache.org/jira/browse/SPARK-35564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358117#comment-17358117
]
Adam Binford commented on SPARK-35564:
--
Turns out this is already happening for certain when and
[
https://issues.apache.org/jira/browse/SPARK-35564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354637#comment-17354637
]
Adam Binford commented on SPARK-35564:
--
A 2x gain would be pretty significant to us, I don't know
Adam Binford created SPARK-35580:
Summary: Support subexpression elimination for higher order
functions
Key: SPARK-35580
URL: https://issues.apache.org/jira/browse/SPARK-35580
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-35564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17354447#comment-17354447
]
Adam Binford commented on SPARK-35564:
--
>Do you mean "Create a subexpression if an expression will
Adam Binford created SPARK-35564:
Summary: Support subexpression elimination for non-common branches
of conditional expressions
Key: SPARK-35564
URL: https://issues.apache.org/jira/browse/SPARK-35564
[
https://issues.apache.org/jira/browse/SPARK-35449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-35449:
-
Description:
[https://github.com/apache/spark/pull/30245] added support for creating
[
https://issues.apache.org/jira/browse/SPARK-35448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347976#comment-17347976
]
Adam Binford edited comment on SPARK-35448 at 5/19/21, 11:21 PM:
-
Just
[
https://issues.apache.org/jira/browse/SPARK-35448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347976#comment-17347976
]
Adam Binford commented on SPARK-35448:
--
Just spitballing after looking through some of the
[
https://issues.apache.org/jira/browse/SPARK-35256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17347974#comment-17347974
]
Adam Binford commented on SPARK-35256:
--
See https://issues.apache.org/jira/browse/SPARK-35448 for a
[
https://issues.apache.org/jira/browse/SPARK-35410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17345044#comment-17345044
]
Adam Binford commented on SPARK-35410:
--
[~viirya], thanks for adding all these great new features!
[
https://issues.apache.org/jira/browse/SPARK-35410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-35410:
-
Attachment: codegen.txt
> Unused subexpressions leftover in WholeStageCodegen subexpression
Adam Binford created SPARK-35410:
Summary: Unused subexpressions leftover in WholeStageCodegen
subexpression elimination
Key: SPARK-35410
URL: https://issues.apache.org/jira/browse/SPARK-35410
[
https://issues.apache.org/jira/browse/SPARK-35309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338966#comment-17338966
]
Adam Binford commented on SPARK-35309:
--
You want "x" to be "struct_alias"? That should be
[
https://issues.apache.org/jira/browse/SPARK-35290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338921#comment-17338921
]
Adam Binford commented on SPARK-35290:
--
Running into some issues trying to get case insensitivity
[
https://issues.apache.org/jira/browse/SPARK-35290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17338526#comment-17338526
]
Adam Binford commented on SPARK-35290:
--
I've also been playing around with rewriting some of the
[
https://issues.apache.org/jira/browse/SPARK-35290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-35290:
-
Description:
We've encountered a few weird edge cases that seem to fail the new null filling
Adam Binford created SPARK-35290:
Summary: unionByName with null filling fails for some nested
structs
Key: SPARK-35290
URL: https://issues.apache.org/jira/browse/SPARK-35290
Project: Spark
Adam Binford created SPARK-35213:
Summary: Corrupt DataFrame for certain withField patterns
Key: SPARK-35213
URL: https://issues.apache.org/jira/browse/SPARK-35213
Project: Spark
Issue Type:
Adam Binford created SPARK-35117:
Summary: UI progress bar no longer highlights in progress tasks
Key: SPARK-35117
URL: https://issues.apache.org/jira/browse/SPARK-35117
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-33133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17217656#comment-17217656
]
Adam Binford commented on SPARK-33133:
--
Now that the error on load is fixed, the next question is
[
https://issues.apache.org/jira/browse/SPARK-33133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17213832#comment-17213832
]
Adam Binford commented on SPARK-33133:
--
I created
Adam Binford created SPARK-33146:
Summary: Encountering an invalid rolling event log folder prevents
loading other applications in SHS
Key: SPARK-33146
URL: https://issues.apache.org/jira/browse/SPARK-33146
[
https://issues.apache.org/jira/browse/SPARK-33133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Binford updated SPARK-33133:
-
Description:
We have run into an issue where our history server fails to load new
Adam Binford created SPARK-33133:
Summary: History server fails when loading invalid rolling event
logs
Key: SPARK-33133
URL: https://issues.apache.org/jira/browse/SPARK-33133
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-32835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17193216#comment-17193216
]
Adam Binford commented on SPARK-32835:
--
I think I already have a working version of this, I'll try
Adam Binford created SPARK-32835:
Summary: Add withField to PySpark Column class
Key: SPARK-32835
URL: https://issues.apache.org/jira/browse/SPARK-32835
Project: Spark
Issue Type:
Adam Binford created SPARK-32589:
Summary: NoSuchElementException: None.get for
needsUnsafeRowConversion
Key: SPARK-32589
URL: https://issues.apache.org/jira/browse/SPARK-32589
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-31836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120375#comment-17120375
]
Adam Binford commented on SPARK-31836:
--
Confirmed also an issue on 2.4.5. Also I could recreate
[
https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078219#comment-17078219
]
Adam Binford edited comment on SPARK-31376 at 4/8/20, 12:40 PM:
I tried
[
https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17078219#comment-17078219
]
Adam Binford commented on SPARK-31376:
--
I tried multiple times to add myself to the dev@ mailing
[
https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077803#comment-17077803
]
Adam Binford commented on SPARK-31376:
--
You can't global sort because you can't sort an infinite
Adam Binford created SPARK-31376:
Summary: Non-global sort support for structured streaming
Key: SPARK-31376
URL: https://issues.apache.org/jira/browse/SPARK-31376
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-30860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038761#comment-17038761
]
Adam Binford commented on SPARK-30860:
--
Yeah I'll see if I can manage to update it. Not the best
Adam Binford created SPARK-30860:
Summary: Different behavior between rolling and non-rolling event
log
Key: SPARK-30860
URL: https://issues.apache.org/jira/browse/SPARK-30860
Project: Spark
86 matches
Mail list logo
