Re: Memory usage

2024-05-02 Thread TDAS
o worries, that's common with containers.If if you're still > concerned about memory here is one recent issue discussing a few > strategies how to introspect it and possible reduce it a bit: > https://github.com/apache/couchdb/discussions/4992 > > Cheers, > -Nick > > On Wed, May 1, 2024 a

Re: Memory usage

2024-05-01 Thread TDAS
correction, just realised it's a container doh... host is Debian, container (official image) is using 1.5GB > On 1 May 2024, at 22:17, TDAS wrote: > > Hey all > > I have an instance of CouchDB that's taking 1.5GB of RAM when it's not really > doing much at all. A

Memory usage

2024-05-01 Thread TDAS
Hey all I have an instance of CouchDB that's taking 1.5GB of RAM when it's not really doing much at all. About 10k docs, one replication, that's about it... any suggestions where I can start looking - version is v3.3.3, running on Debian. Thanks!

Re: [Savonet-users] Logging out when something plays

2024-03-13 Thread TDAS via Savonet-users
side of things. > On 13 Mar 2024, at 09:18, TDAS via Savonet-users > wrote: > > Anyone? > >> On 10 Mar 2024, at 21:14, TDAS via Savonet-users >> wrote: >> >> Hi >> >> Is it possible to get liquidsoap to log when a track successfully plays

Re: [Savonet-users] Logging out when something plays

2024-03-13 Thread TDAS via Savonet-users
Anyone? > On 10 Mar 2024, at 21:14, TDAS via Savonet-users > wrote: > > Hi > > Is it possible to get liquidsoap to log when a track successfully plays > (ideally start and end, but at least the latter)? > > At the moment I just have when it 'identifies' a track, a

[Savonet-users] Logging out when something plays

2024-03-10 Thread TDAS via Savonet-users
Hi Is it possible to get liquidsoap to log when a track successfully plays (ideally start and end, but at least the latter)? At the moment I just have when it 'identifies' a track, and when it 'successfully decodes' one The context is I have someone complaining that certain tracks are not

Re: [Savonet-users] Artist Separation

2024-03-04 Thread TDAS via Savonet-users
gt;> >> P.S. >> I've uploaded it to Pastebin for you to see and test: >> https://pastebin.com/kXKzPMAM >> >> I hope I've helped you. >> Juan >> >> El lun, 4 mar 2024 a las 6:40, TDAS via Savonet-users >> (> <mailto:savonet-users@li

Re: [Savonet-users] Artist Separation

2024-03-04 Thread TDAS via Savonet-users
t;Artist - > Track". Then use substring to extract just the artist name. > If you are interested in that I can help you, now if you want to do it > directly from Liquidsoap I can't help you. > > Juan > > El dom, 3 mar 2024 a las 7:41, TDAS via Savonet-users > ( <

[Savonet-users] Artist Separation

2024-03-03 Thread TDAS via Savonet-users
Hi folks Wondered whether anyone has got a working routine for handling artists and track separation? I'm trying to understand how this might work... Rick ___ Savonet-users mailing list Savonet-users@lists.sourceforge.net

Stored functions for views

2023-12-01 Thread TDAS
Is there an efficient way of using a stored function within views, without having the function redeclared for every document that’s parsed? EG if it was possible to declare a function outside the doc parser: const matcher = ({name, title}) => {/* do stuff here */} const formatter = ({info}) =>

Re: Writing an index with a linked document

2023-11-12 Thread TDAS
table/api/ddoc/search.html#get--db-_design-ddoc-_search-index > > You might mean ?include_fields=["createdDate"] ? > > B. > >> On 12 Nov 2023, at 19:12, TDAS wrote: >> >> …thinking further on this, can I return a number of fields with the index >

Re: Writing an index with a linked document

2023-11-12 Thread TDAS
…thinking further on this, can I return a number of fields with the index that aren’t searched? EG if I have a ‘doc.createdDate’, how can I just return that with the data? > On 12 Nov 2023, at 18:19, TDAS wrote: > > Basically, I was hoping that I could have the search query return

Re: Writing an index with a linked document

2023-11-12 Thread TDAS
You can't fetch another document during the indexing callbacks. > > Perhaps explain what you're trying to achieve? > > \b. > >> On 11 Nov 2023, at 23:54, TDAS wrote: >> >> getDoc doesn’t exist? Did chatgpt just make that up?! Man… >> >> Is there an

Re: Writing an index with a linked document

2023-11-11 Thread TDAS
not to be indexed at all. > > B. > >> On 11 Nov 2023, at 17:30, TDAS wrote: >> >> Hey all >> >> I have Clouseau running, and have written a search index which is working >> nicely. >> >> However when I try to link a document, the sear

Writing an index with a linked document

2023-11-11 Thread TDAS
Hey all I have Clouseau running, and have written a search index which is working nicely. However when I try to link a document, the search stops returning any results. I’ve checked it with chatgpt (so it must be right, hey!) :) Can anyone point out what I’m doing wrong? The doc.owner is

Re: Search issues

2023-09-14 Thread TDAS
That’s really odd, they’re booleans in fauxton… not sure what happened there... > On 14 Sep 2023, at 14:47, Robert Newson wrote: > > Hi, > > It's true not 'true' (boolean not string). > > B. > >> On 13 Sep 2023, at 22:54, TDAS wrote: >> >> Ok

Re: Search issues

2023-09-13 Thread TDAS
] } This is my index: function (doc) { if (doc.type === 'user' && doc.firstname && doc.lastname) { index('name', doc.firstname + ' ' + doc.lastname, { 'store': 'true' }); index('firstname', doc.firstname, { 'store': 'true' }); } } > On 13 Sep 2023, at 22:06, TDAS wro

Re: Search issues

2023-09-13 Thread TDAS
need to > choose which adapter you'd like. > > The next major release of couchdb will include an alternative Lucene indexing > system that works with Java 11 through 20 and will include the Java artifacts > necessary to run the whole stack. > > B. > >> On 13 Sep 202

Search issues

2023-09-13 Thread TDAS
Greetings I’ve set up an index in a design doc, and I’m trying to use _search after a long wait I get: { "error": "ou_est_clouseau”, "reason": "Could not connect to the Clouseau Java service at clouseau@127.0.0.1” } So, I’ve been looking into installing this but am

Find not working

2022-05-09 Thread TDAS
I’ve migrated couchdb over to a new server, using replication. Latest couchdb running on Debian 11. Everything is working except ‘find’ (using Nano, from a NodeJs app). It’s a long time since I set the search tree up, but essentially it looks like the below (if of interest) - I can’t really

[spark] branch branch-3.2 updated: [SPARK-36132][SS][SQL] Support initial state for batch mode of flatMapGroupsWithState

2021-07-20 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch branch-3.2 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.2 by this push: new 0d60cb5 [SPARK-36132][SS][SQL] Support

[spark] branch master updated: [SPARK-36132][SS][SQL] Support initial state for batch mode of flatMapGroupsWithState

2021-07-20 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new efcce23 [SPARK-36132][SS][SQL] Support initial

[spark] branch master updated: [SPARK-35800][SS] Improving GroupState testability by introducing TestGroupState

2021-06-22 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new dfd7b02 [SPARK-35800][SS] Improving GroupState

[spark] branch master updated: [SPARK-32585][SQL] Support scala enumeration in ScalaReflection

2020-10-01 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e62d247 [SPARK-32585][SQL] Support scala

[spark] branch branch-2.4 updated: [SPARK-32794][SS] Fixed rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 sources

2020-09-11 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new c82b6e4 [SPARK-32794][SS] Fixed rare

[spark] branch branch-3.0 updated: [SPARK-32794][SS] Fixed rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 sources

2020-09-09 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new e632e7c [SPARK-32794][SS] Fixed rare

[spark] branch master updated: [SPARK-32794][SS] Fixed rare corner case error in micro-batch engine with some stateful queries + no-data-batches + V1 sources

2020-09-09 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new e4237bb [SPARK-32794][SS] Fixed rare corner case

[spark] branch master updated: [SPARK-29438][SS] Use partition ID of StateStoreAwareZipPartitionsRDD for determining partition ID of state store in stream-stream join

2020-01-30 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new cbb714f [SPARK-29438][SS] Use partition ID

[spark] branch master updated: [SPARK-30609] Allow default merge command resolution to be bypassed by DSv2 tables

2020-01-22 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new d2bca8f [SPARK-30609] Allow default merge command

[spark] branch branch-2.4 updated: [SPARK-27453] Pass partitionBy as options in DataFrameWriter

2019-04-16 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new df9a506 [SPARK-27453] Pass partitionBy

[spark] branch master updated: [SPARK-27453] Pass partitionBy as options in DataFrameWriter

2019-04-16 Thread tdas
This is an automated email from the ASF dual-hosted git repository. tdas pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 26ed65f [SPARK-27453] Pass partitionBy as options

[GitHub] spark issue #23060: [SPARK-26092][SS]Use CheckpointFileManager to write the ...

2018-11-16 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/23060 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

Re: [PATCH v6] clk: qcom: Add lpass clock controller driver for SDM845

2018-10-25 Thread tdas
On 2018-10-19 16:09, Taniya Das wrote: On 10/17/2018 7:50 PM, Stephen Boyd wrote: Quoting Taniya Das (2018-10-17 05:04:10) On 10/17/2018 5:07 PM, Taniya Das wrote: Hello Stephen, On 10/12/2018 11:05 PM, Stephen Boyd wrote: Quoting Taniya Das (2018-10-09 23:12:27) On 10/10/2018 2:22 AM,

Re: [PATCH v6] clk: qcom: Add lpass clock controller driver for SDM845

2018-10-25 Thread tdas
On 2018-10-19 16:09, Taniya Das wrote: On 10/17/2018 7:50 PM, Stephen Boyd wrote: Quoting Taniya Das (2018-10-17 05:04:10) On 10/17/2018 5:07 PM, Taniya Das wrote: Hello Stephen, On 10/12/2018 11:05 PM, Stephen Boyd wrote: Quoting Taniya Das (2018-10-09 23:12:27) On 10/10/2018 2:22 AM,

spark git commit: [SPARK-25639][DOCS] Added docs for foreachBatch, python foreach and multiple watermarks

2018-10-08 Thread tdas
tch - Multiple watermark policy - The semantics of what changes are allowed to the streaming between restarts. ## How was this patch tested? No tests Closes #22627 from tdas/SPARK-25639. Authored-by: Tathagata Das Signed-off-by: Tathagata Das (cherry picked from com

spark git commit: [SPARK-25639][DOCS] Added docs for foreachBatch, python foreach and multiple watermarks

2018-10-08 Thread tdas
ple watermark policy - The semantics of what changes are allowed to the streaming between restarts. ## How was this patch tested? No tests Closes #22627 from tdas/SPARK-25639. Authored-by: Tathagata Das Signed-off-by: Tathagata Das Project: http://git-wip-us.apache.org/repos/asf/spark/repo Com

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223479786 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -2709,6 +2935,78 @@ write.stream(aggDF, "memory", outputMode = "complete"

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223479414 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,211 @@ head(sql("select * from aggregates")) -#

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-08 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22627 @holdenk yeah, i intend to backport this to 2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223456294 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -2709,6 +2935,78 @@ write.stream(aggDF, "memory", outputMode = "complete"

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-08 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22627#discussion_r223456079 --- Diff: docs/structured-streaming-programming-guide.md --- @@ -1989,22 +2026,211 @@ head(sql("select * from aggregates")) -#

[GitHub] spark issue #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch, python...

2018-10-04 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22627 @zsxwing --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22627: [SPARK-25639] [DOCS] Added docs for foreachBatch,...

2018-10-04 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/22627 [SPARK-25639] [DOCS] Added docs for foreachBatch, python foreach and multiple watermarks ## What changes were proposed in this pull request? Added - Python foreach - Scala, Java

[GitHub] spark issue #22507: [SPARK-25495][SS]FetchedData.reset should reset all fiel...

2018-09-25 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22507 LGTM. Thanks for the explanation. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark pull request #22507: [SPARK-25495][SS]FetchedData.reset should reset a...

2018-09-24 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22507#discussion_r220061053 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -874,6 +874,57 @@ abstract class

[GitHub] spark pull request #22507: [SPARK-25495][SS]FetchedData.reset should reset a...

2018-09-24 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22507#discussion_r220059919 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -874,6 +874,57 @@ abstract class

[GitHub] spark issue #22476: [SPARK-24157][SS][FOLLOWUP] Rename to spark.sql.streamin...

2018-09-19 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22476 LGTM. Please make sure to merge it to 2.4 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

spark git commit: [SPARK-25399][SS] Continuous processing state should not affect microbatch execution jobs

2018-09-11 Thread tdas
Repository: spark Updated Branches: refs/heads/branch-2.4 3a6ef8b7e -> 0dbf1450f [SPARK-25399][SS] Continuous processing state should not affect microbatch execution jobs ## What changes were proposed in this pull request? The leftover state from running a continuous processing streaming

spark git commit: [SPARK-25399][SS] Continuous processing state should not affect microbatch execution jobs

2018-09-11 Thread tdas
Repository: spark Updated Branches: refs/heads/master 97d4afaa1 -> 9f5c5b4cc [SPARK-25399][SS] Continuous processing state should not affect microbatch execution jobs ## What changes were proposed in this pull request? The leftover state from running a continuous processing streaming job

[GitHub] spark issue #22386: [SPARK-25399][SS] Continuous processing state should not...

2018-09-11 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22386 LGTM. Just one super nit. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22386: [SPARK-25399] Continuous processing state should ...

2018-09-11 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22386#discussion_r216781599 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStoreRDD.scala --- @@ -74,9 +75,14 @@ class StateStoreRDD[T: ClassTag

[GitHub] spark issue #22293: [SPARK-25288][Tests]Fix flaky Kafka transaction tests

2018-08-30 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22293 I was afraid that this would be flaky. Glad you found a solution quickly. just one comment to improve code readability

[GitHub] spark pull request #22293: [SPARK-25288][Tests]Fix flaky Kafka transaction t...

2018-08-30 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22293#discussion_r214208589 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -652,62 +654,67 @@ abstract class

[GitHub] spark issue #22042: [SPARK-25005][SS]Support non-consecutive offsets for Kaf...

2018-08-28 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22042 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark issue #22230: [SPARK-25214][SS][FOLLOWUP]Fix the issue that Kafka v2 s...

2018-08-24 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22230 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

[GitHub] spark pull request #22207: [SPARK-25214][SS]Fix the issue that Kafka v2 sour...

2018-08-24 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22207#discussion_r212706003 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala --- @@ -0,0 +1,281

[GitHub] spark pull request #22207: [SPARK-25214][SS]Fix the issue that Kafka v2 sour...

2018-08-24 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22207#discussion_r212706908 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala --- @@ -0,0 +1,281

[GitHub] spark pull request #22207: [SPARK-25214][SS]Fix the issue that Kafka v2 sour...

2018-08-24 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22207#discussion_r212706859 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala --- @@ -0,0 +1,281

[GitHub] spark pull request #22207: [SPARK-25214][SS]Fix the issue that Kafka v2 sour...

2018-08-24 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22207#discussion_r212526645 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala --- @@ -0,0 +1,281

[GitHub] spark pull request #22207: [SPARK-25214][SS]Fix the issue that Kafka v2 sour...

2018-08-23 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22207#discussion_r212526373 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala --- @@ -0,0 +1,281

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-23 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r212507190 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -160,6 +160,23 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-23 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r212522664 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -161,6 +161,22 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-23 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r212521083 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -239,56 +335,74 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-23 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r212522432 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -161,6 +161,22 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-23 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r212504622 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReadSupport.scala --- @@ -331,6 +331,7 @@ private[kafka010

spark git commit: [SPARK-25204][SS] Fix race in rate source test.

2018-08-23 Thread tdas
Repository: spark Updated Branches: refs/heads/master a9aacdf1c -> 8ed044928 [SPARK-25204][SS] Fix race in rate source test. ## What changes were proposed in this pull request? Fix a race in the rate source tests. We need a better way of testing restart behavior. ## How was this patch

[GitHub] spark issue #22191: [SPARK-25204][SS] Fix race in rate source test.

2018-08-23 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22191 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

spark git commit: [SPARK-25184][SS] Fixed race condition in StreamExecution that caused flaky test in FlatMapGroupsWithState

2018-08-22 Thread tdas
n is also avoided. ## How was this patch tested? Ran locally many times. Closes #22182 from tdas/SPARK-25184. Authored-by: Tathagata Das Signed-off-by: Tathagata Das Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/31063249 Tr

[GitHub] spark pull request #22182: [SPARK-25184][SS] Fixed race condition in StreamE...

2018-08-21 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/22182 [SPARK-25184][SS] Fixed race condition in StreamExecution that caused flaky test in FlatMapGroupsWithState ## What changes were proposed in this pull request? The race condition that caused

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211805733 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -597,6 +614,254 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211801632 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -346,11 +437,40 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211802112 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -250,33 +341,39 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211801676 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -250,33 +341,39 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211801968 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -250,33 +341,39 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211795985 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -80,6 +90,72 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211805275 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -597,6 +614,254 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211801549 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -250,33 +341,39 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211804454 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -597,6 +614,254 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211802489 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -288,7 +385,7 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211801254 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaDataConsumer.scala --- @@ -250,33 +341,39 @@ private[kafka010] case class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211804879 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -597,6 +614,254 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211803267 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchReader.scala --- @@ -337,6 +338,7 @@ private[kafka010] case

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211805821 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -597,6 +614,254 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211805409 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -597,6 +614,254 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211804704 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -160,6 +160,23 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211803763 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala --- @@ -160,6 +160,23 @@ abstract class

[GitHub] spark pull request #22042: [SPARK-25005][SS]Support non-consecutive offsets ...

2018-08-21 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/22042#discussion_r211805993 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaTestUtils.scala --- @@ -327,6 +332,14 @@ class KafkaTestUtils

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21469 I did. Fixed the import --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews

[GitHub] spark issue #22175: [MINOR] Added import to fix compilation

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22175 Merged as compilation passed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

spark git commit: [MINOR] Added import to fix compilation

2018-08-21 Thread tdas
[error] one error found [error] Compile failed at Aug 21, 2018 4:04:35 PM [12.827s] ``` ## How was this patch tested? It compiles! Closes #22175 from tdas/fix-build. Authored-by: Tathagata Das Signed-off-by: Tathagata Das Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: htt

[GitHub] spark issue #22175: [MINOR] Added import to fix compilation

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/22175 jenkins retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail

[GitHub] spark pull request #22175: [MINOR] Added import to fix compilation

2018-08-21 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/22175 [MINOR] Added import to fix compilation ## What changes were proposed in this pull request? Two back to PRs implicitly conflicted by one PR removing an existing import that the other PR

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21469 Unfortunately this PR broke the master build. Looks like some import that probably got removed in the other PR I merged, which didnt create any direct conflict

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21469 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

spark git commit: [SPARK-24441][SS] Expose total estimated size of states in HDFSBackedStateStoreProvider

2018-08-21 Thread tdas
Repository: spark Updated Branches: refs/heads/master ac0174e55 -> 42035a4fe [SPARK-24441][SS] Expose total estimated size of states in HDFSBackedStateStoreProvider ## What changes were proposed in this pull request? This patch exposes the estimation of size of cache (loadedMaps) in

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21469 LGTM. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h

spark git commit: [SPARK-24763][SS] Remove redundant key data from value in streaming aggregation

2018-08-21 Thread tdas
Repository: spark Updated Branches: refs/heads/master 72ecfd095 -> 6c5cb8585 [SPARK-24763][SS] Remove redundant key data from value in streaming aggregation ## What changes were proposed in this pull request? This patch proposes a new flag option for stateful aggregation: remove redundant

[GitHub] spark issue #21469: [SPARK-24441][SS] Expose total estimated size of states ...

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21469 @HeartSaVioR I think I agree with a second approach that you suggested. So `memoryUsedBytes` => `size for total memory usage of loaded versions` and `customMetric` => `size for memory

[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21733 Good point. That can be minor Pr. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

[GitHub] spark issue #21733: [SPARK-24763][SS] Remove redundant key data from value i...

2018-08-21 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/21733 LGTM. Will merge when tests pass. :) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e

  1   2   3   4   5   6   7   8   9   10   >