[GitHub] [druid] maytasm commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
maytasm commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764440034 I think you can have `build_run_cluster.sh` and `stop_cluster.sh` run your script instead. In `build_run_cluster.sh` and `stop_cluster.sh`, you can check the

[GitHub] [druid] zhangyue19921010 commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
zhangyue19921010 commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764429538 Hi @jihoonson and @himanshug . I did some test on my ec2. https://user-images.githubusercontent.com/69956021/105312969-c3354f80-5bf8-11eb-98c0-cdc0f8e8d1e9.png;> As

[GitHub] [druid] maytasm commented on a change in pull request #10740: Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

2021-01-20 Thread GitBox
maytasm commented on a change in pull request #10740: URL: https://github.com/apache/druid/pull/10740#discussion_r557006534 ## File path: server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java ## @@ -617,6 +625,21 @@ public Object call()

[GitHub] [druid] maytasm commented on a change in pull request #10740: Fix byte calculation for maxBytesInMemory to take into account of Sink/Hydrant Object overhead

2021-01-20 Thread GitBox
maytasm commented on a change in pull request #10740: URL: https://github.com/apache/druid/pull/10740#discussion_r557006534 ## File path: server/src/main/java/org/apache/druid/segment/realtime/appenderator/AppenderatorImpl.java ## @@ -617,6 +625,21 @@ public Object call()

[GitHub] [druid] clintropolis commented on issue #10780: Deprecate support for Apache ZooKeeper 3.4.x

2021-01-20 Thread GitBox
clintropolis commented on issue #10780: URL: https://github.com/apache/druid/issues/10780#issuecomment-764202660 >Should we go to 3.5.x or 3.6.x? https://downloads.apache.org/zookeeper/stable/ shows 3.5.9 as stable version, while 3.6.2 is 'current'

[GitHub] [druid] gianm commented on issue #10780: Deprecate support for Apache ZooKeeper 3.4.x

2021-01-20 Thread GitBox
gianm commented on issue #10780: URL: https://github.com/apache/druid/issues/10780#issuecomment-764200236 Should we go to 3.5.x or 3.6.x? Given that both are supported, I'm leaning towards 3.5.x, because it's more likely to be battle-tested / stable. That's mostly what we want from

[GitHub] [druid] jihoonson commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
jihoonson commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764186954 Oops, I think you are right. It has been a while since I ran the test. Then, can we automate the creation and cleanup of the cluster in integration tests instead of relying on the

[GitHub] [druid] himanshug commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
himanshug commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764173018 > because it spawns them automatically. actually, k8s/druid cluster is not created by the integration test... you must have manually run following scripts... ```

[GitHub] [druid] zhangyue19921010 commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
zhangyue19921010 commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764149785 Sure I will investigate this problem. And will respone later. This is an automated message from the Apache

[GitHub] [druid] maytasm commented on a change in pull request #10742: Granularity interval materialization

2021-01-20 Thread GitBox
maytasm commented on a change in pull request #10742: URL: https://github.com/apache/druid/pull/10742#discussion_r561439141 ## File path: server/src/test/java/org/apache/druid/segment/indexing/granularity/IntervalsByGranularityTest.java ## @@ -76,6 +79,21 @@ public void

[GitHub] [druid] jihoonson commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
jihoonson commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764143172 I can test the script later, but I believe it would work (I didn't know that exists). Maybe we should just call the script somewhere inside of the integration test framework

[GitHub] [druid] jihoonson commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
jihoonson commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764139187 Oh, I didn't run on travis or didn't run the travis script. I just ran the integration test manually and expected to kill all those containers after all tests passed because it

[GitHub] [druid] himanshug commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
himanshug commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764137426 CC @zhangyue19921010 as He added it and might know something. This is an automated message from the Apache Git

[GitHub] [druid] himanshug commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
himanshug commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764133398 that script actually is supposed to destroy the whole k8s cluster setup by minikube .. can you manually run the script and see if it really works... maybe the script itself is

[GitHub] [druid] jihoonson commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
jihoonson commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764115881 Hmm, I'm not sure why the script didn't kill the containers. I ran them in my local machine. This is an automated

[GitHub] [druid] himanshug commented on issue #10750: k8s integration tests don't stop containers after tests

2021-01-20 Thread GitBox
himanshug commented on issue #10750: URL: https://github.com/apache/druid/issues/10750#issuecomment-764092185 I thought https://github.com/apache/druid/blob/master/.travis.yml#L575 does the trick, how did you get above from travis ?

[GitHub] [druid] himanshug commented on issue #10783: pac4j using OAuth authentication

2021-01-20 Thread GitBox
himanshug commented on issue #10783: URL: https://github.com/apache/druid/issues/10783#issuecomment-764073397 For okta, discoveryURI looks something like what is described in

[GitHub] [druid] jihoonson commented on issue #10780: Deprecate support for Apache ZooKeeper 3.4.x

2021-01-20 Thread GitBox
jihoonson commented on issue #10780: URL: https://github.com/apache/druid/issues/10780#issuecomment-763962124 It makes sense to me to deprecate 3.4 in 0.21.0. This is an automated message from the Apache Git Service. To

[GitHub] [druid] jihoonson commented on issue #10784: ParseExceptions from hasNext not caught and handled properly

2021-01-20 Thread GitBox
jihoonson commented on issue #10784: URL: https://github.com/apache/druid/issues/10784#issuecomment-763909460 Yeah, I'm curious if this problem happens only with legacy parsers. If so, an easy workaround would be using inputFormat.

[GitHub] [druid] gianm commented on issue #10784: ParseExceptions from hasNext not caught and handled properly

2021-01-20 Thread GitBox
gianm commented on issue #10784: URL: https://github.com/apache/druid/issues/10784#issuecomment-763907331 I'm not sure since I haven't done a full analysis. But in this case, the task used a legacy `parser`, not an `inputFormat`.

[GitHub] [druid] jihoonson commented on issue #10784: ParseExceptions from hasNext not caught and handled properly

2021-01-20 Thread GitBox
jihoonson commented on issue #10784: URL: https://github.com/apache/druid/issues/10784#issuecomment-763906180 Does this happen with `InputFormat` as well? [`IntermediateRowParsingReader` should handle exceptions thrown in hasNext() via

[GitHub] [druid-website-src] druid-matt opened a new pull request #201: add autonomic, mobiauto, apollo graphql

2021-01-20 Thread GitBox
druid-matt opened a new pull request #201: URL: https://github.com/apache/druid-website-src/pull/201 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [druid] gianm opened a new issue #10784: ParseExceptions from hasNext not caught and handled properly

2021-01-20 Thread GitBox
gianm opened a new issue #10784: URL: https://github.com/apache/druid/issues/10784 FilteringCloseableInputRowIterator is the chokepoint where all calls to `parseExceptionHandler.handle` get made when they come from issues in parsing the original input. It has logic to catch and handle

[GitHub] [druid] gianm commented on issue #10780: Deprecate support for Apache ZooKeeper 3.4.x

2021-01-20 Thread GitBox
gianm commented on issue #10780: URL: https://github.com/apache/druid/issues/10780#issuecomment-763828684 In that case, I suggest add the tests for ZK 3.5 now, and we announce in the very next release that people should upgrade to ZK 3.5 when they can, because a future release will ship

[GitHub] [druid] gianm commented on pull request #10767: Vectorized theta sketch aggregator + rework of VectorColumnProcessorFactory.

2021-01-20 Thread GitBox
gianm commented on pull request #10767: URL: https://github.com/apache/druid/pull/10767#issuecomment-763818055 Pushed up a new patch. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [druid] gianm commented on a change in pull request #10767: Vectorized theta sketch aggregator + rework of VectorColumnProcessorFactory.

2021-01-20 Thread GitBox
gianm commented on a change in pull request #10767: URL: https://github.com/apache/druid/pull/10767#discussion_r561153188 ## File path: processing/src/main/java/org/apache/druid/segment/VectorColumnProcessorFactory.java ## @@ -22,34 +22,59 @@ import

[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

2021-01-20 Thread GitBox
capistrant commented on a change in pull request #10676: URL: https://github.com/apache/druid/pull/10676#discussion_r561152125 ## File path: indexing-service/src/test/java/org/apache/druid/indexing/common/task/TaskSerdeTest.java ## @@ -268,7 +268,8 @@ public void

[GitHub] [druid] gianm commented on a change in pull request #10605: bitwise math function expressions

2021-01-20 Thread GitBox
gianm commented on a change in pull request #10605: URL: https://github.com/apache/druid/pull/10605#discussion_r561147445 ## File path: docs/misc/math-expr.md ## @@ -115,53 +115,53 @@ See javadoc of java.lang.Math for detailed explanation for each function.

[GitHub] [druid] xvrl commented on issue #10780: Deprecate support for Apache ZooKeeper 3.4.x

2021-01-20 Thread GitBox
xvrl commented on issue #10780: URL: https://github.com/apache/druid/issues/10780#issuecomment-763810225 @gianm yes, the migration path is to first upgrade ZK 3.5 before upgrade Druid to whichever version will no longer support ZK 3.4.x. We have been running Druid 0.19 successfully with

[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

2021-01-20 Thread GitBox
capistrant commented on a change in pull request #10676: URL: https://github.com/apache/druid/pull/10676#discussion_r556922336 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AppenderatorDriverRealtimeIndexTask.java ## @@ -584,7 +584,8 @@

[GitHub] [druid] capistrant commented on a change in pull request #10676: Allow client to configure batch ingestion task to wait to complete until segments are confirmed to be available by other

2021-01-20 Thread GitBox
capistrant commented on a change in pull request #10676: URL: https://github.com/apache/druid/pull/10676#discussion_r556922336 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/common/task/AppenderatorDriverRealtimeIndexTask.java ## @@ -584,7 +584,8 @@

[druid] branch master updated: Retain order of AND, OR filter children. (#10758)

2021-01-20 Thread suneet
This is an automated email from the ASF dual-hosted git repository. suneet pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/druid.git The following commit(s) were added to refs/heads/master by this push: new 8b808c4 Retain order of AND, OR filter

[GitHub] [druid] suneet-s merged pull request #10758: Retain order of AND, OR filter children.

2021-01-20 Thread GitBox
suneet-s merged pull request #10758: URL: https://github.com/apache/druid/pull/10758 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [druid] trinitry opened a new issue #10783: pac4j using OAuth authentication

2021-01-20 Thread GitBox
trinitry opened a new issue #10783: URL: https://github.com/apache/druid/issues/10783 Dear ALL, Enabling the pac4j module and using the following settings in runtime.properties of the router druid.auth.authenticatorChain=["pac4j"] druid.auth.authenticator.pac4j.type=pac4j

[GitHub] [druid] clintropolis commented on a change in pull request #10605: bitwise math function expressions

2021-01-20 Thread GitBox
clintropolis commented on a change in pull request #10605: URL: https://github.com/apache/druid/pull/10605#discussion_r560935104 ## File path: core/src/main/antlr4/org/apache/druid/math/expr/antlr/Expr.g4 ## @@ -51,9 +51,12 @@ numericElement : (LONG | DOUBLE | NULL);

[GitHub] [druid] clintropolis commented on a change in pull request #10605: bitwise math function expressions

2021-01-20 Thread GitBox
clintropolis commented on a change in pull request #10605: URL: https://github.com/apache/druid/pull/10605#discussion_r560934713 ## File path: docs/misc/math-expr.md ## @@ -119,13 +119,14 @@ See javadoc of java.lang.Math for detailed explanation for each function.

[GitHub] [druid] clintropolis commented on a change in pull request #10767: Vectorized theta sketch aggregator + rework of VectorColumnProcessorFactory.

2021-01-20 Thread GitBox
clintropolis commented on a change in pull request #10767: URL: https://github.com/apache/druid/pull/10767#discussion_r560893111 ## File path: processing/src/main/java/org/apache/druid/segment/VectorColumnProcessorFactory.java ## @@ -22,34 +22,59 @@ import

[GitHub] [druid] zhangyue19921010 edited a comment on pull request #10779: [Bug Fix] Broker will not wait for its SQL metadata view to fully initialize before starting up, even though set awaitInitial

2021-01-20 Thread GitBox
zhangyue19921010 edited a comment on pull request #10779: URL: https://github.com/apache/druid/pull/10779#issuecomment-763504141 Hi @gianm Thanks for your review. I just add a UT named `testSchemaInit` as you suggested. If condition is ` if (isServerViewInitialized) `, this UT will

[GitHub] [druid] zhangyue19921010 commented on pull request #10779: [Bug Fix] Broker will not wait for its SQL metadata view to fully initialize before starting up, even though set awaitInitialization

2021-01-20 Thread GitBox
zhangyue19921010 commented on pull request #10779: URL: https://github.com/apache/druid/pull/10779#issuecomment-763504141 Hi @gianm Thanks for your review. I just add a UT named `testSchemaInit` as you suggested. If condition is ` if (isServerViewInitialized) `, this UT will fail with

[GitHub] [druid] egor-ryashin commented on a change in pull request #10651: fixed input source sampler buildReader exp

2021-01-20 Thread GitBox
egor-ryashin commented on a change in pull request #10651: URL: https://github.com/apache/druid/pull/10651#discussion_r560815377 ## File path: indexing-service/src/main/java/org/apache/druid/indexing/overlord/sampler/InputSourceSampler.java ## @@ -103,90 +103,102 @@ public

[GitHub] [druid] gianm commented on a change in pull request #10779: [Bug Fix] Broker will not wait for its SQL metadata view to fully initialize before starting up, even though set awaitInitializatio

2021-01-20 Thread GitBox
gianm commented on a change in pull request #10779: URL: https://github.com/apache/druid/pull/10779#discussion_r560770693 ## File path: sql/src/main/java/org/apache/druid/sql/calcite/schema/DruidSchema.java ## @@ -242,7 +242,9 @@ public void start() throws

[GitHub] [druid] jbonofre opened a new issue #10782: User queries storage in zookeeper

2021-01-20 Thread GitBox
jbonofre opened a new issue #10782: URL: https://github.com/apache/druid/issues/10782 ### Motivation Today, the user writes and executes queries in the web console or the query API. These queries can be scheduled via an external tool (like Decanter) to periodically execute

[GitHub] [druid] gianm commented on a change in pull request #10758: Retain order of AND, OR filter children.

2021-01-20 Thread GitBox
gianm commented on a change in pull request #10758: URL: https://github.com/apache/druid/pull/10758#discussion_r560758303 ## File path: processing/src/main/java/org/apache/druid/segment/filter/Filters.java ## @@ -552,4 +594,96 @@ public static boolean filterMatchesNull(Filter

[GitHub] [druid] gianm commented on a change in pull request #10758: Retain order of AND, OR filter children.

2021-01-20 Thread GitBox
gianm commented on a change in pull request #10758: URL: https://github.com/apache/druid/pull/10758#discussion_r560756535 ## File path: processing/src/main/java/org/apache/druid/query/filter/BooleanFilter.java ## @@ -23,13 +23,14 @@ import

[GitHub] [druid] gianm commented on a change in pull request #10758: Retain order of AND, OR filter children.

2021-01-20 Thread GitBox
gianm commented on a change in pull request #10758: URL: https://github.com/apache/druid/pull/10758#discussion_r560756535 ## File path: processing/src/main/java/org/apache/druid/query/filter/BooleanFilter.java ## @@ -23,13 +23,14 @@ import

[GitHub] [druid] gianm commented on a change in pull request #10758: Retain order of AND, OR filter children.

2021-01-20 Thread GitBox
gianm commented on a change in pull request #10758: URL: https://github.com/apache/druid/pull/10758#discussion_r560756111 ## File path: processing/src/main/java/org/apache/druid/segment/filter/Filters.java ## @@ -552,4 +594,96 @@ public static boolean filterMatchesNull(Filter