Re: CVE-2024-45537: Apache Druid: Users can provide MySQL JDBC properties not on allow list

[Abhishek Agarwal]

Also, the credit for finding this CVE goes to L0ne1y.

On Tue, Sep 17, 2024 at 11:29 PM Karan Kumar  wrote:

> Severity: low
>
> Affected versions:
>
> - Apache Druid through 30.0.0
>
> Description:
>
> Apache Druid allows users with certain permissions to read data from other
> database systems using JDBC. This functionality allows trusted users to set
> up Druid lookups or run ingestion tasks. Druid also allows administrators
> to configure a list of allowed properties that users are able to provide
> for their JDBC connections. By default, this allowed properties list
> restricts users to TLS-related properties only. However, when configuration
> a MySQL JDBC connection, users can use a particularly-crafted JDBC
> connection string to provide properties that are not on this allow list.
>
> Users without the permission to configure JDBC connections are not able to
> exploit this vulnerability.
> CVE-2021-26919 describes a similar vulnerability which was partially
> addressed in Apache Druid 0.20.2.
>
> This issue is fixed in Apache Druid 30.0.1.
>
> References:
>
> https://druid.apache.org
> https://www.cve.org/CVERecord?id=CVE-2024-45537
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org
>
>

Re: [DISCUSS] Druid 31.0.0 Release

[Abhishek Agarwal]

I have cut the branch here -  https://github.com/apache/druid/tree/31.0.0

On Fri, Sep 6, 2024 at 10:28 AM Abhishek Agarwal 
wrote:

> +1
>
>
> On Tue, Sep 3, 2024 at 1:08 PM Amatya Avadhanula 
> wrote:
>
>> Hello all,
>> I am starting a discussion for the Druid 31.0.0 release. I am volunteering
>> to be the release manager for the same. We aim to cut the branch for Druid
>> 31.0.0 on 8th September 2024 as we are already delayed a bit. Please use
>> this thread to discuss the important issues or features that are in flight
>> that should be a part of the Druid 31.0.0 release.
>> Regards,
>> Amatya
>>
>

[DISCUSS] Dropping support for Java 8

[Abhishek Agarwal]

Hello everyone,
Starting this thread to discuss, if and when, we can drop Java 8 support.
We have been fully supporting Java 11 and Java 17 for a while now. Anyone,
who is looking to upgrade Druid, can safely select either of these LTS Java
runtimes. There are a few important reasons to drop Java 8 support

- It adds extra burden on build/test pipelines to test all these different
runtimes. We want to shrink this matrix of Java runtime and test suites.
- Being on Java 8 will block us from upgrading dependencies that have
dropped Java 8 support. We can get around it by building profiles and shims
but it adds more complexity. One example is pac4j which is Java 11 based
from 5.x.
- As we drop support for older Java releases, developers can use the
features offered by the more advanced Java versions.

New committer : Zoltan Haindrich

[Abhishek Agarwal]

Hello everyone,

The Druid PMC has invited Zoltan to become a committer, and we are happy to
announce that he has accepted the invitation.

New PMC Member - Laksh Singla

[Abhishek Agarwal]

Hello everyone
We invited Laksh to join the Druid PMC and are pleased to announce that
Laksh has accepted the invite. Laksh has made significant contributions to
Druid, particularly in the MSQ engine.

Congratulations Laksh.

New PMC member - Amatya Avadhanula

[Abhishek Agarwal]

Hello everyone,
We invited Amatya to join Druid PMC and are pleased to announce that Amatya
has accepted the invite. Amatya has been a major contributor in all things
involving coordinator and overlord.

Congratulations Amatya.

Re: [VOTE] Release Apache Druid 29.0.0 [RC1]

[Abhishek Agarwal]

Vad, The vote was closed by Laksh, given that 72 hours had passed and
artifacts had been released. Given we had the three binding +1s, we can
proceed as usual. I suggest that we call out the caveat clearly in the
release notes section.

On Mon, Feb 19, 2024 at 11:53 PM Vadim Ogievetsky 
wrote:

> Thank you for all the work getting this release to this point. As the
> author of https://github.com/apache/druid/pull/15588 I regretfully vote
> -1 on this release. To echo Gian's point I think the chance of some user
> injuring a production Datasource with an innocent query pasted into the
> wrong web console tab is just too large. The fact that the
> "arrayIngestMode" setting itself is not new in this release is immaterial
> to my consideration as I am more worried about "what might happen by
> accident" than "what is technically possible to mess up".
>
> I have made a PR (https://github.com/apache/druid/pull/15927) that
> implements Gian's suggestion of: Making it obvious in the web console
> whether a tab is in "arrayIngestMode: array" or "arrayIngestMode: mvd" or
> "server default". Furthermore this PR also minimizes the use of
> "arrayIngestMode: array" to only when needed while still generating the
> "correct" form of the SQL queries at all times. If this PR was merged and
> back-ported I would change my vote to +1.
>
> I think that Gian's validation PR (
> https://github.com/apache/druid/pull/15920) is an even better solution,
> as it would protect API usage and usage via other frontends also. Obviously
> we could merge both.
>
> Another idea that would solve this issue would be have the ability to set
> context flags via SQL perhaps with a PARAMETER keyword. If this was
> possible the console could exclusively use that for setting query context
> parameters on generated queries.
>
> Vadim
>
> On 2024/02/17 02:52:38 Gian Merlino wrote:
> > Here's a patch with the validation idea:
> https://github.com/apache/druid/pull/15920
> >
> > It adds validation for the most problematic case (mixing strings and
> arrays), provides a way to override the validation, and makes the warning
> log on the controller task when arrayIngestMode is 'mvd' more friendly and
> explanatory.
> >
> > Depending on which direction you're going in, the errors look like
> either:
> >
> >   Cannot write into field[flags] using type[VARCHAR ARRAY] and
> arrayIngestMode[mvd], since the existing type is[VARCHAR ARRAY]. Try
> setting arrayIngestMode to[array] to retain the SQL type[VARCHAR ARRAY]
> >
> > Or:
> >
> >   Cannot write into field[flags] using type[VARCHAR ARRAY] and
> arrayIngestMode[array], since the existing type is[VARCHAR]. Try wrapping
> this field using ARRAY_TO_MV(...) AS "flags"
> >
> > The "try" language pushes people towards the behavior we'd like for the
> future: using arrayIngestMode[array] and wrapping MVDs in ARRAY_TO_MV.
> >
> > I'm changing my vote to a plain 0, given that _most_ of the changes
> related to arrayIngestMode went out in Druid 28. However I do think it
> would be nice to get a patch like this in, given that the Druid 29 web
> console is pushing more people to change their arrayIngestMode.
> >
> > Gian
> >
> > On 2024/02/16 22:24:23 Gian Merlino wrote:
> > > I just learned that arrayIngestMode is not actually new, just
> > > https://github.com/apache/druid/pull/15588 is. However this will
> still make
> > > it more likely that people accidentally break their tables, so I am
> still
> > > -0. Just, slightly less so. I still think it would be a good idea, for
> > > Druid 29, to add string-to-array type validation to Druid 29's INSERT /
> > > REPLACE handling to compensate for the new web console support for
> > > arrayIngestMode, and the UI changes to push people towards setting it
> to
> > > "array".
> > >
> > > I could be convinced that it's ok to do that in a 29.0.1. I don't
> think it
> > > should wait for 30, given the impact that can happen if people end up
> with
> > > mixed types without planning for it.
> > >
> > > On Fri, Feb 16, 2024 at 2:16 PM Gian Merlino  wrote:
> > >
> > > > Thanks for managing this release!
> > > >
> > > > My vote is -0, let me explain why. I am concerned about usability
> issues
> > > > with the new arrayIngestMode feature. There are various issues when
> mixing
> > > > MVD strings and string arrays in the same column: as soon as arrays
> show up
> > > > in a column, various "classic MVD-style" queries will fail at
> validation or
> > > > at runtime. I believe that the new feature, and especially the
> changes to
> > > > the web console in https://github.com/apache/druid/pull/15588, will
> make
> > > > it more likely that people will do this by accident and experience
> > > > brokenness.
> > > >
> > > > When this occurs, there is not an easy way to fix it; data needs to
> be
> > > > reingested or queries need to be adjusted. I believe that in some
> cases,
> > > > queries can't be adjusted without suffering from performance loss.
> > > >
> > > > This is something that c

Re: [Discuss] deprecating and eventually removing non-sql compatible configs

[Abhishek Agarwal]

@clint - Those configs are already in SQL compatible mode by default since
28. Aren't they?

To the original question, I am ok to deprecate these configs given that we
have enough releases for folks to migrate over their queries.

On Fri, Jan 26, 2024 at 4:55 AM Clint Wylie  wrote:

> >Are there any performance implications in making those the defaults?
> For numeric columns without null values there should be no impact. For
> numeric columns with null values, I believe there is currently a very
> minor performance impact for scans/aggregations since the null value
> bitmap is iterated to check which rows are null. String performance
> should not be changed afaik. I will try to find some time to do some
> benchmarking to ensure this is true and measure the stuff to confirm
> this, and look for opportunities to improve things where possible in
> order to make this happen, since performance is important. Longer
> term, I have a hunch that if we actually delete all of the default
> value mode code there might also be some performance improvements due
> to smaller function bytecode sizes and the lack of 'if sql compatible
> mode do this, else do this' being littered everywhere across the
> codebase.
>
> >Would this break compatibility for any existing native queries?
> There are a handful of query shapes which might have some differences,
> the most common are probably with strings since in default value mode
> the empty string and null are treated interchangeably, so these kinds
> of queries will need to be modified.
>
> There are also the possibility for some subtle differences in some
> numeric expressions due to the 'pretend nulls do not exist' behavior
> of the default value mode config, in default value mode you can write
> arguably nonsense expressions that end up evaluating stuff like "3 +
> 'hello'" and it will evaluate to 3, while in SQL compatible mode the
> 'hello' will be unable to be implicitly cast to a number and become
> zero, resulting in null.
>
> Strict boolean mode also has some subtle differences if selecting the
> outputs, since it homogenizes all true/false values to 1/0 instead of
> the javascript style behavior when it is off, where the truthy values
> are just passed through.
>
> https://druid.apache.org/docs/latest/querying/math-expr#logical-operator-modes
>
> The other main change in behavior is due to the SQL three-value logic,
> where a filter like "x != 'hello'" today will in default value mode
> match rows with null values, but in SQL compatible mode will not match
> nulls. These queries will need to be rewritten into the form "x !=
> 'hello' OR x is null", or there is also some native filters added
> recently which could be used to wrap to make the SQL equivalent of "(x
> = 'hello') is not true", https://github.com/apache/druid/pull/15182,
> or "not(istrue(eq(x, 'hello'))". I'm actually open to the idea of
> keeping the toggle config for 2vl/3vl since it would need checked in
> very few places to control the behavior (basically in not filter), my
> main goal here is to eventually get rid of
> druid.generic.useDefaultValueForNull.
>
> I plan to dig into all of this in greater depth in the migration guide
> mentioned in the first email to make the transition as painless as
> possible before we remove it.
>
> On Mon, Jan 22, 2024 at 9:54 AM Xavier Léauté
>  wrote:
> >
> > Two questions:
> > - Are there any performance implications in making those the defaults?
> > - Would this break compatibility for any existing native queries?
> >
> > On Wed, Jan 17, 2024 at 5:53 PM Clint Wylie  wrote:
> >
> > > Hi all,
> > >
> > > I wanted to discuss the deprecation and removal of the Druid configs
> > > related to SQL compatibility, with the eventual goal of always running
> > > in SQL compatible mode. As of Druid 28, these are the default, but in
> > > the interest of dramatically reducing complexity and removing a ton of
> > > code, and also cutting a lot of our CI time in half, I would
> > > eventually like to remove the related configs and the code that
> > > handles the now non-default behaviors completely.
> > >
> > > The related configs are:
> > >
> > > runtime.properties:
> > > druid.generic.useDefaultValueForNull  - (will become always false)
> > > druid.expressions.useStrictBooleans - (will become always true)
> > > druid.generic.useThreeValueLogicForNativeFilters - (will become always
> > > true)
> > > druid.generic.ignoreNullsForStringCardinality - (irrelevant if
> > > druid.generic.useDefaultValueForNull=false)
> > >
> > > query context:
> > > sqlUseBoundAndSelectors - (this is moderately related, and defaults to
> > > value of druid.generic.useDefaultValueForNull, but enhancements to
> > > expressions and sql planning for lookups make this totally unnecessary
> > > to keep around)
> > >
> > > other things to dump while we are at it:
> > > druid.expressions.allowNestedArrays - (will become always true)
> > >
> > > There might be additional configs which can also be removed, we ca

Re: [VOTE] Release Apache Druid 28.0.0 [RC1]

[Abhishek Agarwal]

+1 Binding

src package:
- verified signature/checksum
- LICENSE/NOTICE present
- built binary distribution,
 - Loaded example koalas-to-the-max nested dataset using MSQ and ran
some
queries
 - Tested Kafka ingestion locally

binary package:
- verified signature/checksum
- LICENSE/NOTICE present
- built binary distribution,
 - Loaded example koalas-to-the-max nested dataset using MSQ and ran
some
queries
 - Tested Kafka ingestion locally

docker:
- verified checksum
- started cluster with docker-compose, loaded example koalas-to-the-max
nested dataset using MSQ and ran some
queries
- Added the kafka extension to the environment, started cluster with
docker-compose, and then tested kafka ingestion

On Tue, Nov 7, 2023 at 11:08 AM Laksh Singla  wrote:

> Hi all,
>
> I have created a build for Apache Druid 28.0.0, release
> candidate 1.
>
> Thanks to everyone who has helped contribute to the release! You can read
> the proposed release notes here:
> https://github.com/apache/druid/issues/15326
>
> The release candidate has been tagged in GitHub as
> druid-28.0.0-rc1 (c91d61dc3f7232242c00a6334ea5d0d9cd98e32c),
> available here:
> https://github.com/apache/druid/releases/tag/druid-28.0.0-rc1
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/druid/28.0.0-rc1/
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachedruid-1056/
>
> Staged druid.apache.org website documentation is available here:
> https://druid.staged.apache.org/docs/28.0.0/design/index.html
>
> A Docker image containing the binary of the release candidate can be
> retrieved via:
> docker pull apache/druid:28.0.0-rc1
>
> artifact checksums
> src:
>
> f648143c29610f6874c5946897712bce967a4b9d5304b953c9efc14e514a7ee1269fcac626165544dd485560517fc52ef85ddd2617d3410fb0478e10f3ff2982
> bin:
>
> c971ccbb4220fa2ae9f8a0a0f82c3e4b87750d27ac15d863d1e9f415863ccb8b8e5c764122dc8ef76044a42d27e1108b980ca2b88de8aff660db5a22356d6ab7
> docker: 708f1bd7221734986d049e1b5bd6ad0c985c65d815037f0650dd5ace8654d38c
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/lakshsingla.asc
>
> This key and the key of other committers can also be found in the project's
> KEYS file here:
> https://dist.apache.org/repos/dist/release/druid/KEYS
>
> (If you are a committer, please feel free to add your own key to that file
> by following the instructions in the file's header.)
>
>
> Verify checksums:
> diff <(shasum -a512 apache-druid-28.0.0-src.tar.gz | \
> cut -d ' ' -f1) \
> <(cat apache-druid-28.0.0-src.tar.gz.sha512 ; echo)
>
> diff <(shasum -a512 apache-druid-28.0.0-bin.tar.gz | \
> cut -d ' ' -f1) \
> <(cat apache-druid-28.0.0-bin.tar.gz.sha512 ; echo)
>
> Verify signatures:
> gpg --verify apache-druid-28.0.0-src.tar.gz.asc \
> apache-druid-28.0.0-src.tar.gz
>
> gpg --verify apache-druid-28.0.0-bin.tar.gz.asc \
> apache-druid-28.0.0-bin.tar.gz
>
> Please review the proposed artifacts and vote. Note that Apache has
> specific requirements that must be met before +1 binding votes can be cast
> by PMC members. Please refer to the policy at
> http://www.apache.org/legal/release-policy.html#policy for more details.
>
> As part of the validation process, the release artifacts can be generated
> from source by running:
> mvn clean install -Papache-release,dist -Dgpg.skip
>
> The RAT license check can be run from source by:
> mvn apache-rat:check -Prat
>
> This vote will be open for at least 72 hours. The vote will pass if a
> majority of at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Druid 28.0.0
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
> Thanks!
>

New committer : Brian Le

[Abhishek Agarwal]

Hello everyone,

The Druid PMC has invited Brian Le to become a committer and we are pleased
to announce that Brian has accepted.

Brian has been a consistent contributor to the Apache Druid docs for over a
year and a half now. Thank you for all your work, Brian.

Re: [DISCUSS] Druid 28.0.0 release

[Abhishek Agarwal]

Thank you for volunteering, Laksh.

On Mon, Sep 25, 2023 at 10:50 AM Laksh Singla 
wrote:

> Hello all,
>
> I am starting a discussion for the Druid 28.0.0 release. I am volunteering
> to be the release manager for the same. The branch for Druid 27.0.0 was cut
> on 8th July 2023, and since we do quarterly releases, we should cut the
> branch for Druid 28.0.0 on 9th October 2023 (Monday). Please use this
> thread to discuss the important issues or features that are in flight, that
> should be a part of the Druid 28.0.0 release.
>
> Regards
> Laksh
>

Re: CVEs in contrib extensions

[Abhishek Agarwal]

That sounds reasonable. We can call put these bits in the docs so the
community doesn't feel we don't care about contrib extensions at all.

On Wed, Sep 6, 2023 at 4:14 AM Gian Merlino  wrote:

> I think it would be OK to have a policy that contrib extension dependencies
> are not proactively screened for CVEs. If we adopt such a policy, we do
> need to make it clear to people that they should do their own screening of
> any contrib extensions they use.
>
> However, we can't extend that policy to saying we don't take responsibility
> for security of contrib extensions at all. If there is a vulnerability in
> the code of a contrib extension itself, then we are obligated to fix it. If
> we receive a vulnerability report about a contrib extension, including a
> report about an issue with one of its dependencies (via the process at
> https://www.apache.org/security/#reporting-a-vulnerability) then we should
> take it seriously and investigate. This is the cost of having the code
> exist at all and be part of our source releases. We can only avoid _those_
> costs by removing an extension completely.
>
> On Mon, Sep 4, 2023 at 3:02 AM Abhishek Agarwal 
> wrote:
>
> > Hello all
> > What is our current policy about addressing CVEs in contrib extensions if
> > we have one? As of now, before the release, the release manager will
> either
> > try to fix the CVEs or add a suppression if applicable. Unless any
> > developer has done that same work before the release process begins.
> This,
> > however, is a tedious exercise for the release manager and for us
> > maintainers. With contrib extensions added to the mix, there is a huge
> > surface area for us to cover when it comes to managing CVEs in
> > dependencies.
> >
> > I propose excluding contrib extensions from our CVE checks so that RM can
> > ignore those CVEs during the release. We don't ship the contrib
> extensions
> > in distribution anyway, so it seems like a reasonable stance to me.
> >
>

CVEs in contrib extensions

[Abhishek Agarwal]

Hello all
What is our current policy about addressing CVEs in contrib extensions if
we have one? As of now, before the release, the release manager will either
try to fix the CVEs or add a suppression if applicable. Unless any
developer has done that same work before the release process begins. This,
however, is a tedious exercise for the release manager and for us
maintainers. With contrib extensions added to the mix, there is a huge
surface area for us to cover when it comes to managing CVEs in
dependencies.

I propose excluding contrib extensions from our CVE checks so that RM can
ignore those CVEs during the release. We don't ship the contrib extensions
in distribution anyway, so it seems like a reasonable stance to me.

Re: [VOTE] Release Apache Druid 27.0.0 [RC1]

[Abhishek Agarwal]

+1 (binding)

src package:
- verified signature/checksum
- LICENSE/NOTICE present
- built binary distribution,
 - Loaded example Wikipedia dataset using MSQ and ran some
queries
 - Tested Kafka ingestion locally

binary package:
- verified signature/checksum
- LICENSE/NOTICE present
- built binary distribution,
 - Loaded example wikipedia dataset using MSQ and ran some
queries
 - Tested Kafka ingestion locally

docker:
- verified checksum
- started cluster with docker-compose, Loaded example Wikipedia dataset and
ran some queries
- Added the kafka extension to the environment, started cluster with
docker-compose, and then tested kafka ingestion

On Sun, Aug 6, 2023 at 1:43 PM Amatya Avadhanula  wrote:

> Hi all,
>
> I have created a build for Apache Druid 27.0.0, release
> candidate 1.
>
> Thanks to everyone who has helped contribute to the release! You can read
> the proposed release notes here:
> https://github.com/apache/druid/issues/14761
>
> The release candidate has been tagged in GitHub as
> druid-27.0.0-rc1
> available here:
> https://github.com/apache/druid/tree/druid-27.0.0-rc1
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/druid/27.0.0-rc1/
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachedruid-1044/
>
> Staged druid.apache.org website documentation is available here:
> https://druid.staged.apache.org/docs/27.0.0/design/index.html
>
> A Docker image containing the binary of the release candidate can be
> retrieved via:
> docker pull apache/druid:27.0.0-rc1
>
> artifact checksums
> src:
>
> a3a755d02e2ed55a125ba562de4b4ce467d27af1132a89ecbc73cbb4a38622f7534813267ed44de1ec0fb85227e93d04c1c1ce24959d6d5c41dedbf2d7c6e4ed
> bin:
>
> b840ed0d77b1e5c11e058b161a56a469b0916febc1d478bf7ad2517cf79d2b724ece5cef21d96581122ba0958a5737f41e76496db3b10db5bfb4ab3123e4091b
> docker: ca3df175bc944033c7c56ccf9499c05e2090ae6cefbdcd90095cfce2b7931ead
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/amatya.asc [To be available
> within
> 48 hours: https://issues.apache.org/jira/browse/INFRA-24865]
>
> This key and the key of other committers can also be found in the project's
> KEYS file here:
> https://dist.apache.org/repos/dist/release/druid/KEYS
>
> (If you are a committer, please feel free to add your own key to that file
> by following the instructions in the file's header.)
>
>
> Verify checksums:
> diff <(shasum -a512 apache-druid-27.0.0-src.tar.gz | \
> cut -d ' ' -f1) \
> <(cat apache-druid-27.0.0-src.tar.gz.sha512 ; echo)
>
> diff <(shasum -a512 apache-druid-27.0.0-bin.tar.gz | \
> cut -d ' ' -f1) \
> <(cat apache-druid-27.0.0-bin.tar.gz.sha512 ; echo)
>
> Verify signatures:
> gpg --verify apache-druid-27.0.0-src.tar.gz.asc \
> apache-druid-27.0.0-src.tar.gz
>
> gpg --verify apache-druid-27.0.0-bin.tar.gz.asc \
> apache-druid-27.0.0-bin.tar.gz
>
> Please review the proposed artifacts and vote. Note that Apache has
> specific requirements that must be met before +1 binding votes can be cast
> by PMC members. Please refer to the policy at
> http://www.apache.org/legal/release-policy.html#policy for more details.
>
> As part of the validation process, the release artifacts can be generated
> from source by running:
> mvn clean install -Papache-release,dist -Dgpg.skip
>
> The RAT license check can be run from source by:
> mvn apache-rat:check -Prat
>
> This vote will be open for at least 72 hours. The vote will pass if a
> majority of at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Druid 0.17.0
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
> Thank you!
>

New PMC member - Kashif Faraz

[Abhishek Agarwal]

Hello everyone,
We invited Kashif to join druid PMC and we are pleased to announce that
Kashif has accepted the invite. Of late, Kashif has done a significant
amount of work on the coordinator and overlord so that we can continue to
push the envelope when it comes to the size of the cluster.

Congrats Kashif.

New PMC member - Karan Kumar

[Abhishek Agarwal]

Hi all
We invited Karan to join the Druid PMC and we are pleased to announce that
Karan has accepted the invite. As many of you know already, Karan has made
substantial contributions to the Druid and the MSQ engine that include
features such as fault tolerance and querying from deep storage.

Congrats Karan.

Re: [DISCUSS] Druid 27.0 release

[Abhishek Agarwal]

We have been doing it for the last 3-4 releases that it has almost become
like a guaranteed release activity. Hopefully, by the next release, we
won't have to do this again.

On Wed, Jul 12, 2023 at 7:04 PM Xavier Léauté 
wrote:

> ok, we should still go through a proper PR review and not make unilateral
> changes to the release branches.
> If it's too cumbersome to do so, then we might want to consider removing
> the helm charts from the main branch and wait for IP clearance resolution
> before adding them back.
>
> On Wed, Jul 12, 2023 at 7:40 AM Abhishek Agarwal 
> wrote:
>
> > Xavier,
> > I removed these from release branch since we still don't have a consensus
> > on IP clearance of helm charts
> > https://lists.apache.org/thread/dbypwsj7ykpkqyzm7gvopmprpw76r5oc
> >
> > On Wed, Jul 12, 2023 at 2:23 AM Xavier Léauté
>  > >
> > wrote:
> >
> > > Abhishek,
> > >
> > > Is there a reason for this commit to be in the release branch? I don't
> > see
> > > a PR or review for it
> > >
> > >
> >
> https://github.com/apache/druid/commit/911372d5d6523f425a46fbbafae728bdca441f0f
> > >
> > >
> > > Thanks,
> > > Xavier
> > >
> > > On Sat, Jul 8, 2023 at 6:00 AM Abhishek Agarwal 
> > > wrote:
> > >
> > > > I have cut the branch - https://github.com/apache/druid/tree/27.0.0.
> > > >
> > > > On Sat, Jul 8, 2023 at 12:25 AM Abhishek Agarwal <
> abhis...@apache.org>
> > > > wrote:
> > > >
> > > > > I can do it.
> > > > >
> > > > > On Fri, Jul 7, 2023 at 10:25 PM Amatya Avadhanula <
> ama...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > >> Hello
> > > > >>
> > > > >> As I'm traveling, I may not be able to cut the branch tomorrow.
> > > > >> Could I please request another committer to do it in my place?
> > > > >>
> > > > >> Apologies for the late notice.
> > > > >>
> > > > >> Regards
> > > > >> Amatya
> > > > >>
> > > > >>
> > -
> > > > >> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > > > >> For additional commands, e-mail: dev-h...@druid.apache.org
> > > > >>
> > > > >>
> > > >
> > >
> >
>

Re: About maintaining the Helm's Chart of Apache Druid

[Abhishek Agarwal]

Since no one else has volunteered, I will take on the 1). It's possible
that we don't get IP clearance and if we don't, we will just remove the
code.

On Wed, Mar 1, 2023 at 7:14 AM Gian Merlino  wrote:

> Not as far as I _know_, I mean.
>
> On 2023/03/01 01:43:43 Gian Merlino wrote:
> > Not as far as I do. I think we're stuck since nobody has volunteered to
> do one of the two necessary things:
> >
> > 1) shepherd this code the IP clearance process, or
> > 2) analyze its provenance enough to determine that IP clearance isn't
> necessary.
> >
> > If anyone is willing to do one of the above it would be greatly
> appreciated.
> >
> > At some point, if it seems like nobody will volunteer to do one of the
> above things, we'll need to remove the code, and whoever's interested in
> maintaining it would need to fork it off into another repo.
> >
> > Gian
> >
> > On 2023/02/15 18:37:57 Xavier Léauté wrote:
> > > Did we ever get to a conclusion here on IP clearance? Clint, it looks
> like
> > > the helm charts are still in the Druid repo and being contributed to.
> > >
> > > Thanks,
> > > Xavier
> > >
> > > On Tue, Sep 14, 2021 at 4:19 AM Clint Wylie  wrote:
> > >
> > > > My understanding of that thread suggests that
> > > > https://incubator.apache.org/ip-clearance/ip-clearance-template.html
> > > > is the process that our PMC does before the IPMC can continue the
> > > > clearance process, meaning someone on our PMC fill out i think this
> > > > form
> > > >
> http://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/ip-clearance-template.xml
> > > > ,
> > > > and check the things in and send the emails and such listed out in
> the
> > > > "process" section.
> > > >
> > > > I'm going to remove the helm chart from the 0.22.0 release for now
> > > > since I can't find any record of this having been done yet (I can add
> > > > it back if I missed it and we haven't yet released).
> > > >
> > > > On Sun, Jul 4, 2021 at 11:07 PM Benedict Jin 
> wrote:
> > > > >
> > > > > Hi Jihoon,
> > > > >
> > > > > Last week I asked the AFS, and according to the replay, it seems
> that
> > > > only our IPMC has the authority to launch the IP Clearance process.
> FYI,
> > > >
> https://lists.apache.org/thread.html/rfbfc5951c4524c0e68223e4fbe05a7d7ee26c185ab557d6f77a4989d%40%3Cgeneral.incubator.apache.org%3E
> > > > >
> > > > > Regards,
> > > > > Benedict Jin
> > > > >
> > > > > On 2021/07/02 22:56:08, Jihoon Son  wrote:
> > > > > > Hey Benedict,
> > > > > >
> > > > > > Any updates on this issue? I think we are going to start the
> release
> > > > > > process for 0.22.0 soon.
> > > > > >
> > > > > > On Fri, Jul 2, 2021 at 1:19 AM Benedict Jin  >
> > > > wrote:
> > > > > >
> > > > > > > Hi Xavier,
> > > > > > >
> > > > > > > I'm so happy to hear that and look forward to your changes
> will be
> > > > > > > contributed to upstream. In fact, Helm and Operator are not in
> > > > conflict,
> > > > > > > their relationship is kind like RPM and Systemd. You can even
> > > > convert Helm
> > > > > > > into Operator, or build Operator based on Helm. And I agree
> with you
> > > > that
> > > > > > > it would be better if we can define user scenarios.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Benedict Jin
> > > > > > >
> > > > > > > On 2021/06/25 22:42:32, Xavier Léauté
> 
> > > > > > > wrote:
> > > > > > > > For what it's worth, we have been using a heavily modified
> version
> > > > of
> > > > > > > this
> > > > > > > > helm chart at Confluent.
> > > > > > > >
> > > > > > > > I would say it is good to get a Druid cluster up and running
> > > > quickly, but
> > > > > > > > we had to make some significant changes to make it easier to
> > > > operate a
> > > > > > > > Druid cluster.
> > > > > > > > It's great for initial deployment and getting all the
> required
> > > > > > > dependencies
> > > > > > > > in place, but operations are somewhat painful and require a
> lot of
> > > > > > > internal
> > > > > > > > Druid knowledge to not shoot yourself in the foot.
> > > > > > > > Our original intention was to contribute back those changes
> > > > upstream, but
> > > > > > > > we have not had the time to put it in a shape that would
> allow
> > > > others to
> > > > > > > > use it.
> > > > > > > >
> > > > > > > > We should try to define what we want this chart to be used
> for,
> > > > since I
> > > > > > > > think the Druid k8s operator is probably a better choice for
> > > > someone to
> > > > > > > run
> > > > > > > > and upgrade a meaningful cluster.
> > > > > > > > Another option would be to focus our effort on the Druid
> operator
> > > > and
> > > > > > > maybe
> > > > > > > > build a helm chart to get that and our external dependencies
> in
> > > > place, I
> > > > > > > > think we can provide a better experience that way.
> > > > > > > > One concern with the pure helm chart is that we'll get a lot
> of
> > > > questions
> > > > > > > > on how to operate it that will likely take a lot of time to
> answer.
> > > > > 

Re: [DISCUSS] Druid 27.0 release

[Abhishek Agarwal]

Xavier,
I removed these from release branch since we still don't have a consensus
on IP clearance of helm charts
https://lists.apache.org/thread/dbypwsj7ykpkqyzm7gvopmprpw76r5oc

On Wed, Jul 12, 2023 at 2:23 AM Xavier Léauté 
wrote:

> Abhishek,
>
> Is there a reason for this commit to be in the release branch? I don't see
> a PR or review for it
>
> https://github.com/apache/druid/commit/911372d5d6523f425a46fbbafae728bdca441f0f
>
>
> Thanks,
> Xavier
>
> On Sat, Jul 8, 2023 at 6:00 AM Abhishek Agarwal 
> wrote:
>
> > I have cut the branch - https://github.com/apache/druid/tree/27.0.0.
> >
> > On Sat, Jul 8, 2023 at 12:25 AM Abhishek Agarwal 
> > wrote:
> >
> > > I can do it.
> > >
> > > On Fri, Jul 7, 2023 at 10:25 PM Amatya Avadhanula 
> > > wrote:
> > >
> > >> Hello
> > >>
> > >> As I'm traveling, I may not be able to cut the branch tomorrow.
> > >> Could I please request another committer to do it in my place?
> > >>
> > >> Apologies for the late notice.
> > >>
> > >> Regards
> > >> Amatya
> > >>
> > >> -
> > >> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > >> For additional commands, e-mail: dev-h...@druid.apache.org
> > >>
> > >>
> >
>

Re: [DISCUSS] Druid 27.0 release

[Abhishek Agarwal]

I have cut the branch - https://github.com/apache/druid/tree/27.0.0.

On Sat, Jul 8, 2023 at 12:25 AM Abhishek Agarwal 
wrote:

> I can do it.
>
> On Fri, Jul 7, 2023 at 10:25 PM Amatya Avadhanula 
> wrote:
>
>> Hello
>>
>> As I'm traveling, I may not be able to cut the branch tomorrow.
>> Could I please request another committer to do it in my place?
>>
>> Apologies for the late notice.
>>
>> Regards
>> Amatya
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
>> For additional commands, e-mail: dev-h...@druid.apache.org
>>
>>

Re: [DISCUSS] Druid 27.0 release

[Abhishek Agarwal]

I can do it.

On Fri, Jul 7, 2023 at 10:25 PM Amatya Avadhanula  wrote:

> Hello
>
> As I'm traveling, I may not be able to cut the branch tomorrow.
> Could I please request another committer to do it in my place?
>
> Apologies for the late notice.
>
> Regards
> Amatya
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org
>
>

Re: [DISCUSS] Druid 27.0 release

[Abhishek Agarwal]

+1
Given the delays in the previous release, we should definitely cut the
branch on time so the release is not delayed like last time. That and
having a higher bar for backports.
We decided to deprecate Hadoop 2 in this release. It doesn't require any
code changes. We just need to call it out in the release notes.

On Mon, Jul 3, 2023 at 3:32 PM Amatya Avadhanula  wrote:

> Hello all,
> I am starting a discussion for the 27.0.0 release. I volunteer to be the
> release manager. The branch for druid 26.0.0 was cut on 8th April, 2023.
> Since we do quarterly releases, we should now cut the 27.0.0 branch on 8th
> July 2023. Please use this thread to discuss important issues or features
> that are in flight, and can be released in time for 27.0.0.
>
> I had also volunteered to become the RM for 26.0.1 but I think that we need
> not do 26.0.1, and could just do 27.0.0 in time.
>
> Regards
> Amatya
>

Re: [DISCUSS] Apache Druid Release 26.0.1

[Abhishek Agarwal]

I think we only need to backport high sev bug fixes for this patch release.

On Wed, 28 Jun 2023 at 5:27 PM, Amatya Apache  wrote:

> Hi all,
>
> A release branch for 26.0.1 has been cut to include the patches mentioned
> in https://lists.apache.org/thread/4xl7wthc22bfszn4woytx9q6jz63clr4.
>
> Kindly let us know of any other changes that you'd like to have backported
> to this release.
>
> Thank you
> Amatya
>

Requirements for relaxing restrictions on github actions usage

[Abhishek Agarwal]

Hello,
I raised an INFRA ticket (https://issues.apache.org/jira/browse/INFRA-24657)
for the druid project so the contributors don't need a committer to trigger
PR build/test. Infra has agreed to relax the restrictions enough that a
contributor will need the approval only for their first contribution.

However, as a project, we need to follow certain requirements that are
called out here - https://infra.apache.org/github-actions-policy.html

They all seem fine to me. We are using `pull_request_target` for the
labeler action but that action doesn't export any confidential variables.
If others agree as well, I will just link this thread to the INFRA ticket.

As a follow-up item, I can add a README.md in .github folder that warns
contributors and committers to keep these requirements in mind as they
change GitHub workflows in future.

New Committer : Didip Kerabat

[Abhishek Agarwal]

Hello everyone,

The Project Management Committee (PMC) for Apache Druid has invited Didip
to become a committer and we are pleased to announce that Didip has
accepted.

Didip is very active on the Druid slack channel and helps Druid users
troubleshoot their problems regularly. He has also fixed bugs and added
features that have benefited our users.

Congrats Didip.

New Committer : Laksh Singla

[Abhishek Agarwal]

Hello everyone,

The Project Management Committee (PMC) for Apache Druid has invited Laksh
to become a committer and we are pleased to announce that Laksh has
accepted.

Laksh has made a lot of contributions to MSQ over one year. He has been
helping the druid community on Slack. His most notable contribution is
extending the calcite parser that enabled us to add custom SQL syntax for
MSQ. He also worked on improving the test coverage of MSQ by porting
CalciteQueryTests to the MSQ engine. He has made the parse exceptions more
informative to aid in troubleshooting.

Congratulations Laksh.

New Committer : Tejaswini Bandlamudi

[Abhishek Agarwal]

Hello everyone,

The Project Management Committee (PMC) for Apache Druid has invited
Tejaswini to become a committer and we are pleased to announce that
Tejaswini has accepted.

Tejaswini has been working on Apache Druid for over a year now. Her major
contribution has been porting our test infra from Travis to GHA. She has
fixed various bugs in the ingestion area. She added the feature for a
supervisor to become idle when there is no data on the topic. This can help
cluster operators save money when data sources receive data sparsely.

Congratulation Tejaswini.

Re: [DISCUSS] Druid 26.0 release

[Abhishek Agarwal]

Druid is very stable. If you have specific concerns, please create a GitHub
issue.

25.0.0 was the last release we had. Starting with 24.0.0, we dropped the
leading 0. You can find the details here -
https://lists.apache.org/thread/xcfjoq696hn5w85hdgpxfb9hx4nsj4h1
In a way, we released 1.0 about 24 releases ago.

On Wed, Mar 22, 2023 at 11:09 AM Tony Schwartz 
wrote:

> how stable is druid nowadays?  when are you going to release v. 1.0
> ?Tony Sent from my Verizon, Samsung Galaxy smartphone
>  Original message From: Clint Wylie 
> Date: 3/20/23  22:38  (GMT-05:00) To: dev@druid.apache.org Subject:
> [DISCUSS] Druid 26.0 release Hey all,It is about that time again to think
> about doing another release. Ivolunteer to be release manager, and if no
> objections will aim to makethe branch around the first week of April.Please
> use this thread to call to attention anything which should beresolved
> before we make the release branch.It would be nice to reach a conclusion on
> the helm chart IP clearancequestion before this releasehttps://
> lists.apache.org/thread/ygyzt23m06vc775nq5dsm349rf0j47dg, butI'm not sure
> I have the time/energy to spare to drive it to resolutionand personally
> know very little about helm to commit to maintainingit. Any
> volunteers?-To
> unsubscribe, e-mail: dev-unsubscribe@druid.apache.orgFor additional
> commands, e-mail: dev-h...@druid.apache.org

Re: [Discuss] S3 buckets or IT tests

[Abhishek Agarwal]

I would prefer using something like min.io than running tests against S3.
Primarily because tests no longer remain portable if we use S3. Which also
makes them harder to debug. Setting up minio using GHA seems
straightforward (https://github.com/mozilla/sccache/pull/1513/files).

On Mon, Mar 6, 2023 at 7:01 AM Karan Kumar  wrote:

> Oh yes, https://issues.apache.org/jira/browse/INFRA-23952 Is the ticket I
> am talking about. As the INFRA folks require approval to proceed, can one
> of the PMC's drop a comment regarding approval/rejection on the same jira.
> If we are not able to procure a bucket, then we can start exploring mini
> IO.
>
> On Wed, Feb 22, 2023 at 6:07 PM Gian Merlino  wrote:
>
> > I think the ticket you're referring to is
> > https://issues.apache.org/jira/browse/INFRA-23952.
> >
> > It would definitely be valuable to run S3 integration tests as part of
> the
> > automated test suite in GitHub Actions. If Infra is willing to provide a
> > bucket for this purpose then we would certainly be able to use that. I
> bet
> > we could also use Minio (https://min.io/) or something similar.
> >
> > Gian
> >
> > On 2023/02/15 21:38:47 Karan Kumar wrote:
> > > Hey Folks
> > > S3 read write tests currently are not executed in github actions since
> we
> > > do not have public creds to read/write from s3.
> > > Have raised an ASF infra ticket
> > >
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > > so that they can give us a bucket.
> > > They require a PMC approval.
> > > As discussed on this slack thread
> > >
> >
> https://apachedruidworkspace.slack.com/archives/C030CMF6B70/p1675916945323839
> > ,
> > > starting a formal discussion around it.
> > >
> > > The objective would be to integrate our s3 IT's tests to run on each
> PR.
> > >
> > >
> > >
> > > --
> > > Thanks
> > > Karan
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > For additional commands, e-mail: dev-h...@druid.apache.org
> >
> >
>
> --
> Thanks
> Karan
>

Re: [VOTE] Release Apache Druid 24.0.2 [RC1]

[Abhishek Agarwal]

+1 (binding)

src package:
- verified signature/checksum
- LICENSE/NOTICE present
- built binary distribution, Loaded example wikipedia dataset and ran some
queries

binary package:
- verified signature/checksum
- LICENSE/NOTICE present
- Loaded example wikipedia dataset and ran some queries

docker:
- verified checksum
- started cluster with docker-compose, Loaded example wikipedia dataset and
ran some queries

On Tue, Nov 29, 2022 at 2:00 PM Laksh Singla 
wrote:

> +1 (non-binding)
> Verified that https://github.com/apache/druid/pull/13138 is included and
> working in the build.
>
> Regards
> Laksh
>
> The https://github.com/apache/druid/pull/13138 is
>
> On Sat, Nov 26, 2022 at 2:52 PM Kashif Faraz  wrote:
>
> > Hi
> >
> > I have created a build for Apache Druid 24.0.2, release candidate 1.
> >
> > Thanks to everyone who has helped contribute to the release! You can read
> > the proposed release notes here:
> > https://github.com/apache/druid/issues/13435
> >
> > The release candidate has been tagged in GitHub as druid-24.0.2-rc1
> > (13c9fdcbafcf61211a9381892615c4b4db6a3535),
> > available here:
> > https://github.com/apache/druid/releases/tag/druid-24.0.2-rc1
> >
> > The artifacts to be voted on are located here:
> > https://dist.apache.org/repos/dist/dev/druid/24.0.2-rc1/
> >
> > A staged Maven repository is available for review at:
> > https://repository.apache.org/content/repositories/orgapachedruid-1037/
> >
> > A Docker image containing the binary of the release candidate can be
> > retrieved via:
> > docker pull apache/druid:24.0.2-rc1
> >
> > Artifact checksums:
> > src:
> >
> >
> 1ca1ea4b9a4418813fc22cdba2626725f802ff886e118b9986b89e8a6df9fe304a0fb0eca1370a1c937ac5b136448c8157d58b1f79712347ca5cb67f7f034cfc
> > bin:
> >
> >
> ca22923d0c1b60803289ada2be6c7e74b41641e95e3ad36fb625edd71c62d29f5203221062e5bf7af8199075bbc46606ec3ae8b559b455e1ecfc990f0e669f55
> > docker:
> > e96ae3293242e90e0a78be128c0d12325b8a6fc5c68ce2f2bc53de3aac568bbf
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/kfaraz.asc
> >
> > This key and the key of other committers can also be found in the
> project's
> > KEYS file here:
> > https://dist.apache.org/repos/dist/release/druid/KEYS
> >
> > (If you are a committer, please feel free to add your own key to that
> file
> > by following the instructions in the file's header.)
> >
> > Verify checksums:
> > diff <(shasum -a512 apache-druid-24.0.2-src.tar.gz | \
> > cut -d ' ' -f1) \
> > <(cat apache-druid-24.0.2-src.tar.gz.sha512 ; echo)
> >
> > diff <(shasum -a512 apache-druid-24.0.2-bin.tar.gz | \
> > cut -d ' ' -f1) \
> > <(cat apache-druid-24.0.2-bin.tar.gz.sha512 ; echo)
> >
> > Verify signatures:
> > gpg --verify apache-druid-24.0.2-src.tar.gz.asc \
> > apache-druid-24.0.2-src.tar.gz
> >
> > gpg --verify apache-druid-24.0.2-bin.tar.gz.asc \
> > apache-druid-24.0.2-bin.tar.gz
> >
> > Please review the proposed artifacts and vote. Note that Apache has
> > specific requirements that must be met before +1 binding votes can be
> cast
> > by PMC members. Please refer to the policy at
> > http://www.apache.org/legal/release-policy.html#policy for more details.
> >
> > As part of the validation process, the release artifacts can be generated
> > from source by running:
> > mvn clean install -Papache-release,dist -Dgpg.skip
> >
> > The RAT license check can be run from source by:
> > mvn apache-rat:check -Prat
> >
> > This vote will be open for at least 72 hours. The vote will pass if a
> > majority of at least three +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Druid 24.0.2
> > [ ] 0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> >
> > Thanks
> > Kashif
> >
>

Re: [VOTE] Release Apache Druid 24.0.1 [RC1]

[Abhishek Agarwal]

st -Dgpg.skip
> > >
> > > The RAT license check can be run from source by:
> > > mvn apache-rat:check -Prat
> > >
> > > This vote will be open for at least 72 hours. The vote will pass if a
> > > majority of at least three +1 PMC votes are cast.
> > >
> > > [ ] +1 Release this package as Apache Druid 24.0.1
> > > [ ] 0 I don't feel strongly about it, but I'm okay with the release
> > > [ ] -1 Do not release this package because...
> > >
> > > Thanks
> > > Kashif
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > For additional commands, e-mail: dev-h...@druid.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org
>
>

-- 
Regards,
Abhishek Agarwal

New Committer : Karan Kumar

[Abhishek Agarwal]

Hello everyone,

The Project Management Committee (PMC) for Apache Druid has invited Karan
Kumar to become a committer and we are pleased to announce that Karan has
accepted.

Karan contributed group-by on arrays without unnesting (
https://github.com/apache/druid/pull/12078), an essential requirement for
MSQ. Karan has been making a lot of contributions to MSQ. He has been doing
many code reviews and helps out users regularly on apache druid slack.

Congratulations Karan.

Re: [DISCUSS] Druid 25.0 release

[Abhishek Agarwal]

Yeah. That will be a great feature addition in the 25 release. Though it
will certainly be an experimental feature.

On Wed, Oct 26, 2022 at 10:18 PM Kashif Faraz  wrote:

> Thanks for pointing that out, Rahul.
> We are indeed looking forward to the option of running Druid without middle
> managers and running tasks as k8s jobs!
> The corresponding PR is currently under review too and would most likely be
> included in the Druid 25.0 release.
>
> Thanks
> Kashif
>
> On Wed, Oct 26, 2022 at 9:49 PM rahul gidwani 
> wrote:
>
> > What about the mm-less patch?  That would be nice to have in this next
> > release?
> >
> > Rahul
> >
> >
> >
> > --- Original Message ---
> > On Wednesday, October 26th, 2022 at 9:13 AM, Kashif Faraz <
> > kfa...@apache.org> wrote:
> >
> >
> > > Hello
> > >
> > > On the 16th of September, we released Druid 24.0.0 which was a major
> > > milestone in our journey, introducing significant improvements and a
> > whole
> > > new multi-stage query engine! Keeping up the momentum, we would like to
> > > start preparing for the next quarterly release of Druid 25.0 in
> December.
> > > Since that is still a little way off, we are also working on a patch
> > > release (24.0.1) in the meantime to address some issues identified in
> > Druid
> > > 24.0. I am volunteering to be the manager for both of these upcoming
> > > releases.
> > >
> > > We can use this thread to discuss important issues or features that
> would
> > > be a nice addition to Druid 25.0. There are some much awaited features
> > > already in progress such as support for UNNEST in SQL, fault-tolerance
> > and
> > > sequential sketch merge in multi-stage query engine, front-coded string
> > > dictionaries, segment balancing improvements and a Hadoop3-compatible
> > Druid
> > > distro. We would love to include other PRs that you feel would make
> Druid
> > > better.
> > >
> > > The tentative date for the code-freeze is November 14. So chase after
> > your
> > > reviewer buddies and rack up those commits!
> > >
> > > Regards
> > > Kashif
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > For additional commands, e-mail: dev-h...@druid.apache.org
> >
> >
>

Re: [E] Re: Apache Druid Slack

[Abhishek Agarwal]

I recently learned about Apache Flink making slack conversations google
indexable through linen.dev. That might be worth exploring. Does anyone
know how it works?

On Wed, Oct 19, 2022 at 4:18 AM Eyal Yurman 
wrote:

> What should we do about Slack canceling message history for free tier?
>
> Perhaps we should export history elsewhere?
>
> On Thu, Jan 27, 2022 at 1:10 PM Vadim Ogievetsky 
> wrote:
>
> > Alright, I have setup the new Slack workspace and updated the link on
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__druid.apache.org_community_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=hV9FPFkuEXDmMbTGo3m_iqUc-baVKjOHznj9kig1I1Q&e=
> > to point to it.
> > Please join with the
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__druid.apache.org_community_join-2Dslack&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=07e2sPTwojKxi997mSN4a5Ktcr1lyHcekXuTwEZj9JY&e=
> > link (and share it with others).
> > Sadly there is no way to migrate content from one Slack workspace to
> > workspace, I am sure we will fill the new channel with great content very
> > quickly!
> >
> > On 2022/01/20 18:15:23 Vadim Ogievetsky wrote:
> > > I think that the PMC should create a new Slack channel for Apache Druid
> > and
> > > shift the community towards using it away from the ASF Slack. I
> volunteer
> > > to do this on the PMCs behalf and do all the setup/admin work. I would
> > > share admin access with any PMC member.
> > >
> > > What is the motivation for this?
> > >
> > > As you may have heard, it’s become increasingly difficult for new users
> > > without an @apache.org email address to join the ASF #druid Slack
> > channel.
> > > ASF Infra disabled the option to publicly provide a link to the
> workspace
> > > to anyone who wanted it, after encountering issues with spammers.
> > >
> > > Per Infra’s guidance (
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__infra.apache.org_slack.html&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=qZfDyv1d4QnrCrEjvTgSbm8qeEj-B-dFWKil2k4XOyQ&e=
> > ), new community
> > > members should only be invited as single-channel guests. Unfortunately,
> > > single-channel guests are unable to extend invitations to new members,
> > > including their colleagues who are using Druid. Only someone with full
> > > member privileges is able to extend an invitation to new members. This
> > lack
> > > of consistency doesn’t make the community feel inclusive.
> > >
> > > There is a workaround in place (
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid-2Dwebsite-2Dsrc_pull_278&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=LZAv4yoEkFJcPmJckcAR0xXyuo7V81ZfR2X_jxQjH58&e=
> > ) – users can send an
> > > email to druid-u...@googlegroups.com to request an invite to the Slack
> > > channel from an existing member – but this still poses a barrier to
> > entry,
> > > and isn’t a viable permanent solution. It also creates potential
> privacy
> > > issues as not everyone is at liberty to announce they’re using Druid
> nor
> > > wishes to display their email address in a public forum.
> > >
> > > I propose we make our own free Slack channel for Apache Druid and
> > encourage
> > > people to migrate to it. Then we can have our own policy on Slack
> > > invitations - I would like to restore the ability for anyone on the web
> > to
> > > join our Slack.
> > >
> > > This is not a 100% original idea, in fact this is what other Apache
> > > projects have done, notably Apache Pinot (see "join our slack" on
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__pinot.apache.org_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=GYTpyHD-1U_Er2eKNXk41b9aiy4zW9EtCt-2YXxSh9o&e=
> > ). I propose we do the same.
> > >
> > > Vadim
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > > For additional commands, e-mail: dev-h...@druid.apache.org
> > >
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > For additional commands, e-mail: dev-h...@druid.apache.org
> >
> >
>

Re: [DISCUSS] Release 24.0.1

[Abhishek Agarwal]

Thank you, Kashif. It works for me.

On Mon, Oct 17, 2022 at 7:29 PM Kashif Faraz  wrote:

> Hi Abhishek
>
> If you haven't started with the release process already, I would like to
> volunteer to perform this release so that we can expedite it.
> Please let me know if that works for you.
>
> Regards
> Kashif
>
> On Mon, Sep 26, 2022 at 3:41 PM Abhishek Agarwal <
> abhishek.agar...@imply.io> wrote:
>
>> Hi All,
>> Recently we discovered a regression (
>> https://github.com/apache/druid/pull/13138) in the 24.0.0 release.
>> Because
>> of this regression, Hadoop ingestion will not work if the user has
>> overridden any of the `druid.extensions.*` config. Some examples below
>> - If a custom load list is specified, Hadoop ingestion task will still
>> pick
>> up all the extensions in "extension" directory.
>> - If a custom Hadoop dependency directory is being used, those jars will
>> no
>> longer be available for Hadoop Ingestion.
>>
>> I have created a branch https://github.com/apache/druid/tree/24.0.1 to
>> backport the hotfix. We can also choose to backport some other bug fixes
>> that we missed in 24.0.0. Though, of course, we want to backport only
>> critical bug fixes.
>>
>

New Committer : Paul Rogers

[Abhishek Agarwal]

Hello everyone,

The Project Management Committee (PMC) for Apache Druid has invited Paul
Rogers to become a committer and we are pleased to announce that Paul has
accepted.

Paul has been very active in the community. He has done a lot of thorough
code reviews. He single-handedly built the new IT framework. He has been
cleaning up the SQL wiring for some time now. He has put many
forward-looking design proposals that are detailed and in-depth e.g. druid
catalog, calcite planner test framework, etc. Paul also brings in a wealth
of experience and ideas from the Drill community.

Congratulations Paul.

New Committer : Amatya Avadhanula

[Abhishek Agarwal]

Hello everyone,

The Project Management Committee (PMC) for Apache Druid has invited Amatya
Avadhanula to become a committer and we are pleased to announce that Amatya
has accepted.

Amatya has been making a lot of fixes on the kinesis ingestion and systems
(overlord, task locking) side. He optimized the task fetching in overlord
from the console which is going to benefit many heavy clusters. He fixed
the kinesis ingestion to work with empty shards which had been a major
limitation in working with kinesis ingestion. His work has been
high-quality with great test coverage. He has also been helping the
community on slack and Github.

Congratulations Amatya.

[DISCUSS] Release 24.0.1

[Abhishek Agarwal]

Hi All,
Recently we discovered a regression (
https://github.com/apache/druid/pull/13138) in the 24.0.0 release. Because
of this regression, Hadoop ingestion will not work if the user has
overridden any of the `druid.extensions.*` config. Some examples below
- If a custom load list is specified, Hadoop ingestion task will still pick
up all the extensions in "extension" directory.
- If a custom Hadoop dependency directory is being used, those jars will no
longer be available for Hadoop Ingestion.

I have created a branch https://github.com/apache/druid/tree/24.0.1 to
backport the hotfix. We can also choose to backport some other bug fixes
that we missed in 24.0.0. Though, of course, we want to backport only
critical bug fixes.

Re: [ANNOUNCE] Apache Druid 24.0.0 release

[Abhishek Agarwal]

Hello everyone,
As you might have noticed, there is also a change in the versioning scheme
in the 24.0.0 release. Starting with this release, we have dropped the
leading 0 from the release version and promoted all other digits one place
to the left. Druid is now at major version 24, a jump up from the prior
0.23.0 release. In terms of backward compatibility or breaking changes,
this release is not significantly different than other previous major
releases such as 0.23.0 or 0.22.0. We are continuing with the same policy
as we have used in prior releases: minimizing the number of changes that
require special attention when upgrading, and calling out any that do exist
in the release notes. For the 24.0.0 release, please refer to the Upgrading
to 24.0.0 section
<https://github.com/apache/druid/releases/tag/druid-24.0.0#24.0.0-upgrading-to-24.0>
for a list of backward-incompatible changes in this release.

Please reach out to druid community <https://druid.apache.org/community/> if
you have any questions.

On Fri, Sep 16, 2022 at 10:14 PM Abhishek Agarwal 
wrote:

> The Apache Druid team is proud to announce the release of Apache Druid
> 24.0.0.
> Druid is a high-performance analytics data store for event-driven data.
>
> Apache Druid 24.0.0 contains over 300 new features, performance
> enhancements, bug fixes, and major documentation improvements from 67
> contributors. 24.0.0 is a significant milestone with new ground-breaking
> capabilities - Multi-stage query engine and nested columns.
>
> This release also includes
>  - completely revamped query view in the web console.
>  - official support for Java 11 and experimental support for Java 17.
>
>
> Source and binary distributions can be downloaded from:
> https://druid.apache.org/downloads.html
>
> Release notes are at:
> https://github.com/apache/druid/releases/tag/druid-24.0.0
>
> A big thank you to all the contributors to this milestone release!
>

[ANNOUNCE] Apache Druid 24.0.0 release

[Abhishek Agarwal]

The Apache Druid team is proud to announce the release of Apache Druid
24.0.0.
Druid is a high-performance analytics data store for event-driven data.

Apache Druid 24.0.0 contains over 300 new features, performance
enhancements, bug fixes, and major documentation improvements from 67
contributors. 24.0.0 is a significant milestone with new ground-breaking
capabilities - Multi-stage query engine and nested columns.

This release also includes
 - completely revamped query view in the web console.
 - official support for Java 11 and experimental support for Java 17.


Source and binary distributions can be downloaded from:
https://druid.apache.org/downloads.html

Release notes are at:
https://github.com/apache/druid/releases/tag/druid-24.0.0

A big thank you to all the contributors to this milestone release!

Re: [VOTE] Release Apache Druid 24.0.0 [RC2]

[Abhishek Agarwal]

The vote has passed. The final results are here -
https://lists.apache.org/thread/0f7fgkgsydkxjmhyr32wgdwtv8tnq91h

On Sat, Sep 10, 2022 at 2:02 PM Abhishek Agarwal 
wrote:

> Hi all,
> I have created a build for Apache Druid 24.0.0, release candidate 2.
>
> Thanks to everyone who has helped contribute to the release. You can read
> the draft release notes here:
> https://github.com/apache/druid/issues/12825
>
> The release candidate has been tagged in GitHub as druid-24.0.0-rc2
> (09991dcf17fb9984b65f291f0db47e38058253e4), available here:
> https://github.com/apache/druid/releases/tag/druid-24.0.0-rc2
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/druid/24.0.0-rc2/
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachedruid-1032
>
> Staged druid.apache.org website documentation is available here:
> https://druid.staged.apache.org/docs/24.0.0/design/index.html
>
> A Docker image containing the binary of the release candidate can be
> retrieved via:
> docker pull apache/druid:24.0.0-rc2
>
> artifact checksums
> src:
> 867e3c4787e4e3f2a35e066cc982698d3e1a77fe90be7de163bd893908c13fec7fd8a84e19020f6f32fb517502a8eb729ae4492d51549f72a57f2bbf42b08e9c
>
> bin:
>
> 8b15192a7fb5a6f979b0c8f51bfd0dee5d26a5912ec2a3d25f4739ebbf89cadebed353ba30977a0b13b6410f89a2991eb7ee5ebea4217026129ce2262875dc91
>
> docker (sha256):
> c5ab9872c2f6e340f1429fd0bd514f163b0e491a9d808ee85de9a1b8f81ffe0b
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/abhishek
> This key and the key of other committers can also be found in the project's
> KEYS file here:
> https://dist.apache.org/repos/dist/release/druid/KEYS
>
> (If you are a committer, please feel free to add your own key to that file
> by following the instructions in the file's header.)
>
> Verify checksums:
> diff <(shasum -a512 apache-druid-24.0.0-src.tar.gz | cut -d ' ' -f1) <(cat
> \
> apache-druid-24.0.0-src.tar.gz.sha512 ; echo)
>
> diff <(shasum -a512 apache-druid-24.0.0-bin.tar.gz | cut -d ' ' -f1) <(cat
> \
> apache-druid-24.0.0-bin.tar.gz.sha512 ; echo)
>
> Verify signatures:
> gpg --verify apache-druid-24.0.0-src.tar.gz.asc \
> apache-druid-24.0.0-src.tar.gz
>
> gpg --verify apache-druid-24.0.0-bin.tar.gz.asc \
> apache-druid-24.0.0-bin.tar.gz
>
> Please review the proposed artifacts and vote. Note that Apache has
> specific requirements that must be met before +1 binding votes can be cast
> by PMC members. Please refer to the policy at
> http://www.apache.org/legal/release-policy.html#policy for more details.
>
> As part of the validation process, the release artifacts can be generated
> from source by running:
> mvn clean install -Papache-release,dist -Dgpg.skip
>
> The RAT license check can be run from source by:
> mvn apache-rat:check -Prat
>
> This vote will be open for at least 72 hours. The vote will pass if a
> majority of at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Druid 24.0.0
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
> Thanks!
>

[RESULT] [VOTE] Release Apache Druid 24.0.0 [RC2]

[Abhishek Agarwal]

Thanks to everyone who participated in the vote! The results are as follows:

Frank Chen: 0 (binding)
Vadim Ogievetsky: +1 (binding) (assumed binding since Vadim is a PMC member)
Clint Wylie: +1 (binding)
Jonathan Wei: +1 (binding)

The vote has passed with 3 binding +1s.

[VOTE] Release Apache Druid 24.0.0 [RC2]

[Abhishek Agarwal]

Hi all,
I have created a build for Apache Druid 24.0.0, release candidate 2.

Thanks to everyone who has helped contribute to the release. You can read
the draft release notes here:
https://github.com/apache/druid/issues/12825

The release candidate has been tagged in GitHub as druid-24.0.0-rc2
(09991dcf17fb9984b65f291f0db47e38058253e4), available here:
https://github.com/apache/druid/releases/tag/druid-24.0.0-rc2

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/druid/24.0.0-rc2/

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachedruid-1032

Staged druid.apache.org website documentation is available here:
https://druid.staged.apache.org/docs/24.0.0/design/index.html

A Docker image containing the binary of the release candidate can be
retrieved via:
docker pull apache/druid:24.0.0-rc2

artifact checksums
src:
867e3c4787e4e3f2a35e066cc982698d3e1a77fe90be7de163bd893908c13fec7fd8a84e19020f6f32fb517502a8eb729ae4492d51549f72a57f2bbf42b08e9c

bin:
8b15192a7fb5a6f979b0c8f51bfd0dee5d26a5912ec2a3d25f4739ebbf89cadebed353ba30977a0b13b6410f89a2991eb7ee5ebea4217026129ce2262875dc91

docker (sha256):
c5ab9872c2f6e340f1429fd0bd514f163b0e491a9d808ee85de9a1b8f81ffe0b

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/abhishek
This key and the key of other committers can also be found in the project's
KEYS file here:
https://dist.apache.org/repos/dist/release/druid/KEYS

(If you are a committer, please feel free to add your own key to that file
by following the instructions in the file's header.)

Verify checksums:
diff <(shasum -a512 apache-druid-24.0.0-src.tar.gz | cut -d ' ' -f1) <(cat \
apache-druid-24.0.0-src.tar.gz.sha512 ; echo)

diff <(shasum -a512 apache-druid-24.0.0-bin.tar.gz | cut -d ' ' -f1) <(cat \
apache-druid-24.0.0-bin.tar.gz.sha512 ; echo)

Verify signatures:
gpg --verify apache-druid-24.0.0-src.tar.gz.asc \
apache-druid-24.0.0-src.tar.gz

gpg --verify apache-druid-24.0.0-bin.tar.gz.asc \
apache-druid-24.0.0-bin.tar.gz

Please review the proposed artifacts and vote. Note that Apache has
specific requirements that must be met before +1 binding votes can be cast
by PMC members. Please refer to the policy at
http://www.apache.org/legal/release-policy.html#policy for more details.

As part of the validation process, the release artifacts can be generated
from source by running:
mvn clean install -Papache-release,dist -Dgpg.skip

The RAT license check can be run from source by:
mvn apache-rat:check -Prat

This vote will be open for at least 72 hours. The vote will pass if a
majority of at least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Druid 24.0.0
[ ] 0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...

Thanks!

[CANCEL] [VOTE] Release Apache Druid 24.0.0 [RC1]

[Abhishek Agarwal]

Canceling this vote due to the bug described in
https://github.com/apache/druid/pull/13059. The fix is merged to 24.0
branch. I will start another vote for RC2 shortly.

On Thu, Sep 8, 2022 at 10:26 PM Abhishek Agarwal 
wrote:

> Hi all,
> I have created a build for Apache Druid 24.0.0, release candidate 1.
>
> Thanks to everyone who has helped contribute to the release. You can read
> the draft release notes here:
> https://github.com/apache/druid/issues/12825
>
> The release candidate has been tagged in GitHub as druid-24.0.0-rc1
> (2319f10cb6c56bd1781355bce168f25c5168f008), available here:
> https://github.com/apache/druid/releases/tag/druid-24.0.0-rc1
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/druid/24.0.0-rc1/
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachedruid-1031
>
> Staged druid.apache.org website documentation is available here:
> https://druid.staged.apache.org/docs/24.0.0/design/index.html
>
> A Docker image containing the binary of the release candidate can be
> retrieved via:
> docker pull apache/druid:24.0.0-rc1
>
> artifact checksums
> src:
> edeb2a0f22bfd09910fc0411de4a3e97115912b0f51ded337b496380e93049abd2ef9c2549ef81b8449eef79ec4f2e71ddbbab2bc3036443a325d15fe1f32939
>
> bin:
>
> 582eb7c1e67e850dd3bf5fe3bec238b0944252bb676020b6891d56c2b5d756ecc9e8fd58651efda3b192bf018818328c7fe3278833003f78cae82651aa4829c3
>
> docker (sha256):
> e563237bf217dad11770f80bdd6ef223a0182cd6f3e730437341b9716c773897
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/abhishek
> This key and the key of other committers can also be found in the project's
> KEYS file here:
> https://dist.apache.org/repos/dist/release/druid/KEYS
>
> (If you are a committer, please feel free to add your own key to that file
> by following the instructions in the file's header.)
>
> Verify checksums:
> diff <(shasum -a512 apache-druid-24.0.0-src.tar.gz | cut -d ' ' -f1) <(cat
> apache-druid-24.0.0-src.tar.gz.sha512 ; echo)
>
> diff <(shasum -a512 apache-druid-24.0.0-bin.tar.gz | cut -d ' ' -f1) <(cat
> apache-druid-24.0.0-bin.tar.gz.sha512 ; echo)
>
> Verify signatures:
> gpg --verify apache-druid-24.0.0-src.tar.gz.asc \
> apache-druid-24.0.0-src.tar.gz
>
> gpg --verify apache-druid-24.0.0-bin.tar.gz.asc \
> apache-druid-24.0.0-bin.tar.gz
>
> Please review the proposed artifacts and vote. Note that Apache has
> specific requirements that must be met before +1 binding votes can be cast
> by PMC members. Please refer to the policy at
> http://www.apache.org/legal/release-policy.html#policy for more details.
>
> As part of the validation process, the release artifacts can be generated
> from source by running:
> mvn clean install -Papache-release,dist -Dgpg.skip
>
> The RAT license check can be run from source by:
> mvn apache-rat:check -Prat
>
> This vote will be open for at least 72 hours. The vote will pass if a
> majority of at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Druid 24.0.0
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
> Thanks!
>

[VOTE] Release Apache Druid 24.0.0 [RC1]

[Abhishek Agarwal]

Hi all,
I have created a build for Apache Druid 24.0.0, release candidate 1.

Thanks to everyone who has helped contribute to the release. You can read
the draft release notes here:
https://github.com/apache/druid/issues/12825

The release candidate has been tagged in GitHub as druid-24.0.0-rc1
(2319f10cb6c56bd1781355bce168f25c5168f008), available here:
https://github.com/apache/druid/releases/tag/druid-24.0.0-rc1

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/druid/24.0.0-rc1/

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachedruid-1031

Staged druid.apache.org website documentation is available here:
https://druid.staged.apache.org/docs/24.0.0/design/index.html

A Docker image containing the binary of the release candidate can be
retrieved via:
docker pull apache/druid:24.0.0-rc1

artifact checksums
src:
edeb2a0f22bfd09910fc0411de4a3e97115912b0f51ded337b496380e93049abd2ef9c2549ef81b8449eef79ec4f2e71ddbbab2bc3036443a325d15fe1f32939

bin:
582eb7c1e67e850dd3bf5fe3bec238b0944252bb676020b6891d56c2b5d756ecc9e8fd58651efda3b192bf018818328c7fe3278833003f78cae82651aa4829c3

docker (sha256):
e563237bf217dad11770f80bdd6ef223a0182cd6f3e730437341b9716c773897

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/abhishek
This key and the key of other committers can also be found in the project's
KEYS file here:
https://dist.apache.org/repos/dist/release/druid/KEYS

(If you are a committer, please feel free to add your own key to that file
by following the instructions in the file's header.)

Verify checksums:
diff <(shasum -a512 apache-druid-24.0.0-src.tar.gz | cut -d ' ' -f1) <(cat
apache-druid-24.0.0-src.tar.gz.sha512 ; echo)

diff <(shasum -a512 apache-druid-24.0.0-bin.tar.gz | cut -d ' ' -f1) <(cat
apache-druid-24.0.0-bin.tar.gz.sha512 ; echo)

Verify signatures:
gpg --verify apache-druid-24.0.0-src.tar.gz.asc \
apache-druid-24.0.0-src.tar.gz

gpg --verify apache-druid-24.0.0-bin.tar.gz.asc \
apache-druid-24.0.0-bin.tar.gz

Please review the proposed artifacts and vote. Note that Apache has
specific requirements that must be met before +1 binding votes can be cast
by PMC members. Please refer to the policy at
http://www.apache.org/legal/release-policy.html#policy for more details.

As part of the validation process, the release artifacts can be generated
from source by running:
mvn clean install -Papache-release,dist -Dgpg.skip

The RAT license check can be run from source by:
mvn apache-rat:check -Prat

This vote will be open for at least 72 hours. The vote will pass if a
majority of at least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Druid 24.0.0
[ ] 0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...

Thanks!

Re: [DISCUSS] Druid 24.0 release

[Abhishek Agarwal]

FYI, I have created the 24.0.0 branch from
https://github.com/apache/druid/commit/8ee8786d3ce32f39d8517b5a06211101b2a4

On Wed, Jul 27, 2022 at 9:01 AM Charles Smith 
wrote:

> In preparation for the 24.0 release, I've opened an issue where we can
> compose the release notes:
> https://github.com/apache/druid/issues/12825
>
> Since the milestone isn't set yet, we can work from the commit for the
> 0.23.0 release to start filling in the various sections of the release
> notes.
>
> Thanks,
>
> Charles
>
> On Wed, Jul 20, 2022 at 11:41 PM Abhishek Agarwal <
> abhishek.agar...@imply.io>
> wrote:
>
> > Hello Druids,
> > We cut the 0.23.0 branch in the last week of April. So while it isn't a
> > long ago that druid 0.23.0 was released, technically we will cross the
> > 3-month window at the end of July. It feels like the right time to
> discuss
> > the 24.0 release.
> >
> > I am volunteering again for the 24.0 release. Let's use this thread to
> > identify any important issues and features that we want to get in. I am
> > particularly looking forward to the support for nested columns and
> > multi-stage distributed queries in the upcoming release. There are some
> PRs
> > yet to be raised for the latter (
> > https://github.com/apache/druid/issues/12262#issuecomment-1162615382).
> We
> > should get those in before the code freeze for 24.0. Are there any other
> > in-flight PRs that we should/can get into 24.0?
> >
> > I am also referring to this release as "24.0" instead of "0.24.0". This
> is
> > based on the discussion we had here -
> > https://lists.apache.org/thread/w7phy1kz52lq3p9mty597tmtd3br858m.
> >
>

Re: [E] [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x for 24.0

[Abhishek Agarwal]

Yes. We should deprecate it first which is similar to dropping the support
(no more active development) but we will still ship it for a release or
two. In a way, we are already in that mode to a certain extent. Many
features are being built with native ingestion as a first-class citizen.
E.g. range partitioning is still not supported on Hadoop ingestion. It's
hard for developers to build and test their business logic for all the
ingestion modes.

It will be good to hear what gaps do community sees between native
ingestion vs Hadoop-based batch ingestion. And then work toward fixing
those gaps before dropping the Hadoop ingestion entirely. For example, if
users want the resource elasticity that a Hadoop cluster gives, we could
push forward PRs such as https://github.com/apache/druid/pull/10910. It's
not the same as a Hadoop cluster but nonetheless will let user reuse their
existing infrastructure to run druid jobs.

On Tue, Aug 9, 2022 at 9:43 AM Gian Merlino  wrote:

> It's always good to deprecate things for some time prior to removing them,
> so we don't need to (nor should we) remove Hadoop 2 support right now. My
> vote is that in this upcoming release, we should deprecate it. The main
> problem in my eyes is the one Abhishek brought up: the dependency
> management situation with Hadoop 2 is really messy, and I'm not sure
> there's a good way to handle them given the limited classloader isolation.
> This situation becomes tougher to manage with each release, and we haven't
> had people volunteering to find and build comprehensive solutions. It is
> time to move on.
>
> The concern Samarth raised, that people may end up stuck on older Druid
> versions because they aren't able to upgrade to Hadoop 3, is valid. I can
> see two good solutions to this. First: we can improve native ingest to the
> point where people feel broadly comfortable moving Hadoop 2 workloads to
> native. The work planned as part of doing ingest via multi-stage
> distributed query <https://github.com/apache/druid/issues/12262> is going
> to be useful here, by improving the speed and scalability of native ingest.
> Second: it would also be great to have something similar that runs on
> Spark, for people that have made investments in Spark. I suspect that most
> people that used Hadoop 2 have moved on to Hadoop 3 or Spark, so supporting
> both of those would ease a lot of the potential pain of dropping Hadoop 2
> support.
>
> On Spark: I'm not familiar with the current state of the Spark work. Is it
> stuck? If so could something be done to unstick it? I agree with Abhishek
> that I wouldn't want to block moving off Hadoop 2 on this. However, it'd be
> great if we could get it done before actually removing Hadoop 2 support
> from the code base.
>
>
> On Wed, Aug 3, 2022 at 6:17 AM Abhishek Agarwal  >
> wrote:
>
> > I was thinking that moving from Hadoop 2 to Hadoop 3 will be a
> > low-resistance path than moving from Hadoop to Spark. even if we get that
> > PR merged, it will take good time for spark integration to reach the same
> > level of maturity as Hadoop or Native ingestion. BTW I am not making an
> > argument against spark integration. it will certainly be nice to have
> Spark
> > as an option. Just that spark integration doesn't become a blocker for us
> > to get off Hadoop.
> >
> > btw are you using Hadoop 2 right now with the latest druid version? If
> so,
> > did you run into similar errors that I posted in my last email?
> >
> > On Wed, Jul 27, 2022 at 12:02 AM Samarth Jain 
> > wrote:
> >
> > > I am sure there are other companies out there who are still on Hadoop
> 2.x
> > > with migration to Hadoop 3.x being a no-go.
> > > If Druid was to drop support for Hadoop 3.x completely, I am afraid it
> > > would prevent users from updating to newer versions of Druid which
> would
> > be
> > > a shame.
> > >
> > > FWIW, we have found in practice for high volume use cases that
> compaction
> > > based on Druid's Hadoop based batch ingestion is a lot more scale-able
> > than
> > > the native compaction.
> > >
> > > Having said that, as an alternative, if we can merge Julian's Spark
> based
> > > ingestion PR <https://github.com/apache/druid/issues/9780>s in Druid,
> > that
> > > might provide an alternate way for users to get rid of the Hadoop
> > > dependency.
> > >
> > > On Tue, Jul 26, 2022 at 3:19 AM Abhishek Agarwal <
> > > abhishek.agar...@imply.io>
> > > wrote:
> > >
> > > > Reviving this conversation again.
> &g

Re: [E] [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x for 24.0

[Abhishek Agarwal]

I was thinking that moving from Hadoop 2 to Hadoop 3 will be a
low-resistance path than moving from Hadoop to Spark. even if we get that
PR merged, it will take good time for spark integration to reach the same
level of maturity as Hadoop or Native ingestion. BTW I am not making an
argument against spark integration. it will certainly be nice to have Spark
as an option. Just that spark integration doesn't become a blocker for us
to get off Hadoop.

btw are you using Hadoop 2 right now with the latest druid version? If so,
did you run into similar errors that I posted in my last email?

On Wed, Jul 27, 2022 at 12:02 AM Samarth Jain 
wrote:

> I am sure there are other companies out there who are still on Hadoop 2.x
> with migration to Hadoop 3.x being a no-go.
> If Druid was to drop support for Hadoop 3.x completely, I am afraid it
> would prevent users from updating to newer versions of Druid which would be
> a shame.
>
> FWIW, we have found in practice for high volume use cases that compaction
> based on Druid's Hadoop based batch ingestion is a lot more scale-able than
> the native compaction.
>
> Having said that, as an alternative, if we can merge Julian's Spark based
> ingestion PR <https://github.com/apache/druid/issues/9780>s in Druid, that
> might provide an alternate way for users to get rid of the Hadoop
> dependency.
>
> On Tue, Jul 26, 2022 at 3:19 AM Abhishek Agarwal <
> abhishek.agar...@imply.io>
> wrote:
>
> > Reviving this conversation again.
> > @Will - Do you still have concerns about HDFS stability? Hadoop 3 has
> been
> > around for some time now and is very stable as far as I know.
> >
> > The dependencies coming from Hadoop 2 are also old enough that they cause
> > dependency scans to fail. E.g. Log4j 1.x dependencies that are coming
> from
> > Hadoop 2, get flagged during these scans. We have also seen issues when
> > customers try to use Hadoop ingestion with the latest log4j2 library.
> >
> > Exception in thread "main" java.lang.NoSuchMethodError:
> >
> >
> org.apache.log4j.helpers.OptionConverter.convertLevel(Ljava/lang/String;Lorg/apache/logging/log4j/Level;)Lorg/apache/logging/log4j/Level;
> > at
> >
> >
> org.apache.log4j.config.PropertiesConfiguration.parseLogger(PropertiesConfiguration.java:393)
> > at
> >
> >
> org.apache.log4j.config.PropertiesConfiguration.configureRoot(PropertiesConfiguration.java:326)
> > at
> >
> >
> org.apache.log4j.config.PropertiesConfiguration.doConfigure(PropertiesConfiguration.java:303)
> >
> >
> > Instead of fixing these point issues, we would be better served by
> > completely moving to Hadoop 3 entirely. Hadoop 3 does get more frequent
> > releases and dependencies are well isolated.
> >
> > On Tue, Oct 12, 2021 at 12:05 PM Karan Kumar 
> > wrote:
> >
> > > Hello
> > > We can also use maven profiles. We keep hadoop2 support by default and
> > add
> > > a new maven profile with hadoop3. This will allow the user to choose
> the
> > > profile which is best suited for the use case.
> > > Agreed, it will not help in the Hadoop dependency problems but does
> > enable
> > > our users to use druid with multiple flavors.
> > > Also with hadoop3, as clint mentioned, the dependencies come pre-shaded
> > so
> > > we significantly reduce our effort in solving the dependency problems.
> > > I have the PR in the last phases where I am able to run the entire test
> > > suit unit + integration tests on both the default ie hadoop2 and the
> new
> > > hadoop3 profile.
> > >
> > >
> > >
> > > On 2021/06/09 11:55:31, Will Lauer 
> > > wrote:
> > > > Clint,
> > > >
> > > > I fully understand what type of headache dealing with these
> dependency
> > > > issues is. We deal with this all the time, and based on conversations
> > > I've
> > > > had with our internal hadoop development team, they are quite aware
> of
> > > them
> > > > and just as frustrated by them as you are. I'm certainly in favor of
> > > doing
> > > > something to improve this situation, as long as it doesn't abandon a
> > > large
> > > > section of the user base, which I think DROPPING hadoop2 would do.
> > > >
> > > > I think there are solutions there that can help solve the conflicting
> > > > dependency problem. Refactoring Hadoop support into an independent
> > > > extension is certainly a start. But I think the dependency problem is
> > &g

Re: [E] [DISCUSS] Hadoop 3, dropping support for Hadoop 2.x for 24.0

[Abhishek Agarwal]

Reviving this conversation again.
@Will - Do you still have concerns about HDFS stability? Hadoop 3 has been
around for some time now and is very stable as far as I know.

The dependencies coming from Hadoop 2 are also old enough that they cause
dependency scans to fail. E.g. Log4j 1.x dependencies that are coming from
Hadoop 2, get flagged during these scans. We have also seen issues when
customers try to use Hadoop ingestion with the latest log4j2 library.

Exception in thread "main" java.lang.NoSuchMethodError:
org.apache.log4j.helpers.OptionConverter.convertLevel(Ljava/lang/String;Lorg/apache/logging/log4j/Level;)Lorg/apache/logging/log4j/Level;
at
org.apache.log4j.config.PropertiesConfiguration.parseLogger(PropertiesConfiguration.java:393)
at
org.apache.log4j.config.PropertiesConfiguration.configureRoot(PropertiesConfiguration.java:326)
at
org.apache.log4j.config.PropertiesConfiguration.doConfigure(PropertiesConfiguration.java:303)


Instead of fixing these point issues, we would be better served by
completely moving to Hadoop 3 entirely. Hadoop 3 does get more frequent
releases and dependencies are well isolated.

On Tue, Oct 12, 2021 at 12:05 PM Karan Kumar 
wrote:

> Hello
> We can also use maven profiles. We keep hadoop2 support by default and add
> a new maven profile with hadoop3. This will allow the user to choose the
> profile which is best suited for the use case.
> Agreed, it will not help in the Hadoop dependency problems but does enable
> our users to use druid with multiple flavors.
> Also with hadoop3, as clint mentioned, the dependencies come pre-shaded so
> we significantly reduce our effort in solving the dependency problems.
> I have the PR in the last phases where I am able to run the entire test
> suit unit + integration tests on both the default ie hadoop2 and the new
> hadoop3 profile.
>
>
>
> On 2021/06/09 11:55:31, Will Lauer 
> wrote:
> > Clint,
> >
> > I fully understand what type of headache dealing with these dependency
> > issues is. We deal with this all the time, and based on conversations
> I've
> > had with our internal hadoop development team, they are quite aware of
> them
> > and just as frustrated by them as you are. I'm certainly in favor of
> doing
> > something to improve this situation, as long as it doesn't abandon a
> large
> > section of the user base, which I think DROPPING hadoop2 would do.
> >
> > I think there are solutions there that can help solve the conflicting
> > dependency problem. Refactoring Hadoop support into an independent
> > extension is certainly a start. But I think the dependency problem is
> > bigger than that. There are always going to be conflicts between
> > dependencies in the core system and in extensions as the system gets
> > bigger. We have one right now internally that prevents us from enabling
> SQL
> > in our instance of Druid due to conflicts between versions of protobuf
> used
> > by Calcite vs one of our critical extensions. Long term, I think you are
> > going to need to carefully think through a ClassLoader based strategy to
> > truly separate the impact of various dependencies.
> >
> > While I'm not seriously suggesting it for Druid, OSGi WOULD solve this
> > problem. It's a system that allows you to explicitly declare what each
> > bundle exposes to the system, and what each bundle consumes from the
> > system, allowing multiple conflicting dependencies to co-exist without
> > impacting each other. OSGi is the big hammer approach, but I bet a more
> > appropriate solution would be a simpler custom-ClassLoader based solution
> > that hid all dependencies in extensions, keeping them from impacting the
> > core, and that only exposed "public" pieces of the core to extensions. If
> > Druid's core could be extended without impacting the various extensions,
> > and the extensions' dependencies could be modified without impacting the
> > core, this would go a long way towards solving the problem that you have
> > described.
> >
> > Will
> >
> > 
> >
> > Will Lauer
> >
> > Senior Principal Architect, Audience & Advertising Reporting
> > Data Platforms & Systems Engineering
> >
> > M 508 561 6427
> > 1908 S. First St
> > Champaign, IL 61822
> >
> >    <
> http://twitter.com/verizonmedia>
> > 
> > 
> >
> >
> >
> > On Wed, Jun 9, 2021 at 12:47 AM Clint Wylie  wrote:
> >
> > > @itai, I think pending the outcome of this discussion that it makes
> sense
> > > to have a wider community thread to announce any decisions we make
> here,
> > > thanks for bringing that up.
> > >
> > > @rajiv, Minio support seems unrelated to this discussion. It seems
> like a
> > > reasonable request, but I recommend starting another thread to see if
> > > someone is interested in taking up this effort.
> > >
> > > @jihoon I definitely agree that Hadoop should be refactored to be an
> > > extension long

New Committer : Rohan Garg

[Abhishek Agarwal]

Hello everyone,

The Project Management Committee (PMC) for Apache Druid has invited Rohan
Garg to become a committer and we are pleased to announce that Rohan has
accepted.

Rohan has been an active contributor for about a year now. He has made over
20 commits. A majority of his work has been around SQL planning for better
query performance. He has also helped the community via PR reviews.

Congratulations Rohan.

[DISCUSS] Druid 24.0 release

[Abhishek Agarwal]

Hello Druids,
We cut the 0.23.0 branch in the last week of April. So while it isn't a
long ago that druid 0.23.0 was released, technically we will cross the
3-month window at the end of July. It feels like the right time to discuss
the 24.0 release.

I am volunteering again for the 24.0 release. Let's use this thread to
identify any important issues and features that we want to get in. I am
particularly looking forward to the support for nested columns and
multi-stage distributed queries in the upcoming release. There are some PRs
yet to be raised for the latter (
https://github.com/apache/druid/issues/12262#issuecomment-1162615382). We
should get those in before the code freeze for 24.0. Are there any other
in-flight PRs that we should/can get into 24.0?

I am also referring to this release as "24.0" instead of "0.24.0". This is
based on the discussion we had here -
https://lists.apache.org/thread/w7phy1kz52lq3p9mty597tmtd3br858m.

CVE-2022-28889: Apache Druid: Clickjacking in the web console

[Abhishek Agarwal]

Description:

In Apache Druid 0.22.1 and earlier, the server did not set appropriate headers 
to prevent clickjacking. Druid 0.23.0 and later prevent clickjacking using the 
Content-Security-Policy header.

Mitigation:

Upgrade to Druid 0.23.0 or later.


-
To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
For additional commands, e-mail: dev-h...@druid.apache.org

CVE-2021-44791: Apache Druid: Reflected XSS on certain HTTP endpoints

[Abhishek Agarwal]

Severity: low

Description:

In Apache Druid 0.22.1 and earlier, certain specially-crafted links result in 
unescaped URL parameters being sent back in HTML responses. This makes it 
possible to execute reflected XSS attacks.

Mitigation:

Upgrade to Druid 0.23.0 or later.

Credit:

This issue was discovered by DangKhai from Viettel Cyber Security


-
To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
For additional commands, e-mail: dev-h...@druid.apache.org

Re: [DISCUSS] Removing code related to `FireHose`

[Abhishek Agarwal]

@Clint Wylie  - what do you think of the above
proposal?

On Fri, Jun 24, 2022 at 5:11 PM Abhishek Agarwal 
wrote:

> I didn’t include them (RealtimeIndexTask and
> AppenderatorDriverRealtimeIndexTask) in my previous email because they have
> not been marked deprecated yet. We should mark them deprecated officially
> in the next release and remove them in the release after that.
>
> So looks like the classes that we can definitely remove are
> implementations of `FiniteFirehoseFactory` and mark the `Firehose`
> interface deprecated.
>
> On Fri, 24 Jun 2022 at 4:36 AM, Clint Wylie  wrote:
>
>> If we remove RealtimeIndexTask and AppenderatorDriverRealtimeIndexTask
>> then we can remove EventReceiverFirehoseFactory. The former was
>> primarily used by tranquility which has been sunset, the latter I'm
>> not sure was ever used for anything. I'm personally in favor of
>> removing both of them since push based ingestion is very fragile in my
>> experience, but I think some of the oldest integration tests use
>> RealtimeIndexTask and so would need to be removed/updated/rewritten to
>> use something else as appropriate.
>>
>> I don't think we can completely remove InputRowParser until we drop
>> Hadoop support (or modify Hadoop ingestion to use
>> InputSource/InputFormat?), since it still relies on using the older
>> spec. As far as I know, Thrift is the only data format that has not
>> been fully migrated to use InputFormat, though there is an old PR that
>> is mostly done  here https://github.com/apache/druid/pull/11360.
>>
>> On Thu, Jun 23, 2022 at 5:11 AM Abhishek Agarwal
>>  wrote:
>> >
>> > Hello,
>> > The `FiniteFirehoseFactory` and `InputRowParser` classes were
>> deprecated in
>> > 0.17.0 (https://github.com/apache/druid/pull/8823) in favour of
>> > `InputSource`.  0.17.0 was released more than 2 years ago in Jan 2020.
>> >
>> > I think it is about time that we remove this code entirely. Removing
>> > `InputRowParser` may not be as trivial as `EventReceiverFirehoseFactory`
>> > depends on it. I didn't find any alternatives for
>> > `EventReceiverFirehoseFactory` and it is not marked deprecated as well.
>> >
>> > But we can still remove `FiniteFirehoseFactory` and the implementations
>> > safely as there are alternatives available.
>> >
>> > Thoughts/Suggestions?
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
>> For additional commands, e-mail: dev-h...@druid.apache.org
>>
>>

Re: [DISCUSS] Removing code related to `FireHose`

[Abhishek Agarwal]

I didn’t include them (RealtimeIndexTask and
AppenderatorDriverRealtimeIndexTask) in my previous email because they have
not been marked deprecated yet. We should mark them deprecated officially
in the next release and remove them in the release after that.

So looks like the classes that we can definitely remove are implementations
of `FiniteFirehoseFactory` and mark the `Firehose` interface deprecated.

On Fri, 24 Jun 2022 at 4:36 AM, Clint Wylie  wrote:

> If we remove RealtimeIndexTask and AppenderatorDriverRealtimeIndexTask
> then we can remove EventReceiverFirehoseFactory. The former was
> primarily used by tranquility which has been sunset, the latter I'm
> not sure was ever used for anything. I'm personally in favor of
> removing both of them since push based ingestion is very fragile in my
> experience, but I think some of the oldest integration tests use
> RealtimeIndexTask and so would need to be removed/updated/rewritten to
> use something else as appropriate.
>
> I don't think we can completely remove InputRowParser until we drop
> Hadoop support (or modify Hadoop ingestion to use
> InputSource/InputFormat?), since it still relies on using the older
> spec. As far as I know, Thrift is the only data format that has not
> been fully migrated to use InputFormat, though there is an old PR that
> is mostly done  here https://github.com/apache/druid/pull/11360.
>
> On Thu, Jun 23, 2022 at 5:11 AM Abhishek Agarwal
>  wrote:
> >
> > Hello,
> > The `FiniteFirehoseFactory` and `InputRowParser` classes were deprecated
> in
> > 0.17.0 (https://github.com/apache/druid/pull/8823) in favour of
> > `InputSource`.  0.17.0 was released more than 2 years ago in Jan 2020.
> >
> > I think it is about time that we remove this code entirely. Removing
> > `InputRowParser` may not be as trivial as `EventReceiverFirehoseFactory`
> > depends on it. I didn't find any alternatives for
> > `EventReceiverFirehoseFactory` and it is not marked deprecated as well.
> >
> > But we can still remove `FiniteFirehoseFactory` and the implementations
> > safely as there are alternatives available.
> >
> > Thoughts/Suggestions?
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org
>
>

[DISCUSS] Removing code related to `FireHose`

[Abhishek Agarwal]

Hello,
The `FiniteFirehoseFactory` and `InputRowParser` classes were deprecated in
0.17.0 (https://github.com/apache/druid/pull/8823) in favour of
`InputSource`.  0.17.0 was released more than 2 years ago in Jan 2020.

I think it is about time that we remove this code entirely. Removing
`InputRowParser` may not be as trivial as `EventReceiverFirehoseFactory`
depends on it. I didn't find any alternatives for
`EventReceiverFirehoseFactory` and it is not marked deprecated as well.

But we can still remove `FiniteFirehoseFactory` and the implementations
safely as there are alternatives available.

Thoughts/Suggestions?

[ANNOUNCE] Apache Druid 0.23.0 release

[Abhishek Agarwal]

The Apache Druid team is proud to announce the release of Apache Druid
0.23.0.
Druid is a high-performance analytics data store for event-driven data.

Apache Druid 0.23.0 contains over 450 new features, performance
enhancements, bug fixes, and major documentation improvements from 81
contributors. Major new features and improvements include:

- Multi-dimensional range partitioning
- Improved support for "Arrays"
- Much better ease of use
- Improved Replace data behavior
- Much more powerful auto-compaction
- Lot of usability improvements to the web console

Source and binary distributions can be downloaded from:
https://druid.apache.org/downloads.html

Release notes are at:
https://github.com/apache/druid/releases/tag/druid-0.23.0

A big thank you to all the contributors to this milestone release!

Re: [VOTE] Release Apache Druid 0.23.0 [RC2]

[Abhishek Agarwal]

Thank you, everyone. The vote has passed. You can see the results at
https://lists.apache.org/thread/g8l979xdp0ovt7f0j4sdomkrkolq99k0

On Mon, Jun 20, 2022 at 10:41 AM Vadim Ogievetsky 
wrote:

> +1 (binding)
>
> Source:
> - Did a build (the one the console does as part of CI)
>
> Binary:
> - Run the quickstart
> - Played with the depth of the console
> - Ran some queries
>
> On 2022/06/15 14:22:23 Abhishek Agarwal wrote:
> > Hi all,
> > I have created a build for Apache Druid 0.23.0, release candidate 2.
> >
> > Thanks to everyone who has helped contribute to the release. You can read
> > the draft release notes here:
> > https://github.com/apache/druid/issues/12510
> >
> > The release candidate has been tagged in GitHub as druid-0.23.0-rc2
> > (c085dd1a5e3db02806f1fc8f54fd7f9297ee0f77), available here:
> > https://github.com/apache/druid/tree/druid-0.23.0-rc2
> >
> > The artifacts to be voted on are located here:
> > https://dist.apache.org/repos/dist/dev/druid/0.23.0-rc2/
> >
> > A staged Maven repository is available for review at:
> > https://repository.apache.org/content/repositories/orgapachedruid-1030
> >
> > Staged druid.apache.org website documentation is available here:
> > https://druid.staged.apache.org/docs/0.23.0/design/index.html
> >
> > A Docker image containing the binary of the release candidate can be
> > retrieved via:
> > docker pull apache/druid:0.23.0-rc2
> >
> > artifact checksums
> > src:
> >
> 50ff149fa274ac10c01040dc30bbf28fc24732da5a27f1b88f89b13bbee5c9ec8a68f5d57e7caabc5813cac5f551e1c4ca94dc8faa9c6d8efc022f3bd3bef7e3
> >
> > bin:
> >
> 5e7420bdc66247f0309fc0c1eafeff7ba935ed195f3a28c2da4b23a28ff637300b46b1c3d048379ccd570dde164968494ff5d8847e82eaf9b0edb518f8665328
> >
> > docker (sha256):
> > ed9719968b4be3a2f1643907a1a5a5814e2ee1215f3604da8b2da2ac05f65912
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/abhishek
> > This key and the key of other committers can also be found in the
> project's
> > KEYS file here:
> > https://dist.apache.org/repos/dist/release/druid/KEYS
> >
> > (If you are a committer, please feel free to add your own key to that
> file
> > by following the instructions in the file's header.)
> >
> > Verify checksums:
> > diff <(shasum -a512 apache-druid-0.23.0-src.tar.gz | cut -d ' ' -f1)
> <(cat
> > apache-druid-0.23.0-src.tar.gz.sha512 ; echo)
> >
> > diff <(shasum -a512 apache-druid-0.23.0-bin.tar.gz | cut -d ' ' -f1)
> <(cat
> > apache-druid-0.23.0-bin.tar.gz.sha512 ; echo)
> >
> > Verify signatures:
> > gpg --verify apache-druid-0.23.0-src.tar.gz.asc
> > apache-druid-0.23.0-src.tar.gz
> >
> > gpg --verify apache-druid-0.23.0-bin.tar.gz.asc
> > apache-druid-0.23.0-bin.tar.gz
> >
> > Please review the proposed artifacts and vote. Note that Apache has
> > specific requirements that must be met before +1 binding votes can be
> cast
> > by PMC members. Please refer to the policy at
> > http://www.apache.org/legal/release-policy.html#policy for more details.
> >
> > As part of the validation process, the release artifacts can be generated
> > from the source by running:
> > mvn clean install -Papache-release,dist -Dgpg.skip
> >
> > The RAT license check can be run from the source by:
> > mvn apache-rat:check -Prat
> >
> > This vote will be open for at least 72 hours. The vote will pass if a
> > majority of at least three +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Druid 0.23.0
> > [ ] 0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> > Thanks!
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org
>
>

[RESULT] [VOTE] Release Apache Druid 0.23.0 [RC2]

[Abhishek Agarwal]

Hello,

Thanks to everyone who participated in the vote! The results are as follows:

Atul Mohan: +1 (binding)
Clint Wylie: +1 (binding)
Jonathan Wei: +1 (binding)
Vadim Ogievetsky: +1 (binding)

The vote has passed with 4 binding +1s.

Re: new committer: Victoria Lim

[Abhishek Agarwal]

Congratulations Victoria.

On Wed, 15 Jun 2022 at 8:29 PM, Charles Smith 
wrote:

> Cheers, Victoria! 👏
>
> On Wed, Jun 15, 2022 at 2:04 AM Kashif Faraz  wrote:
>
> > Congrats, Victoria! 🎉🎉
> >
> > On Wed, Jun 15, 2022 at 9:07 AM suneet Saldanha 
> wrote:
> >
> > > Hey Druids,
> > >
> > > The Druid PMC has invited Victoria Lim (@vtlim on GitHub)
> > > to become a committer and we are pleased to announce that
> > > Victoria has accepted.
> > >
> > > Victoria has made major improvements to the docs like refactoring the
> > > sql query docs, adding docs for cluster tiering and refactoring the
> > > compaction
> > > and auto-compaction docs.
> > >
> > > Congratulations Victoria!
> > >
> >
>

[VOTE] Release Apache Druid 0.23.0 [RC2]

[Abhishek Agarwal]

Hi all,
I have created a build for Apache Druid 0.23.0, release candidate 2.

Thanks to everyone who has helped contribute to the release. You can read
the draft release notes here:
https://github.com/apache/druid/issues/12510

The release candidate has been tagged in GitHub as druid-0.23.0-rc2
(c085dd1a5e3db02806f1fc8f54fd7f9297ee0f77), available here:
https://github.com/apache/druid/tree/druid-0.23.0-rc2

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/druid/0.23.0-rc2/

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachedruid-1030

Staged druid.apache.org website documentation is available here:
https://druid.staged.apache.org/docs/0.23.0/design/index.html

A Docker image containing the binary of the release candidate can be
retrieved via:
docker pull apache/druid:0.23.0-rc2

artifact checksums
src:
50ff149fa274ac10c01040dc30bbf28fc24732da5a27f1b88f89b13bbee5c9ec8a68f5d57e7caabc5813cac5f551e1c4ca94dc8faa9c6d8efc022f3bd3bef7e3

bin:
5e7420bdc66247f0309fc0c1eafeff7ba935ed195f3a28c2da4b23a28ff637300b46b1c3d048379ccd570dde164968494ff5d8847e82eaf9b0edb518f8665328

docker (sha256):
ed9719968b4be3a2f1643907a1a5a5814e2ee1215f3604da8b2da2ac05f65912

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/abhishek
This key and the key of other committers can also be found in the project's
KEYS file here:
https://dist.apache.org/repos/dist/release/druid/KEYS

(If you are a committer, please feel free to add your own key to that file
by following the instructions in the file's header.)

Verify checksums:
diff <(shasum -a512 apache-druid-0.23.0-src.tar.gz | cut -d ' ' -f1) <(cat
apache-druid-0.23.0-src.tar.gz.sha512 ; echo)

diff <(shasum -a512 apache-druid-0.23.0-bin.tar.gz | cut -d ' ' -f1) <(cat
apache-druid-0.23.0-bin.tar.gz.sha512 ; echo)

Verify signatures:
gpg --verify apache-druid-0.23.0-src.tar.gz.asc
apache-druid-0.23.0-src.tar.gz

gpg --verify apache-druid-0.23.0-bin.tar.gz.asc
apache-druid-0.23.0-bin.tar.gz

Please review the proposed artifacts and vote. Note that Apache has
specific requirements that must be met before +1 binding votes can be cast
by PMC members. Please refer to the policy at
http://www.apache.org/legal/release-policy.html#policy for more details.

As part of the validation process, the release artifacts can be generated
from the source by running:
mvn clean install -Papache-release,dist -Dgpg.skip

The RAT license check can be run from the source by:
mvn apache-rat:check -Prat

This vote will be open for at least 72 hours. The vote will pass if a
majority of at least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Druid 0.23.0
[ ] 0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...

Thanks!

[CANCEL] [VOTE] Release Apache Druid 0.23.0 [RC1]

[Abhishek Agarwal]

Canceling this vote due to version issues and extra files. I will fix these
and start another vote.

On Thu, Jun 9, 2022 at 3:33 PM Clint Wylie  wrote:

> -1 (binding)
> The RAT check, mvn apache-rat:check -Prat, fails for me. it looks like
> there are some stray files in the source distribution:
>
> extensions-core/kubernetes-extensions/.attach_pid19113
> extensions-core/druid-ranger-security/.attach_pid15985
>
> possibly due to running tests in the packaging place, which looks like
> it might have something to do with these files when running tests
> perhaps? At least this is my best guess stumbling upon
>
> https://github.com/raphw/byte-buddy/blob/master/byte-buddy-agent/src/main/java/net/bytebuddy/agent/VirtualMachine.java#L480
> ,
> since mockito uses byte-buddy to attach to process for certain types
> of tests. This might be an issue that is new to this release, since I
> think we migrated some tests recently?
>
> This, coupled with the wrong date in NOTICE, the docker-compose
> version, and web-console docs links issues I think make it worth doing
> an RC2.
>
> On Wed, Jun 8, 2022 at 9:43 AM Atul Mohan  wrote:
> >
> > In the process of verifying the release, it looks like the
> > docker-compose.yml that we package is pointing to 0.22.0. I've raised
> > https://github.com/apache/druid/pull/12618 to fix this.
> >
> > Thanks,
> > Atul
> >
> > On Wed, Jun 8, 2022 at 7:25 AM Kashif Faraz  wrote:
> >
> > > +1
> > >
> > > verified:
> > > - gpg signature
> > > - shasum
> > > - building the source artifacts (tests skipped)
> > > - rat license check
> > > - running the binary
> > > - basic sanity ingestion and query
> > >
> > > Thanks
> > > Kashif Faraz
> > > kfa...@apache.org
> > >
> > >
> > > On Mon, Jun 6, 2022 at 1:00 PM Abhishek Agarwal <
> abhishek.agar...@imply.io
> > > >
> > > wrote:
> > >
> > > > Hi all,
> > > > I have created a build for Apache Druid 0.23.0, release candidate 1.
> > > >
> > > > Thanks to everyone who has helped contribute to the release. You can
> read
> > > > the draft release notes here:
> > > > https://github.com/apache/druid/issues/12510
> > > >
> > > > The release candidate has been tagged in GitHub as druid-0.23.0-rc1
> > > > (efa67c4eddb31dc6e9ff7d0c070b9d1b1bf10a45), available here:
> > > > https://github.com/apache/druid/tree/druid-0.23.0-rc1
> > > >
> > > > The artifacts to be voted on are located here:
> > > > https://dist.apache.org/repos/dist/dev/druid/0.23.0-rc1/
> > > >
> > > > A staged Maven repository is available for review at:
> > > >
> https://repository.apache.org/content/repositories/orgapachedruid-1029/
> > > >
> > > > Staged druid.apache.org website documentation is available here:
> > > > https://druid.staged.apache.org/docs/0.23.0/design/index.html
> > > >
> > > > A Docker image containing the binary of the release candidate can be
> > > > retrieved via:
> > > > docker pull apache/druid:0.23.0-rc1
> > > >
> > > > artifact checksums
> > > > src:
> > > >
> > > >
> > >
> a8b8124297f4fb954d8f3cb79cf2ea867a689fe8d8b2470718d0d95badcd9e66d3658efcb064c50617b5ea4cbfbc3e650b01d742de7bc7b39050b1ba5da6b068
> > > >
> > > > bin:
> > > >
> > > >
> > >
> da1e0703db699fa523b7fa86b57eec5f77aa0b4141bee44324d81d5c8058a414bc896145f3dd7f5e12320593055e20b551ce597628de133a062d091e40c38a54
> > > >
> > > > docker (sha256):
> > > > 510c9be199d6b8568c62c6d4bcf1bb3c98690baf9521eccf8c6b335e90eeef97
> > > >
> > > > Release artifacts are signed with the following key:
> > > > https://people.apache.org/keys/committer/abhishek
> > > > This key and the key of other committers can also be found in the
> > > project's
> > > > KEYS file here:
> > > > https://dist.apache.org/repos/dist/release/druid/KEYS
> > > >
> > > > (If you are a committer, please feel free to add your own key to that
> > > file
> > > > by following the instructions in the file's header.)
> > > >
> > > > Verify checksums:
> > > > diff <(shasum -a512 apache-druid-0.23.0-src.tar.gz | cut -d ' ' -f1)
> > > <(cat
> > > > apache-druid-0.23.0-src.tar.gz.sha512 ; echo)
> > 

[VOTE] Release Apache Druid 0.23.0 [RC1]

[Abhishek Agarwal]

Hi all,
I have created a build for Apache Druid 0.23.0, release candidate 1.

Thanks to everyone who has helped contribute to the release. You can read
the draft release notes here:
https://github.com/apache/druid/issues/12510

The release candidate has been tagged in GitHub as druid-0.23.0-rc1
(efa67c4eddb31dc6e9ff7d0c070b9d1b1bf10a45), available here:
https://github.com/apache/druid/tree/druid-0.23.0-rc1

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/druid/0.23.0-rc1/

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachedruid-1029/

Staged druid.apache.org website documentation is available here:
https://druid.staged.apache.org/docs/0.23.0/design/index.html

A Docker image containing the binary of the release candidate can be
retrieved via:
docker pull apache/druid:0.23.0-rc1

artifact checksums
src:
a8b8124297f4fb954d8f3cb79cf2ea867a689fe8d8b2470718d0d95badcd9e66d3658efcb064c50617b5ea4cbfbc3e650b01d742de7bc7b39050b1ba5da6b068

bin:
da1e0703db699fa523b7fa86b57eec5f77aa0b4141bee44324d81d5c8058a414bc896145f3dd7f5e12320593055e20b551ce597628de133a062d091e40c38a54

docker (sha256):
510c9be199d6b8568c62c6d4bcf1bb3c98690baf9521eccf8c6b335e90eeef97

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/abhishek
This key and the key of other committers can also be found in the project's
KEYS file here:
https://dist.apache.org/repos/dist/release/druid/KEYS

(If you are a committer, please feel free to add your own key to that file
by following the instructions in the file's header.)

Verify checksums:
diff <(shasum -a512 apache-druid-0.23.0-src.tar.gz | cut -d ' ' -f1) <(cat
apache-druid-0.23.0-src.tar.gz.sha512 ; echo)

diff <(shasum -a512 apache-druid-0.23.0-bin.tar.gz | cut -d ' ' -f1) <(cat
apache-druid-0.23.0-bin.tar.gz.sha512 ; echo)

Verify signatures:
gpg --verify apache-druid-0.23.0-src.tar.gz.asc \
apache-druid-0.23.0-src.tar.gz

gpg --verify apache-druid-0.23.0-bin.tar.gz.asc \
apache-druid-0.23.0-bin.tar.gz

Please review the proposed artifacts and vote. Note that Apache has
specific requirements that must be met before +1 binding votes can be cast
by PMC members. Please refer to the policy at
http://www.apache.org/legal/release-policy.html#policy for more details.

As part of the validation process, the release artifacts can be generated
from source by running:
mvn clean install -Papache-release,dist -Dgpg.skip

The RAT license check can be run from source by:
mvn apache-rat:check -Prat

This vote will be open for at least 72 hours. The vote will pass if a
majority of at least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Druid 0.23.0
[ ] 0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...

Thanks!

Re: [DISCUSS] Druid 0.23 release

[Abhishek Agarwal]

Thank you, Suneet.

On Fri, May 27, 2022 at 9:46 PM Suneet Saldanha  wrote:

> Agreed. Naming is hard, and I wouldn't want to slow down this release
> trying to get
> consensus on the correct versioning scheme. They all seem to have merits.
> I will start
> another thread to discuss this so we will have a decision by the time the
> next release
> rolls around.
>
> On 2022/05/27 02:57:39 rahul gidwani wrote:
> > What about a 1.0 release?  I think there is no backwards compatibility
> promised until Druid gets to 1.0+.  I think it would be really helpful to
> customers to start making upgrades rollable and guaranteeing compatibility
> between minor versions.  Any plans for this to happen in the near future?
> >
> > > On May 26, 2022, at 7:16 PM, Gian Merlino  wrote:
> > >
> > > I'm supportive of changing the versioning to something without the
> leading
> > > zero in the next release where this is practical. If it's the one after
> > > 0.23.0, then I would go with 24.0. IMO, going with 1.0 would send a
> message
> > > that this is the first mature release. But that isn't the case: we have
> > > been doing mature releases for a long time now. Going with 24.0 is
> clearer
> > > in that regard.
> > >
> > > Happy to repeat this opinion on a new thread too :)
> > >
> > >> On Thu, May 26, 2022 at 6:49 PM Frank Chen 
> wrote:
> > >>
> > >> For 0.23, I don't think we need to make changes because I think it
> may take
> > >> us some time to reach an agreement on the naming.
> > >>
> > >> We can start a new thread to discuss the versioning schema.
> > >>
> > >>
> > >> On Thu, May 26, 2022 at 8:19 PM Abhishek Agarwal <
> > >> abhishek.agar...@imply.io>
> > >> wrote:
> > >>
> > >>> We should definitely move away from the `0.xx` versioning scheme we
> have
> > >>> been using. However, the next version that we pick up is debatable.
> > >> `23.x`
> > >>> seems an odd jump from `0.23`. Can we increment the version to `1.x`
> > >> maybe?
> > >>> I also like the idea of using Yeah and Month that Frank has
> suggested.
> > >>>
> > >>> I don't think that 0.23 is the right release to make this change
> though.
> > >>> 0.23 has already been delayed because of CVE investigations and bug
> > >> fixes.
> > >>> I would like to get this release out of the door as soon as possible.
> > >>>
> > >>>> On Thu, May 26, 2022 at 2:40 PM Frank Chen 
> wrote:
> > >>>
> > >>>> I agree.
> > >>>>
> > >>>> This is also a question that I want to ask why the version is still
> > >> 0.xx
> > >>>> which gives many people a hint that Druid is still under mature.
> > >>>>
> > >>>> There are many versioning schemas. One popular way is combining the
> > >>> release
> > >>>> year and month in the version.
> > >>>> For example, if we're going to release a version in May this year,
> the
> > >>> main
> > >>>> version can be 22.5.
> > >>>>
> > >>>>
> > >>>> Versioning is one thing, LTS strategy should also be clear.
> > >>>> Since we're going to release several versions a year, we should
> plan in
> > >>>> advance which one should be scheduled as a LTS version and maintain
> it
> > >>> for
> > >>>> a period of time if there are some vital bugs and security issues.
> > >>>>
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Thu, May 26, 2022 at 1:21 PM Suneet Saldanha 
> > >>> wrote:
> > >>>>
> > >>>>> Hi all,
> > >>>>>
> > >>>>> I've been thinking that we should consider re-branding this release
> > >> as
> > >>>>> the Druid 23.0 instead of 0.23 release. I think this is appropriate
> > >>>> because
> > >>>>> typically a `0.XX` software version implies that the software is in
> > >>> it's
> > >>>>> infancy.
> > >>>>>
> > >>>>> Druid is quite mature, and we've been putting good guardrails in
> > >> place
> > >>&

Re: [DISCUSS] Druid 0.23 release

[Abhishek Agarwal]

We should definitely move away from the `0.xx` versioning scheme we have
been using. However, the next version that we pick up is debatable. `23.x`
seems an odd jump from `0.23`. Can we increment the version to `1.x` maybe?
I also like the idea of using Yeah and Month that Frank has suggested.

I don't think that 0.23 is the right release to make this change though.
0.23 has already been delayed because of CVE investigations and bug fixes.
I would like to get this release out of the door as soon as possible.

On Thu, May 26, 2022 at 2:40 PM Frank Chen  wrote:

> I agree.
>
> This is also a question that I want to ask why the version is still 0.xx
> which gives many people a hint that Druid is still under mature.
>
> There are many versioning schemas. One popular way is combining the release
> year and month in the version.
> For example, if we're going to release a version in May this year, the main
> version can be 22.5.
>
>
> Versioning is one thing, LTS strategy should also be clear.
> Since we're going to release several versions a year, we should plan in
> advance which one should be scheduled as a LTS version and maintain it for
> a period of time if there are some vital bugs and security issues.
>
>
>
>
> On Thu, May 26, 2022 at 1:21 PM Suneet Saldanha  wrote:
>
> > Hi all,
> >
> > I've been thinking that we should consider re-branding this release as
> > the Druid 23.0 instead of 0.23 release. I think this is appropriate
> because
> > typically a `0.XX` software version implies that the software is in it's
> > infancy.
> >
> > Druid is quite mature, and we've been putting good guardrails in place to
> > detect and prevent breaking API changes in each release. Druid has also
> > been running in production clusters for many different use cases for
> quite
> > some
> > time now. I think version 23.0 is more in line with the maturity of the
> > project.
> >
> > Is there a reason not to change the version for the next release? Any
> > other thoughts?
> >
> > On 2022/04/11 10:21:11 Abhishek Agarwal wrote:
> > > Thank you for creating that PR, Frank. In the last release, we excluded
> > > helm charts since we were not sure about IP clearance. From
> > > https://incubator.apache.org/ip-clearance/, we should decide on IP
> > > clearance whether we include helm charts in artifacts or not. Any
> > thoughts?
> > >
> > > On Wed, Mar 30, 2022 at 4:44 PM Frank Chen 
> wrote:
> > >
> > > > Hi Abhishek,
> > > >
> > > > Thank you for starting the release work.
> > > >
> > > > This PR should be merged to address a problem caused by a previous
> PR:
> > > > https://github.com/apache/druid/pull/12067
> > > > I've added it to the 0.23 milestone.
> > > >
> > > >
> > > > Thank you.
> > > >
> > > >
> > > > On Wed, Mar 30, 2022 at 2:15 PM Abhishek Agarwal <
> > > > abhishek.agar...@imply.io>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > > It's time to kick-off the process for druid 0.23 release. I will
> need
> > > > help
> > > > > from the community in surfacing any important issues that need to
> be
> > > > > addressed before 0.23 release. We can use this thread to discuss
> > those
> > > > > issues and take a call on how to unblock the release.
> > > > >
> > > > > I have also created 0.23 milestone (
> > > > > https://github.com/apache/druid/milestone/45). Any issues that we
> > must
> > > > > want
> > > > > to fix in the 0.23 release, can be tagged with this milestone.
> > > > >
> > > >
> > >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > For additional commands, e-mail: dev-h...@druid.apache.org
> >
> >
>

Re: [DISCUSS] Druid 0.23 release

[Abhishek Agarwal]

Thank you for creating that PR, Frank. In the last release, we excluded
helm charts since we were not sure about IP clearance. From
https://incubator.apache.org/ip-clearance/, we should decide on IP
clearance whether we include helm charts in artifacts or not. Any thoughts?

On Wed, Mar 30, 2022 at 4:44 PM Frank Chen  wrote:

> Hi Abhishek,
>
> Thank you for starting the release work.
>
> This PR should be merged to address a problem caused by a previous PR:
> https://github.com/apache/druid/pull/12067
> I've added it to the 0.23 milestone.
>
>
> Thank you.
>
>
> On Wed, Mar 30, 2022 at 2:15 PM Abhishek Agarwal <
> abhishek.agar...@imply.io>
> wrote:
>
> > Hello everyone,
> > It's time to kick-off the process for druid 0.23 release. I will need
> help
> > from the community in surfacing any important issues that need to be
> > addressed before 0.23 release. We can use this thread to discuss those
> > issues and take a call on how to unblock the release.
> >
> > I have also created 0.23 milestone (
> > https://github.com/apache/druid/milestone/45). Any issues that we must
> > want
> > to fix in the 0.23 release, can be tagged with this milestone.
> >
>

Re: Re: 0.23

[Abhishek Agarwal]

Hi Baris
I don't see an attachment in your email.

On Sat, Apr 9, 2022 at 5:38 AM Baris Dogan  wrote:

> Hi,
>
> Could you just put the following CVE findings (attached into the email)
> into the next release scope which come out of trivy 0.23 tests.
>
> On 2022/03/25 03:52:50 Gian Merlino wrote:
> > I agree it's a good time to do a release. Most of the release-manager
> steps
> > involve having commit privileges, but nevertheless, you might find it
> > interesting to read about the process:
> >
> https://github.com/apache/druid/blob/master/distribution/asf-release-process-guide.md
> >
> > You've actually already done the first step: start a thread on the dev
> > list. The next part is we have a discussion & see if there's anything
> > critical we should get into this release before we branch it off.
> >
> > Anyone have any comments on that?
> >
> > On Wed, Mar 23, 2022 at 9:54 PM Eyal Yurman 
>
> > wrote:
> >
> > > Hi,
> > >
> > > Anyone plan to work on releasing 0.23?
> > > I'll be really glad to manage the 0.23 release but I'm not a
> committer.
> > > Assuming this requires committer privileges, I'd be glad if anyone can
> > > volunteer.
> > >
> > > BTW, have you noticed that we shifted away from the official quarterly
> > > releases? Perhaps we should discuss our release process. Especially
> since
> > > we also don't release minor versions (except for hotfixes immediately
> after
> > > a major release).
> > >
> >
> --
> Zeki Baris Dogan
> MSc. Software Engineer.
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org

[DISCUSS] Druid 0.23 release

[Abhishek Agarwal]

Hello everyone,
It's time to kick-off the process for druid 0.23 release. I will need help
from the community in surfacing any important issues that need to be
addressed before 0.23 release. We can use this thread to discuss those
issues and take a call on how to unblock the release.

I have also created 0.23 milestone (
https://github.com/apache/druid/milestone/45). Any issues that we must want
to fix in the 0.23 release, can be tagged with this milestone.

Re: 0.23

[Abhishek Agarwal]

Hi Eyal
Thanks for bringing this up. I am going to volunteer for shepherding 0.23
release. I will start a new thread soon.

On Fri, Mar 25, 2022 at 9:23 AM Gian Merlino  wrote:

> I agree it's a good time to do a release. Most of the release-manager steps
> involve having commit privileges, but nevertheless, you might find it
> interesting to read about the process:
>
> https://github.com/apache/druid/blob/master/distribution/asf-release-process-guide.md
>
> You've actually already done the first step: start a thread on the dev
> list. The next part is we have a discussion & see if there's anything
> critical we should get into this release before we branch it off.
>
> Anyone have any comments on that?
>
> On Wed, Mar 23, 2022 at 9:54 PM Eyal Yurman  .invalid>
> wrote:
>
> > Hi,
> >
> > Anyone plan to work on releasing 0.23?
> > I'll be really glad to manage the 0.23 release but I'm not a committer.
> > Assuming this requires committer privileges, I'd be glad if anyone can
> > volunteer.
> >
> > BTW, have you noticed that we shifted away from the official quarterly
> > releases? Perhaps we should discuss our release process. Especially since
> > we also don't release minor versions (except for hotfixes immediately
> after
> > a major release).
> >
>

slack vs druid user forum

[Abhishek Agarwal]

Hi
I created a PR https://github.com/apache/druid-website-src/pull/281 to mark
slack as a preferred medium to get help. I wanted to do this because I
usually see more active participation from developers on the slack channel
as opposed to the druid user forum. (Please correct me if that assumption
is wrong). Though discussion on user-forum gets indexed by search engines
unlike the content on slack. so slack has that disadvantage.

any thoughts on this matter?

Re: Apache Druid Slack

[Abhishek Agarwal]

+1
There is a lot of friction in joining apache infra slack. Also, I can't
tell if it is a recent policy change but any new invitation to the druid
channel now needs to be approved by ASF slack admin. That seems like one
more manual step to get over. Though is there a way to migrate/copy the
channel contents? There is a whole lot of history that we would want to
retain in the new slack channel.

On Fri, Jan 21, 2022 at 10:48 PM Gian Merlino  wrote:

> It sounds like a good idea to me. It's not ideal that the current Slack
> workspace is hard for new people to join.
>
> On Thu, Jan 20, 2022 at 10:15 AM Vadim Ogievetsky 
> wrote:
>
> > I think that the PMC should create a new Slack channel for Apache Druid
> and
> > shift the community towards using it away from the ASF Slack. I volunteer
> > to do this on the PMCs behalf and do all the setup/admin work. I would
> > share admin access with any PMC member.
> >
> > What is the motivation for this?
> >
> > As you may have heard, it’s become increasingly difficult for new users
> > without an @apache.org email address to join the ASF #druid Slack
> channel.
> > ASF Infra disabled the option to publicly provide a link to the workspace
> > to anyone who wanted it, after encountering issues with spammers.
> >
> > Per Infra’s guidance (https://infra.apache.org/slack.html), new
> community
> > members should only be invited as single-channel guests. Unfortunately,
> > single-channel guests are unable to extend invitations to new members,
> > including their colleagues who are using Druid. Only someone with full
> > member privileges is able to extend an invitation to new members. This
> lack
> > of consistency doesn’t make the community feel inclusive.
> >
> > There is a workaround in place (
> > https://github.com/apache/druid-website-src/pull/278) – users can send
> an
> > email to druid-u...@googlegroups.com to request an invite to the Slack
> > channel from an existing member – but this still poses a barrier to
> entry,
> > and isn’t a viable permanent solution. It also creates potential privacy
> > issues as not everyone is at liberty to announce they’re using Druid nor
> > wishes to display their email address in a public forum.
> >
> > I propose we make our own free Slack channel for Apache Druid and
> encourage
> > people to migrate to it. Then we can have our own policy on Slack
> > invitations - I would like to restore the ability for anyone on the web
> to
> > join our Slack.
> >
> > This is not a 100% original idea, in fact this is what other Apache
> > projects have done, notably Apache Pinot (see "join our slack" on
> > https://pinot.apache.org/). I propose we do the same.
> >
> > Vadim
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > For additional commands, e-mail: dev-h...@druid.apache.org
> >
> >
>

Re: new committer: Kashif Faraz

[Abhishek Agarwal]

Congratulations Kashif. You have done really great work, particularly on
the ingestion side. Looking forward to many more contributions in future.

On Thu, Jan 13, 2022 at 11:04 PM suneet Saldanha  wrote:

> Hey Druids,
>
> The Druid PMC has invited Kashif Faraz (@kfaraz on GitHub)
> to become a committer and we are pleased to announce that
> Kashif has accepted.
>
> Kashif has contributed several impactful features to Druid like support
> for range partitioning on multiple dimensions and the ability to route sql
> queries at the router.
>
> Congratulations Kashif!
>

Re: New PMC member: Frank Chen

[Abhishek Agarwal]

Congratulations Frank.

On Tue, Nov 2, 2021 at 2:44 AM Jihoon Son  wrote:

> Hey Druids,
>
> The Druid PMC has invited Frank Chen (@FrankChen021 on github) to
> become a PMC member and we are pleased to announce that he has
> accepted. Frank has been active in various parts of the Druid
> community including contributing features, fixing bugs, reviewing PRs,
> responding to user issues, releases, and so on.
>
> Congratulations Frank!
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org
>
>

Re: New PMC member: Atul Mohan

[Abhishek Agarwal]

Congratulations Atul.

On Tue, Nov 2, 2021 at 2:43 AM Jihoon Son  wrote:

> Hey Druids,
>
> The Druid PMC has invited Atul Mohan (@a2l007 on github) to become a
> PMC member and we are pleased to announce that he has accepted. Atul
> has been making substantial contributions in the Druid community, not
> only code contributions but also helping people in the user community.
>
> Congratulations Atul!
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org
>
>

Re: New committer: Agustin Gonzalez Tuchmann

[Abhishek Agarwal]

Congratulations Agustin. Keep up the good work.

On Tue, Nov 2, 2021 at 2:42 AM Jihoon Son  wrote:

> Hey Druids,
>
> The Druid PMC has invited Agustin Gonzalez Tuchmann (@loquisgon on
> github) to become a committer and we are pleased to announce that he
> has
> accepted. Agustin has made several significant contributions in the
> ingestion, including reducing memory pressure for batch ingestion.
>
> Congratulations Agustin!
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> For additional commands, e-mail: dev-h...@druid.apache.org
>
>

Re: Propose a scheme for Coordinator to pull metadata incrementally

[Abhishek Agarwal]

If an entry is deleted from the metadata, how is the coordinator going to
update its own state?

On Tue, Apr 6, 2021 at 3:38 PM Itai Yaffe  wrote:

> Hey,
> I'm not a Druid developer, so it's quite possible I'm missing many
> considerations here, but from a first glance, I like your offer, as it
> resembles the *tsColumn *in JDBC lookups (
>
> https://druid.apache.org/docs/latest/development/extensions-core/lookups-cached-global.html#jdbc-lookup
> ).
>
> Anyway, just my 2 cents.
>
> Thanks!
>   Itai
>
> On Tue, Apr 6, 2021 at 6:07 AM Benedict Jin  wrote:
>
> > Hi all,
> >
> > Recently, when the Coordinator in our company's Druid cluster pulls
> > metadata, there is a performance bottleneck. The main reason is the huge
> > amount of metadata, which leads to a very slow process of scanning the
> full
> > table of metadata storage and deserializing metadata. The size of the
> full
> > metadata has been reduced through TTL, Compaction, Rollup, and etc., but
> > the effect is not very significant. Therefore, I want to design a scheme
> > for Coordinator to pull metadata incrementally, that is, each time
> > Coordinator only pulls newly added metadata, so as to reduce the query
> > pressure of metadata storage and the pressure of deserializing metadata.
> > The general idea is to add a column last_update to the druid_segments
> table
> > to record the update time of each record. Furthermore, when we query the
> > metadata table, we can add filter conditions for the last_update column
> to
> > avoid full table scan operations. Moreover, whether it is MySQL or
> > PostgreSQL as the metadata storage medium, it can support
> >  automatic update of the timestamp field, which is somewhat similar to
> the
> > characteristics of triggers. So, have you encountered this problem
> before?
> > If so, how did you solve it? In addition, do you have any suggestions or
> > comments on the above incremental acquisition of metadata? Please let me
> > know, thanks a lot.
> >
> > Regards,
> > Benedict Jin
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > For additional commands, e-mail: dev-h...@druid.apache.org
> >
> >
>

Re: new committer: Abhishek Agarwal

[Abhishek Agarwal]

Thank you, everyone.

On Tue, Mar 30, 2021 at 1:37 PM Tijo Thomas  wrote:

> Congratulations  Abhishek
>
> On Tue, Mar 30, 2021 at 4:39 AM Suneet Saldanha 
> wrote:
>
> > Congratulations Abhishek! Thanks for your code reviews and cool
> > contributions like the GROUPING_ID function and caching for join queries
> >
> > On Tue, Mar 23, 2021 at 8:36 PM Jonathan Wei  wrote:
> >
> > > Hey Druids,
> > >
> > > The Druid PMC has invited Abhishek Agarwal (@abhishekagarwal87 on
> GitHub)
> > > to become a committer and we are pleased to announce that Abhishek has
> > > accepted.
> > >
> > > Abhishek has made several significant contributions in the deeper parts
> > of
> > > the ingestion
> > > and query systems and is also active in doing code reviews.
> > >
> > > Congratulations Abhishek!
> > >
> >
>
>
> --
> Tijo Thomas
> Solutions Architect  | => Imply , Bangalore , India
>