Re: [E] Re: Apache Druid Slack
What should we do about Slack canceling message history for free tier? Perhaps we should export history elsewhere? On Thu, Jan 27, 2022 at 1:10 PM Vadim Ogievetsky wrote: > Alright, I have setup the new Slack workspace and updated the link on > https://urldefense.proofpoint.com/v2/url?u=https-3A__druid.apache.org_community_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=hV9FPFkuEXDmMbTGo3m_iqUc-baVKjOHznj9kig1I1Q&e= > to point to it. > Please join with the > https://urldefense.proofpoint.com/v2/url?u=https-3A__druid.apache.org_community_join-2Dslack&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=07e2sPTwojKxi997mSN4a5Ktcr1lyHcekXuTwEZj9JY&e= > link (and share it with others). > Sadly there is no way to migrate content from one Slack workspace to > workspace, I am sure we will fill the new channel with great content very > quickly! > > On 2022/01/20 18:15:23 Vadim Ogievetsky wrote: > > I think that the PMC should create a new Slack channel for Apache Druid > and > > shift the community towards using it away from the ASF Slack. I volunteer > > to do this on the PMCs behalf and do all the setup/admin work. I would > > share admin access with any PMC member. > > > > What is the motivation for this? > > > > As you may have heard, it’s become increasingly difficult for new users > > without an @apache.org email address to join the ASF #druid Slack > channel. > > ASF Infra disabled the option to publicly provide a link to the workspace > > to anyone who wanted it, after encountering issues with spammers. > > > > Per Infra’s guidance ( > https://urldefense.proofpoint.com/v2/url?u=https-3A__infra.apache.org_slack.html&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=qZfDyv1d4QnrCrEjvTgSbm8qeEj-B-dFWKil2k4XOyQ&e= > ), new community > > members should only be invited as single-channel guests. Unfortunately, > > single-channel guests are unable to extend invitations to new members, > > including their colleagues who are using Druid. Only someone with full > > member privileges is able to extend an invitation to new members. This > lack > > of consistency doesn’t make the community feel inclusive. > > > > There is a workaround in place ( > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid-2Dwebsite-2Dsrc_pull_278&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=LZAv4yoEkFJcPmJckcAR0xXyuo7V81ZfR2X_jxQjH58&e= > ) – users can send an > > email to druid-u...@googlegroups.com to request an invite to the Slack > > channel from an existing member – but this still poses a barrier to > entry, > > and isn’t a viable permanent solution. It also creates potential privacy > > issues as not everyone is at liberty to announce they’re using Druid nor > > wishes to display their email address in a public forum. > > > > I propose we make our own free Slack channel for Apache Druid and > encourage > > people to migrate to it. Then we can have our own policy on Slack > > invitations - I would like to restore the ability for anyone on the web > to > > join our Slack. > > > > This is not a 100% original idea, in fact this is what other Apache > > projects have done, notably Apache Pinot (see "join our slack" on > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__pinot.apache.org_&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=XsVtrQ1n66sQ9WUOQT8K162x-6-hXEAYJwF2JdxJbMU&m=adK2ZRhIznXu2_uvV3MLkV-rmdB2gTTJbxUdYqZHQ_DabAl8uQCbiEDLwBUVdYxF&s=GYTpyHD-1U_Er2eKNXk41b9aiy4zW9EtCt-2YXxSh9o&e= > ). I propose we do the same. > > > > Vadim > > > > - > > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > > For additional commands, e-mail: dev-h...@druid.apache.org > > > > > > - > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > >
Re: [E] Re: 0.23
Thank you! Much appreciated. On Mon, Mar 28, 2022 at 7:58 AM Abhishek Agarwal wrote: > Hi Eyal > Thanks for bringing this up. I am going to volunteer for shepherding 0.23 > release. I will start a new thread soon. > > On Fri, Mar 25, 2022 at 9:23 AM Gian Merlino wrote: > >> I agree it's a good time to do a release. Most of the release-manager >> steps >> involve having commit privileges, but nevertheless, you might find it >> interesting to read about the process: >> >> https://github.com/apache/druid/blob/master/distribution/asf-release-process-guide.md >> <https://urldefense.com/v3/__https://github.com/apache/druid/blob/master/distribution/asf-release-process-guide.md__;!!Op6eflyXZCqGR5I!Ua4IHq98uxkm8ND-RNYibHHdDTAej5ujw42RkLv6V-H5gUjkZnXGqXdllhGdqL4$> >> >> You've actually already done the first step: start a thread on the dev >> list. The next part is we have a discussion & see if there's anything >> critical we should get into this release before we branch it off. >> >> Anyone have any comments on that? >> >> On Wed, Mar 23, 2022 at 9:54 PM Eyal Yurman > .invalid> >> wrote: >> >> > Hi, >> > >> > Anyone plan to work on releasing 0.23? >> > I'll be really glad to manage the 0.23 release but I'm not a committer. >> > Assuming this requires committer privileges, I'd be glad if anyone can >> > volunteer. >> > >> > BTW, have you noticed that we shifted away from the official quarterly >> > releases? Perhaps we should discuss our release process. Especially >> since >> > we also don't release minor versions (except for hotfixes immediately >> after >> > a major release). >> > >> >
0.23
Hi, Anyone plan to work on releasing 0.23? I'll be really glad to manage the 0.23 release but I'm not a committer. Assuming this requires committer privileges, I'd be glad if anyone can volunteer. BTW, have you noticed that we shifted away from the official quarterly releases? Perhaps we should discuss our release process. Especially since we also don't release minor versions (except for hotfixes immediately after a major release).
Re: [E] Re: Log4j vulnerability - hotfix?
Thank you for the fast response.
On Fri, Dec 10, 2021 at 11:35 AM Gian Merlino wrote:
> We're working on this right now and will be getting a vote / release for
> 0.22.1 out asap.
>
> Btw, the log4j announcement mentions a mitigation that does work for our
> current version (2.8.2). It's part (b) here, specifying "%m{nolookups}" in
> the PatternLayout configuration:
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.apache.org_thread_bfnl1stql187jytr0t5k0hv0go6b76g4&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=SuBO953fsmU44ZHE0kwkYBIV-hvc-I5wqLvTFRA4RyA&m=gSJrS9K_MvHCpvyQVGe6FxHPFR1dN56YiTbzDAVkNqc&s=qpDSWATFc6fc441gjKMF6hgdvcZJKCTNltN5EolGlwo&e=
> . However,
> I haven't personally tested this, so I cannot provide any more details
> beyond pointing to the announcement.
>
> On Fri, Dec 10, 2021 at 10:27 AM Lucas Capistrant <
> capistrant.lu...@gmail.com> wrote:
>
> > Since it is “critical” severity, I think it would be a good idea to
> > seriously consider pushing out a minor version of 0.22.x. Especially
> since
> > the mitigation strategy outlined in the CVE is not available in the log4j
> > version that exists today in the current stable release. There is past
> > precedent for such releases: see 0.20.2
> >
> > On Fri, Dec 10, 2021 at 12:14 PM Eyal Yurman > .invalid>
> > wrote:
> >
> > > Hello, regarding
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_druid_pull_12051&d=DwIFaQ&c=sWW_bEwW_mLyN3Kx2v57Q8e-CRbmiT9yOhqES_g_wVY&r=SuBO953fsmU44ZHE0kwkYBIV-hvc-I5wqLvTFRA4RyA&m=gSJrS9K_MvHCpvyQVGe6FxHPFR1dN56YiTbzDAVkNqc&s=WGWSrb1gDnt3pidi5VPE0ibY1jS_KC7J9n56Bm7YPOU&e=
> which merged
> > > to
> > > master,
> > >
> > > Is it a common practice for the project to backport and release a new
> > minor
> > > for the latest version?
> > >
> >
>
Log4j vulnerability - hotfix?
Hello, regarding https://github.com/apache/druid/pull/12051 which merged to master, Is it a common practice for the project to backport and release a new minor for the latest version?
Re: [E] New committer: Atul Mohan
Congratulations Atul!! On Wed, Sep 2, 2020 at 2:28 PM Gian Merlino wrote: > Hey Druids, > > > > The Druid PMC has invited Atul Mohan (@a2l007 < > https://urldefense.com/v3/__https://github.com/a2l007__;!!Op6eflyXZCqGR5I!Tqf7cWMahjsInOtN155BveNwJ3l6aLmes5sdwqR9XDMZ5AJSzc1pM-3GWGXdOg$ > > > > on github) to become a committer and we are pleased to announce that he has > > accepted. Atul has been actively working on various parts of Druid, > > including indexing from SQL sources and result-level caching. > > > > Congratulations Atul! > >
Re: Moving Average error
Hi, I am sorry for the late response. Gian is correct, this extension is
restricted to the broker service and will throw an error in the log for
other services.
(As mentioned here:
https://github.com/apache/druid/blob/master/docs/development/extensions-contrib/moving-average-query.md#operations
).
On Wed, Apr 29, 2020 at 10:25 PM Gian Merlino wrote:
> Hey Damiano,
>
> This is a contrib extension so you might get limited support for it here.
>
> That being said, at first, I suggest looking through the logs mentioned by
> the supervise errors (like
> /home/damiano/druid/0.17.1/var/sv/coordinator-overlord.log). IIRC you might
> need to disable the extension on those server types. You could do it by
> putting a separate druid.extensions.loadList in the server-type-specific
> runtime properties.
>
> On Mon, Apr 20, 2020 at 4:12 AM Damiano Porta
> wrote:
>
> > Hello,
> >
> > I am trying to use the window-function extension
> > (druid-moving-average-query), but i am getting the following error:
> > (using 0.17.1 ver.)
> >
> > [Mon Apr 20 13:27:21 2020] Running command[broker], logging
> > to[/home/damiano/druid/0.17.1/var/sv/broker.log]: bin/run-druid broker
> > conf/druid/single-server/micro-quickstart
> > [Mon Apr 20 13:27:21 2020] Running command[router], logging
> > to[/home/damiano/druid/0.17.1/var/sv/router.log]: bin/run-druid router
> > conf/druid/single-server/micro-quickstart
> > [Mon Apr 20 13:27:21 2020] Running command[historical], logging
> > to[/home/damiano/druid/0.17.1/var/sv/historical.log]: bin/run-druid
> > historical conf/druid/single-server/micro-quickstart
> > [Mon Apr 20 13:27:21 2020] Running command[middleManager], logging
> > to[/home/damiano/druid/0.17.1/var/sv/middleManager.log]: bin/run-druid
> > middleManager conf/druid/single-server/micro-quickstart
> > [Mon Apr 20 13:27:27 2020] Command[coordinator-overlord] exited (pid =
> > 6734, exited = 1)
> > [Mon Apr 20 13:27:27 2020] Command[coordinator-overlord] failed, see
> > logfile for more details:
> > /home/damiano/druid/0.17.1/var/sv/coordinator-overlord.log
> > [Mon Apr 20 13:27:27 2020] Command[router] exited (pid = 6736, exited =
> 1)
> > [Mon Apr 20 13:27:27 2020] Command[router] failed, see logfile for more
> > details: /home/damiano/druid/0.17.1/var/sv/router.log
> > [Mon Apr 20 13:27:27 2020] Command[middleManager] exited (pid = 6738,
> > exited = 1)
> > [Mon Apr 20 13:27:27 2020] Command[middleManager] failed, see logfile
> > for more details: /home/damiano/druid/0.17.1/var/sv/middleManager.log
> > [Mon Apr 20 13:27:30 2020] Running command[coordinator-overlord],
> > logging to[/home/damiano/druid/0.17.1/var/sv/coordinator-overlord.log]:
> > bin/run-druid coordinator-overlord
> > conf/druid/single-server/micro-quickstart
> >
> > I have documents with this format:
> >
> > {"stra_id": long, "amount": long}
> >
> > i would like to aggregate such documents by doing a cumulative sum
> > (running sum) over the documents instead of just getting the value of
> > the total absolute sum?
> >
> > Is it possible?
> >
> > Thank you in advance
> >
> > Damiano
> >
> >
> >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org
> > For additional commands, e-mail: dev-h...@druid.apache.org
> >
> >
>
Re: 0.15.1 and 0.16.0
Hi, is there still time to include https://github.com/apache/incubator-druid/pull/8192 in 0.15.1? It fixes the extension released in 0.15.0 and is a rather minor change otherwise. On Wed, Jul 24, 2019 at 12:43 PM Prashant Deva wrote: > i wonder if rolling upgrading from 0.15.0 to 0.15.1 should first update > coordinator before other nodes. > if not, then you will run into 8137 during the upgrade. > > Prashant > > > On Wed, Jul 24, 2019 at 12:04 PM Jihoon Son wrote: > > > +1 for 0.15.1 > > > > The list of PRs looks good to me. > > Thank you for volunteering! > > > > Jihoon > > > > On Tue, Jul 23, 2019 at 3:38 PM Clint Wylie wrote: > > > > > Hi all, > > > > > > What appears to be a pretty brutal bug was reported earlier today, > > > https://github.com/apache/incubator-druid/issues/8137, that I think > > > warrants a 0.15.1 release. I was already planning on starting a thread > > > about volunteering to make a branch and release manage 0.16.0 sometime > in > > > the next week or so, but given this bug that popped up, I will go ahead > > and > > > volunteer to do both a 0.15.1 release as soon as the fix for that bug > is > > > merged, and 0.16.0 sometime in the next couple of weeks. Having a > 0.15.1 > > > release will give us a chance to fix a few other bugs while we are at > it, > > > on a much quicker scale than we could expect to go straight to trying > to > > > push out 0.16.0. > > > > > > I've picked what I think are some important bug fixes to put in 0.15.1 > > > (assuming they all backport to the 0.15.0-incubating branch cleanly): > > > > > > https://github.com/apache/incubator-druid/pull/7666 > sketches-core-0.13.4 > > > https://github.com/apache/incubator-druid/pull/7719 add bloom filter > > > fallback aggregator when types are unknown > > > https://github.com/apache/incubator-druid/pull/7823 Fix timestamp ceil > > > lower bound > > > https://github.com/apache/incubator-druid/pull/7827 discard filter > when > > > processing subtotalsSpec > > > https://github.com/apache/incubator-druid/pull/7917 WorkerTaskManager > to > > > create disk files atomically and ignore task file corruption > > > https://github.com/apache/incubator-druid/pull/8013 Fix > > > ExpressionVirtualColumn capabilities; fix groupBy's improper uses of > > > StorageAdapter#getColumnCapabilities. > > > https://github.com/apache/incubator-druid/pull/8026 set > > > DRUID_AUTHORIZATION_CHECKED attribute for router endpoints > > > https://github.com/apache/incubator-druid/pull/8044 SupervisorManager: > > Add > > > authorization checks to bulk endpoints. > > > https://github.com/apache/incubator-druid/pull/8055 force native order > > > when > > > wrapping ByteBuffer > > > https://github.com/apache/incubator-druid/pull/8140 fix issue with > > > CuratorLoadQueuePeon shutting down executors it does not own > > > > > > doc fixes: > > > > > > https://github.com/apache/incubator-druid/pull/8002 Improve pull-deps > > > reference in extensions page. > > > https://github.com/apache/incubator-druid/pull/8003 Add missing > > reference > > > to Materialized-View extension. > > > https://github.com/apache/incubator-druid/pull/8079 Fix documentation > > > formatting > > > https://github.com/apache/incubator-druid/pull/8087 fix references to > > > bin/supervise in tutorial docs > > > > > > Anything else that should go in? (bug or documentation fixes only > > please). > > > > > > Cheers, > > > Clint > > > > > >
Re: npm ci error while building from source
Aha! mystery solved. .npmrc was pointing to an internal npm registry which couldn't provide the packages. Thanks Adam! On Thu, Jul 11, 2019 at 10:32 AM Adam Peck wrote: > Do you have any registry overrides in a .npmrc file? > https://docs.npmjs.com/files/npmrc > > On Thu, Jul 11, 2019 at 11:21 AM Gian Merlino wrote: > > > I haven't seen something like that before. The package version 4.2.6 is > > here on NPM: > https://www.npmjs.com/package/@types/react-copy-to-clipboard > > > > Is it possible your build environment is blocking NPM? > > > > On Thu, Jul 11, 2019 at 9:43 AM Eyal Yurman > > wrote: > > > > > The mvn command was actually: > > > mvn clean install -Pdist -DskipTests, from IntelliJ > > > (It shouldn't matter though) > > > > > > On Thu, Jul 11, 2019 at 9:39 AM Eyal Yurman < > eyurma...@verizonmedia.com> > > > wrote: > > > > > > > Hi, > > > > > > > > Has anyone encountered the following error? > > > > > > > > mvn clean install -PskipTests > > > > > > > > [INFO] --- frontend-maven-plugin:1.6:npm (npm-install) @ > druid-console > > > --- > > > > [INFO] Running 'npm ci' in > > > > /Users/eyurman14/git/yurmix-incubating-druid/web-console > > > > [ERROR] npm ERR! code ETARGET > > > > [ERROR] npm ERR! notarget No matching version found for > > > > @types/react-copy-to-clipboard@4.2.6 > > > > [ERROR] npm ERR! notarget In most cases you or one of your > dependencies > > > > are requesting > > > > [ERROR] npm ERR! notarget a package version that doesn't exist. > > > > [ERROR] > > > > [ERROR] npm ERR! A complete log of this run can be found in: > > > > [ERROR] npm ERR! > > > > /Users/eyurman14/.npm/_logs/2019-07-11T16_34_32_404Z-debug.log > > > > > > > > > > > > > >
Re: npm ci error while building from source
The mvn command was actually: mvn clean install -Pdist -DskipTests, from IntelliJ (It shouldn't matter though) On Thu, Jul 11, 2019 at 9:39 AM Eyal Yurman wrote: > Hi, > > Has anyone encountered the following error? > > mvn clean install -PskipTests > > [INFO] --- frontend-maven-plugin:1.6:npm (npm-install) @ druid-console --- > [INFO] Running 'npm ci' in > /Users/eyurman14/git/yurmix-incubating-druid/web-console > [ERROR] npm ERR! code ETARGET > [ERROR] npm ERR! notarget No matching version found for > @types/react-copy-to-clipboard@4.2.6 > [ERROR] npm ERR! notarget In most cases you or one of your dependencies > are requesting > [ERROR] npm ERR! notarget a package version that doesn't exist. > [ERROR] > [ERROR] npm ERR! A complete log of this run can be found in: > [ERROR] npm ERR! > /Users/eyurman14/.npm/_logs/2019-07-11T16_34_32_404Z-debug.log > >
npm ci error while building from source
Hi, Has anyone encountered the following error? mvn clean install -PskipTests [INFO] --- frontend-maven-plugin:1.6:npm (npm-install) @ druid-console --- [INFO] Running 'npm ci' in /Users/eyurman14/git/yurmix-incubating-druid/web-console [ERROR] npm ERR! code ETARGET [ERROR] npm ERR! notarget No matching version found for @types/react-copy-to-clipboard@4.2.6 [ERROR] npm ERR! notarget In most cases you or one of your dependencies are requesting [ERROR] npm ERR! notarget a package version that doesn't exist. [ERROR] [ERROR] npm ERR! A complete log of this run can be found in: [ERROR] npm ERR! /Users/eyurman14/.npm/_logs/2019-07-11T16_34_32_404Z-debug.log
Re: deprecating the old repo
I think after this final pr, is will be proper to archive the repository so it doesn't look active ( https://help.github.com/en/articles/archiving-a-github-repository). It will be accessible, but read-only, and won't come up in the default GitHub search. Here's an example for such a repository (archived and with a comment): https://github.com/yahoo/druid-extensions On Fri, Apr 26, 2019 at 11:14 AM Julian Hyde wrote: > One trick that people sometimes use is to create a new branch called > “obsolete” or similar, update the README in that branch to point to the new > project location, and make that branch the default branch in GitHub. > > > On Apr 26, 2019, at 11:02 AM, Gian Merlino wrote: > > > > That makes sense to me. Are you interested in doing a PR to the > > docker-druid repo to make its README point to the new Dockerfiles? If so, > > that should do it. > > > > On Fri, Apr 26, 2019 at 3:33 AM Jokin Cuadrado > wrote: > > > >> Hi, I was searching for a way to run druid on docker for some > >> experimentation, and the first results has been a repo on the droid-io > >> organization https://github.com/druid-io/docker-druid > >> > >> I found after (Thanks to dylan pointing out in slack) that there are > newer > >> dockerfiles commited on the apache incubatin repo. > >> > >> As the ol repo sits unmodified and with some open pull request, I think > >> that cleaning the old repo, marking as deprecated and pointing to the > new > >> repo would be nice. The https://github.com/druid-io organization should > >> maybe also point to the incubator project, as the only active project > it's > >> the website code. > >> > >> Regards, > >> Jokin. > >> > > > - > To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org > For additional commands, e-mail: dev-h...@druid.apache.org > >
Re: Experimental feature 'graduation' in 0.14
What does this mean for release of new query features without Druid SQL support? On Mon, Apr 22, 2019 at 2:07 PM Jihoon Son wrote: > +1 > I'm mostly using only SQL. > > Jihoon > > On Mon, Apr 22, 2019 at 12:24 PM Jonathan Wei wrote: > > > +1, I think it has enough feature parity with the native JSON queries, > and > > it's stable enough to be moved out of experimental. > > > > On Thu, Apr 18, 2019 at 6:23 PM Fangjin Yang wrote: > > > > > Strong +1 > > > > > > I think there's been enough production usage of Druid SQL, it matches > > what > > > native JSON-over-http can do, and it should no longer be labeled as > > > experimental. > > > > > > On Thu, Apr 18, 2019 at 6:06 PM Gian Merlino wrote: > > > > > > > Hey all, > > > > > > > > Reviving this thread. Now that > > > > https://github.com/apache/incubator-druid/pull/6742 and > > > > https://github.com/apache/incubator-druid/pull/6862 have been > released > > > in > > > > 0.14, I'm re-proposing graduating Druid SQL from experimental status > in > > > the > > > > next release, 0.15. I don't think we need a formal vote on this, but > if > > > > there seems to be general consensus, I'll do a PR before the next > > > 3-monthly > > > > 0.15 code freeze (which is in about 2 weeks). > > > > > > > > On Thu, Jan 31, 2019 at 9:20 AM Gian Merlino > wrote: > > > > > > > > > It sounds like the general feeling is +1 on Kafka and maybe wait > > > another > > > > > release for SQL. I will do a PR to mark Kafka ingest as > > > non-experimental, > > > > > then, and on SQL we'll see whether #6742 and #6862 look solid in > > 0.14. > > > > > > > > > > On Tue, Jan 15, 2019 at 8:39 AM Gian Merlino > > wrote: > > > > > > > > > >> Hi Mat, > > > > >> > > > > >> Ah, right. IMO > https://github.com/apache/incubator-druid/pull/6742 > > > is a > > > > >> decent workaround towards making #6176 less of a problem. It would > > > > prevent > > > > >> incorrect results from happening (the broker will not start up its > > > http > > > > >> server & announce itself, and so it won't get picked up by > clients, > > if > > > > it > > > > >> never got the initialization event). If paired with monitoring > that > > > > >> restarts unhealthy brokers, the issue should be fully > worked-around > > in > > > > >> practice. > > > > >> > > > > >> Even though there's an (imo) viable workaround, it would still be > > good > > > > to > > > > >> fix the root cause of #6176. I just raised > > > > >> https://github.com/apache/incubator-druid/pull/6862 to update > > Curator > > > > >> and see if that helps -- there is a bug fixed in the latest > release > > > that > > > > >> looks like it could cause the behavior we're seeing ( > > > > >> https://issues.apache.org/jira/browse/CURATOR-476). > > > > >> > > > > >> My feeling is that it's still reasonable to remove the > experimental > > > > label > > > > >> from Druid SQL in 0.14, especially since #6742 will make SQL and > > > native > > > > >> queries behave at parity (initialization getting missed will delay > > > > broker > > > > >> startup for _both_ cases). So in that sense they are at least on > the > > > > same > > > > >> footing. And hopefully #6862 will fix them both, together. > > > > >> > > > > >> On Tue, Jan 15, 2019 at 7:56 AM Pierre-Emile Ferron < > > > > pe.fer...@gmail.com> > > > > >> wrote: > > > > >> > > > > >>> A remaining issue with SQL is > > > > >>> https://github.com/apache/incubator-druid/issues/6176 > > > > >>> > > > > >>> We've seen it happen several times in production on 0.12, where > > > > >>> thankfully > > > > >>> SQL doesn't power anything critical. The current workarounds are: > > > > >>> 1. Restart the broker. Obviously not a good solution. > > > > >>> 2. Migrate to HTTP segment discovery. I'm fine with that, and we > > are > > > > >>> actually planning to do it soon in our clusters, but I'm still > > > > concerned > > > > >>> about other Druid users—the default setting is still ZK, which > > means > > > > that > > > > >>> SQL would still have this issue by default. > > > > >>> > > > > >>> Before marking SQL as non-experimental, I'd suggest either fixing > > the > > > > >>> root > > > > >>> cause, or making HTTP segment discovery the default and then > > > explicitly > > > > >>> deprecating ZK segment discovery. > > > > >>> > > > > >>> > > > > >>> On Mon, Jan 14, 2019 at 2:18 PM Gian Merlino > > > wrote: > > > > >>> > > > > >>> > I'd like to propose graduating a couple of features out of > > > > >>> 'experimental' > > > > >>> > status in 0.14. Both are popular features (judging by mailing > > list > > > & > > > > >>> github > > > > >>> > issue/PR activity). Both have been around for a while and have > > > > >>> attained a > > > > >>> > good level of quality and stability of API & behavior. I > believe > > > > >>> removing > > > > >>> > the 'experimental' banner from these features would more > > accurately > > > > >>> reflect > > > > >>> > reality, and be a good signal to the user community. > > > > >>> > > > > > >>> > 1) Ka
Druid module separation (And a related lookups question)
Hi, Can someone help me understand the logic for the separation of core, server and processing modules, and why? More specificly, I'm trying to understand the split of the package `org.apache.druid.query.lookup` between the two latter modules. Eyal.
Re: Druid configuration documentation should not be organized in tables
Just for comparison, this looks nice, isn't it? https://spark.apache.org/docs/latest/configuration.html Original .md: https://github.com/apache/spark/blob/master/docs/configuration.md On Tue, Mar 5, 2019 at 3:20 PM Clint Wylie wrote: > I definitely agree that a lot of these configuration option docs have > outgrown the tables. I do however think the giant table as a reference to > lookup configuration property names and default values is handy; I think > maybe a system of links in the tables, like "see coordinator balancing docs > for more details" or whatever, where the implications of settings could be > described in greater detail might be more appropriate than embedding all of > the information in the tables. I do like the idea of like nested hidden > larger descriptions so you can sort of have both a table and a fly out > large description to keep it similar to what we have and not have to update > or add documentation in multiple places, but I have no idea how that works > in markdown world or if it's possible. > > On Tue, Mar 5, 2019 at 3:04 PM Roman Leventov > wrote: > > > On Mon, 4 Mar 2019 at 21:36, Gian Merlino wrote: > > > > > I'm a bit scared that doing this would make the already large list of > > > configuration options ( > > > http://druid.io/docs/latest/configuration/index.html) > > > become even more daunting and difficult to wrap one's head around. > > > > > > > Sounds like a good nudge for ourselves to always seek for opportunities > to > > remove configurations! Besides, we may reorganize the page to show only > > configuration names, and then a full description is open (like a > "spoiler" > > but not exactly, I'm not sure how this HTML control is called. Similar to > > how FAQ lists are organized sometimes) when the option name is clicked. > > Markdown->HTML converter that we use may support this, so the doc may > need > > to become a proper HTML page. > > > > Since, so often, multiple parameters interact with each other, maybe it > > > would make sense for the larger explanatory texts to be written in > their > > > own sections, with the parameter tables linking to them? It makes sense > > > with processing buffer size and num threads configs, for example, since > > > they aren't just important alone but they do relate to each other as > > well. > > > > > > > I'm afraid the necessity to find a place for a section, inner feeling > that > > a section should be "big enough" to justify its existence, the necessity > of > > inter-linking in markdown will make it even less likely in practice that > > developers will create good docs with comprehensive discussions of > > configuration options and their values. > > >
Downloading binaries of previous Druid versions
Hi, Are binaries of previous versions from before 0.13.0-incubating still available for download? Or should one build the old version from source? Thanks, Eyal.
Run TeamCity inspections locally
Hi, Does anyone know how to run the same code inspections done by TeamCity CI in IntelliJ? I'm getting somewhat different results when using "Analyze->Inspect Code" with "Inspection profile": Druid. Eyal.
Re: Knowledge sharing between Druid developers via technical talks
Thanks for the response, that sounds great! Since the meetups are user-focused, perhaps a separate "track" which is open to all but is dev-focused? This could be before/after the main event. I promise that once I get enough experience with the code base, I'd volunteer to present, but hopefully, there are much better candidates at the moment :) On Mon, Feb 18, 2019 at 1:36 PM Gian Merlino wrote: > I am interested especially if the format is something live. An in-person > meetup with a recording distributed afterwards would be my preference, if > people are into that. Maybe something at one of the Druid meetups? > > On Wed, Feb 13, 2019 at 8:38 PM Eyal Yurman > wrote: > >> Hi, >> >> This is something usually being done in companies, but I think it is >> useful >> for any community, especially our community which is so distributed. >> >> I think it would be absolutely wonderful if we can find people willing to >> share their knowledge with other contributors via the form of a tech-talk. >> I.e. it would be very useful if someone could take a subject (Just for >> example, groupBy query) and present the high-level >> architecture/implementation. >> >> I know this requires significant effort, but I hope to convince you of the >> benefits it would provide to the Druid project: >> - Helping any newcomer being more effective, thus providing better >> contribution ROI against work effort. >> - Serving as a high-quality medium of communication within the group of >> committers, which would lead to more trust and understanding. >> >> Recording and uploaded such sessions will make them Apache-Way compatible >> (Along with serving future viewers). >> >> So, anyone up to the challenge? :) >> >> Eyal. >> >
Knowledge sharing between Druid developers via technical talks
Hi, This is something usually being done in companies, but I think it is useful for any community, especially our community which is so distributed. I think it would be absolutely wonderful if we can find people willing to share their knowledge with other contributors via the form of a tech-talk. I.e. it would be very useful if someone could take a subject (Just for example, groupBy query) and present the high-level architecture/implementation. I know this requires significant effort, but I hope to convince you of the benefits it would provide to the Druid project: - Helping any newcomer being more effective, thus providing better contribution ROI against work effort. - Serving as a high-quality medium of communication within the group of committers, which would lead to more trust and understanding. Recording and uploaded such sessions will make them Apache-Way compatible (Along with serving future viewers). So, anyone up to the challenge? :) Eyal.
Re: Dev sync
I agree that syncing is done better via other mediums. Actually, I have a different suggestion for live meetings but I'll post it separately, so not to hijack this conversation. On Wed, Feb 13, 2019 at 3:15 PM Gian Merlino wrote: > I personally join about 1 in 10 of them so, from that perspective, I feel > that I am getting what I need in terms of communication out of the lists > and github and don't need extra utility from the dev syncs. Even if we stop > doing them, meeting face to face is still nice, and I always like to see > people at meetups :) > > On Tue, Feb 12, 2019 at 9:41 AM Charles Allen > wrote: > > > I am unable to host the dev sync this week. > > > > Is anyone finding utility out of these? the dev list seems pretty active > > these days, so the legacy utility of the dev sync is very muted (this is > a > > good thing). Unless people are finding specific utility out of a weekly > > video sync up, I propose it be postponed indefinitely until a need can be > > identified. > > > > Thoughts? > > >
Re: Off list major development
Hi, I have created an Issue together with @jon-wei, if anyone wants to chime in: https://github.com/apache/incubator-druid/issues/6949 (Create a proposal template #6949) On Tue, Jan 15, 2019 at 12:07 PM Jihoon Son wrote: > Good point. > If some authors raise PRs without noticing the need for a proposal, we > shouldn't ask them to close their PRs only because of the absence of the > proposal. > > "Design review" without a proposal for simple PRs would be good if we can > determine well what PRs need and what don't. > But, how do we know? Even for the same PR, someone may think it needs a > proposal but another may not. > > If someone don't notice the need for a proposal and raise a PR without it, > I'm fine with that. > However, we should still encourage writing a proposal before writing code > because we can avoid unnecessary effort. > > I think this kind of issue usually happens for first time contributors and > they will be better once they get used to Druid development. > And I believe someday even first contributors would follow this policy once > it gets settled down well in the community as Kafka community does. > > Jihoon > > On Tue, Jan 15, 2019 at 4:31 AM Roman Leventov > wrote: > > > In such small PRs, authors likely won't be aware that they need to > create a > > proposal in the first place. The first reviewer just adds the "Design > > Review" tag. It's also absolutely not about considering designs and > gauging > > the proposal, it's just verifying that a configuration / parameter / HTTP > > endpoint name is reasonable and aligned with the rest of Druid. So I > think > > that a separate proposal issue for such PRs is unnecessary bureaucracy. > > > > On Tue, 15 Jan 2019 at 07:45, Jihoon Son wrote: > > > > > Roman, > > > > > > > Jihoon in > > > > > > > > > https://lists.apache.org/thread.html/e007fbf362c2a870a2d88d04431789289807e00fd91d087559a01d1f@%3Cdev.druid.apache.org%3E > > > and later Gian in this thread suggested that _every_ piece of work that > > > should be labelled as "Design Review" according to the current rules > > should > > > be accompanied by an issue. I don't agree with this, there are some PRs > > as > > > small as a few dozens of lines of code, that add some configuration > > > parameter and therefore should be labelled "Design Review". I don't > > thing a > > > separate proposal issue is needed for them, and even for a little > larger > > > PRs too. > > > > > > What I'm concerned with is how people feel if their design is not > > accepted > > > even though they wrote code. Of course, as Clint said, sometimes code > > helps > > > better understanding of the proposal. But, I believe this is the case > > when > > > the proposal is quite complicated and not easy to understand without > > code. > > > Also the authors should be aware of that they might rewrite the entire > > code > > > if the design should be changed. > > > > > > If writing code is simple, I don't see why the authors don't wait until > > the > > > review for their proposal is finished. > > > > > > Jihoon > > > > > > On Fri, Jan 11, 2019 at 9:51 AM Fangjin Yang wrote: > > > > > > > I agree with Gian, as an Apache committer, your responsibility is for > > the > > > > betterment of the project. I agree it is in the best interest of the > > > > project to stop thinking about what orgs people belong to. We are > all a > > > > part of the Apache software foundation, regardless of what our roles > > and > > > > titles are outside of it. > > > > > > > > On Fri, Jan 11, 2019 at 2:22 AM Roman Leventov < > leventov...@gmail.com> > > > > wrote: > > > > > > > > > It's not that people from one org could abuse the project and push > > some > > > > > change, but that they have similar perspective (bubble effect) and > > some > > > > > important aspects of a large feature could escape their attention. > > > > > > > > > > I suggest it to be not a rigid rule, but a recommendation for > authors > > > of > > > > > large proposals to try to attract reviewers from other orgs. > > > > > > > > > > On Fri, 11 Jan 2019 at 02:51, Julian Hyde > wrote: > > > > > > > > > > > I agree with Gian. > > > > > > > > > > > > As an Apache committer, you only have one affiliation: you are > > > working > > > > in > > > > > > the best interests of the project. > > > > > > > > > > > > Obviously, in the real world there are other pressures. But we do > > our > > > > > best > > > > > > to compensate for them. > > > > > > > > > > > > Also, as a a community we try to design our process so as to > avoid > > > > undue > > > > > > influences. For instance, when I advocate for logging cases > early, > > I > > > am > > > > > > trying to mitigate the effect of product managers and VPs of > > > > engineering, > > > > > > who like to have their say in meeting rooms rather than on public > > > > mailing > > > > > > lists. That’s just one example; if we see other influences at > play, > > > > let’s > > > > > > evolve our process to try to level the playing field.
Re: Contributing an extension
Thanks!! Btw, sorry for the dual messages, the mail server blocked my first one and after I resubscribed to the list, it has sent it :) On Tue, Jan 29, 2019, 18:26 Jonathan Wei Hi Eyal, > > Thanks for the contrib, I'll have some time to start reviewing in the next > week or two. It seems like a cool feature with interest from users, so I'd > lean towards keeping it open for review/merge. > > Best, > Jon > > On Tue, Jan 29, 2019 at 5:29 PM Eyal Yurman > wrote: > > > Hi, > > > > This pr is waiting for a few months for review: > > https://github.com/apache/incubator-druid/pull/6430 > > https://github.com/apache/incubator-druid/issues/6320 > > > > So far only one reviewer (@b-slim) was kind enough to start looking at > it. > > > > Since this is an contrib extension, it is not really blocking me (It is > in > > production for the past 2 years...). But I have noticed a large number of > > users commenting/emojiing the PR/issue, so I think the community would > > appreciate it if this would be released. > > > > Since this is an extension, would you say it's better if I release it in > my > > company's own repository, or should I continue to request review and > merge? > > > > Thanks! > > > > Eyal. > > >
Contributing an extension
Hi, This pr is waiting for a few months for review: https://github.com/apache/incubator-druid/pull/6430 https://github.com/apache/incubator-druid/issues/6320 So far only one reviewer (@b-slim) was kind enough to start looking at it. Since this is an contrib extension, it is not really blocking me (It is in production for the past 2 years...). But I have noticed a large number of users commenting/emojiing the PR/issue, so I think the community would appreciate it if this would be released. Since this is an extension, would you say it's better if I release it in my company's own repository, or should I continue to request review and merge? Thanks! Eyal.
Contributing an extension
Hi, A few months ago I worked on open sourcing an extension we've been using internally for a couple of years. https://github.com/apache/incubator-druid/pull/6430 https://github.com/apache/incubator-druid/issues/6320 This has been upvoted by a few community users, so I was happy to put in the effort. Unfortunately, I haven't found any committer who could spend the time reviewing this work... Is there anyone who could do that sometime next month? I could always release it under a separate repository, but since I am committed to continuing maintaining and expanding the module, I'd prefer to keep it as part of the Druid codebase. Thanks! Eyal.
Re: This week's dev sync
Thank you! Regarding moving-average mentioned, I have a pr waiting for review ( https://github.com/apache/incubator-druid/pull/6430) and there is another related pr coming by a colegue of mine. If someone can review that would be wonderful. Thanks! On Tue, Dec 11, 2018, 12:49 Clint Wylie Thanks for taking notes! > > On Tue, Dec 11, 2018 at 10:15 AM Charles Allen > wrote: > > > Notes: > > * Charles is official note taker for this session. > > * Release is looking good so far. David working on getting total release > > cut. > > * Unsure what the status of the website build for release is. If there > are > > blockers it is asked to be called out in the dev list. > > * Charles mentioned https://github.com/apache/incubator-druid/pull/5913 > > and > > its failure in group by queries as a main blocker for adoption. Jihoon > has > > https://github.com/apache/incubator-druid/pull/6629 as an alternative > > approach to a very similar problem. The authors in question have been in > > sync that parallel development was a risk so this is not a surprise to > > either of us. > > * Clint mentioned lots of pretty big PRs outstanding. > > * There is a growing interest in various groups about result aggregations > > like moving averages and cumulative totals. If there are pockets of > effort > > in similar post-query processing or result level processing, please make > > sure it is known in the community. > > * It is proposed the dev sync for Dec 25th and Jan 1st be skipped. > > > > > > On Tue, Dec 11, 2018 at 9:51 AM Charles Allen > > wrote: > > > > > To join the video meeting, click this link: > > > https://meet.google.com/ozi-rtfg-ags > > > Otherwise, to join by phone, dial +1 442-666-1256 and enter this PIN: > > 6867# > > > To view more phone numbers, click this link: > > > https://tel.meet/ozi-rtfg-ags?hs=5 > > > > > >
Binary artifacts in Druid git
Hi, There was discussion today in the Dev sync today regarding binary artifacts. It sounds like having binary files in the source code can fail an audit for the 0.13.0-incubating release, but it is not clear which files are considered "artifacts". I opened an issue with a list of files, I hope this helps: https://github.com/apache/incubator-druid/issues/6476 - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
Re: Subscription request for dev
Hi Mohammad, You can subscribe to the list by emailing "dev-subscr...@druid.apache.org". On 2018/10/09 15:14:02, Mohammad.J.Khan wrote: > Hi, > > > > I’d like to subscribe to the Druid’s dev mailing lists > > > > Thanks, > > Mohammad > > Mohammad J. Khan | Lead Data Engineer | Enterprise Data, Analytics and > Business Intelligence | •Target | 612-239-9959 > - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
Re: Towards 0.13 (Apache release)
Hi, my only issue that GitHub loses the commit history beyond the rename (i.e.: https://github.com/apache/incubator-druid/blob/master/server/src/main/java/org/apache/druid/server/ClientQuerySegmentWalker.java). Luckily it is viewable locally via "git log --follow" or with using your favorite SCM GUI tool. On 2018/08/29 16:34:52, Gian Merlino wrote: > Hi everyone, > > As we continue towards 0.13 I started looking into the "great renaming" (of > all packages from io.druid -> org.apache.druid) and am getting a PR ready. > I know Slim is working on > https://github.com/apache/incubator-druid/pull/6215 too (automated license > checking and some header fixups). > > Other than these Apache related items, we have 26 open issues/PRs in the > 0.13.0 milestone: https://github.com/apache/incubator-druid/milestone/25. > Is this everything we want to include? Is anything there we should bump to > the next release? Is anything _not_ there that needs to be added? > > Let's figure out when we can target a code freeze -- the start of the RC > train for our first Apache release!! > - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
Re: Percentile calculations with Druid
Hi, I would look at data sketches as potentially a better alternative, with the recent release of "Numeric quantiles sketch aggregator" (As noted in the release notes: https://github.com/apache/incubator-druid/releases/druid-0.12.0). Unfortunately, the documentation wasn't ready with that release, thus is not available on the public website yet. It is marked to be released on the next major milestone (0.13.0) and viable on the Druid master branch: https://github.com/apache/incubator-druid/blob/master/docs/content/development/extensions-core/datasketches-quantiles.md On 2018/08/31 10:13:56, Abhishek Kaushik wrote: > Hi, > > I wanted to know which algorithm/s druid uses to compute percentile values. > I came across this doc: > http://druid.io/docs/latest/development/extensions-core/approximate-histograms.html > but it mentions that there are "no formal error bounds on the > approximation". So, just want to know what all options are available if one > wishes to compute percentiles? > > Thanks in advance. > - To unsubscribe, e-mail: dev-unsubscr...@druid.apache.org For additional commands, e-mail: dev-h...@druid.apache.org
