Re: [druid-user] Druid 12.1 Datasource load fails in Coordinator due change in implementation from Set to Map

[Gian Merlino]

Yes, this should be out soon. We regret the regression! If you are
comfortable patching #5878 into your build, that should fix it. It will
also be included in 0.12.2.

Gian

On Thu, Jun 28, 2018 at 10:35 AM, Samarth Jain 
wrote:

> Adding the dev email group.
>
> We are currently hitting this problem in our environment too where loading
> 200K segments is taking forever where as on 10.1 the load happened in less
> than 5 minutes. I see a pull request (https://github.com/druid-io/
> druid/pull/5878) that potentially fixes this issue that was checked in to
> master. I believe this fix would be part of the 0.12.2 release whenever it
> comes out.
>
>
>
>
> On Thu, Jun 28, 2018 at 1:50 AM, Venu Reddy 
> wrote:
>
>> Hi Team,
>>
>> We have close to ~500,000 active data segment in the Metadata store
>> (Postgres)
>> Coordinator is running on a 4 CPU, Centos server
>>
>> We have updated from 0.10.0 to Druid 0.12.1, Post this when we bring up
>> the Co-ordinator we see the below behaviour
>>
>> The datasource loading keeps running and goes into a hung state in inside
>> the poll() function in *SQLMetadataSegmentManager.java*.
>> On further debugging we see that below portion is the one that is taking
>> time
>>
>> if (!dataSource.getSegments().contains(segment)) {
>>   dataSource.addSegment(segment);
>> }
>>
>> And it seems like the main reason it is taking time is due to the change
>> in the file *DruidDataSource.java* from *ConcurrentSkipListSet* (and a
>> HashMap) to *ConcurrentSkipListMap*
>>
>> We added additional logging statements to time the above section in the
>> *SQLMetadataSegmentManager.java *and we see that as the loop runs
>> collecting segments, initially the time taken is less than a milli second
>> but as the loop runs inserting more records into the
>> *ConcurrentSkipListMap,* the insertions take ~8 ms by ~50k records and
>> then increase all the way to ~300 ms when we reach 300K records
>>
>> We also added the same timers to the *lower version* of Druid and with
>> *ConcurrentSkipListSet *the implementation the loop completes processing
>> the 500k records in 5 mins.
>> Also when we try with a higher config machine 32 CPU, we still see the
>> same behaviour.
>>
>> In Summary it seems like *ConcurrentSkipListMap* is slower than
>> *ConcurrentSkipListSet* and is resulting in some sort of timeout in
>> version 0.12.1 whereas the same number of segments are getting loaded
>> without issues in under 10 mins in version 0.10.0. Also when we check the
>> code, the code in 0.11.0 seems identical to 0.10.0 however the 0.12.1 has
>> this change.
>>
>> Regards,
>> Venu
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Druid User" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to druid-user+unsubscr...@googlegroups.com.
>> To post to this group, send email to druid-u...@googlegroups.com.
>> To view this discussion on the web visit https://groups.google.com/d/ms
>> gid/druid-user/50a75795-57af-455c-955b-7153379b9253%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Druid User" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to druid-user+unsubscr...@googlegroups.com.
> To post to this group, send email to druid-u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/druid-user/CAMfSBKJH46KBnKhLsTU2n_YF8Nm1LE0HCJxkP5neMRT8aJFTug%
> 40mail.gmail.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>

Sync this week

[Charles Allen]

I am indisposed this week. Can anyone else start the sync up?

Re: Sync this week

[Jonathan Wei]

Sure:

https://meet.google.com/kxu-vxec-kmr



On Tue, Jul 3, 2018 at 9:56 AM, Charles Allen  wrote:

> I am indisposed this week. Can anyone else start the sync up?
>

Re: Druid repo migration plan

[Gian Merlino]

Here is the ticket, btw: https://issues.apache.org/jira/browse/INFRA-16674.
Repo move should be happening real soon now!

On Mon, Jul 2, 2018 at 11:55 PM Gian Merlino  wrote:

> Our infra ticket is progressing along and it looks like we're just about
> ready to pull the trigger on moving the repo. So, committers, please make
> sure your ASF gitbox stuff is working: https://gitbox.apache.org/setup/
>
> On Fri, Jun 22, 2018 at 1:22 PM Gian Merlino  wrote:
>
>> Thanks for the tips, Max!! I think we are, hopefully, doing okay on some
>> of these. My thoughts inline.
>>
>> > Since you need elevated rights on both
>> orgs to move the repo (say airbnb and apache) and that both parties aren't
>> ok with that, it's typical to use a middleman org like `apacheinfra`.
>>
>> Luckily, our org is limited to just Druid stuff (
>> https://github.com/druid-io) so we should be OK to add Apache Infra
>> people with elevated rights.
>>
>> > * make merge hook checks optional, so that if coverage, travis, or code
>> quality checks do not prevent merging, since it's likely those check won't
>> trigger and as a non-admin you won't be able to force-merge
>>
>> We have a couple (Travis and TeamCity) and they're already optional.
>>
>> > * consider unprotecting protected branches so that you can push to
>> master
>> if controlling master is important in your workflow. This way you can
>> effectively merge PRs without clicking the button on GH.
>>
>> Master _is_ important. Although I think if we can't do PRs, then pushing
>> directly to master is probably not going to be too helpful anyway (the PRs
>> are essential to our code review workflow). So I think we have to hope for
>> the best here?
>>
>> > * make sure core committers have their Gitbox access setup, I think it
>> can
>> be a bit tricky and may involve your mentor / infra pulling some levers on
>> whimsy
>>
>> I went through this process (via https://gitbox.apache.org/setup/) to
>> get the ability to push to
>> https://github.com/apache/incubator-druid-website, which is powering
>> https://druid.apache.org/. It took a little while and was kind of
>> confusing but it does work now. Other Druid committers: sounds like getting
>> set up on GitBox early is a good thing, so please check it out!
>>
>> On Fri, Jun 22, 2018 at 11:51 AM Maxime Beauchemin <
>> maximebeauche...@gmail.com> wrote:
>>
>>> @julian gotcha, I thought this was a more official vote
>>>
>>> The Superset GH move INFRA ticket shows how the move can be really
>>> tricky/slow/disruptive. There was quite a period of instability for us
>>> and
>>> a lot of slow back and forth with Apache infra. Hopefully the process has
>>> been ironed out since then. Be prepared and go into it knowing that you
>>> may
>>> not be able to merge PRs for days/weeks.
>>> https://issues.apache.org/jira/browse/INFRA-14267
>>>
>>> On the ticket you open with INFRA, make it really clear what your GH
>>> integrations are and validate that they are all approved/supported by
>>> Apache prior to the move. Some integrations (like codeclimate) require
>>> rights on the GH org (Apache) and INFRA is categoric against that. If
>>> some
>>> services aren't supported make sure to disable the integrations prior to
>>> the move, find replacement services. Also make sure INFRA will
>>> adjust/tweak
>>> the integration post move as you likely need admin rights to do so.
>>>
>>> A caveat is around the "redirect chain" on GH. This is what allows
>>> hitting `
>>> github.com/airbnb/superset`  to
>>> redirect to `
>>> github.com/apache/incubator-superset`
>>>  to redirect to the right
>>> place. This
>>> also allows `git remote`s to just work post move. This redirect chain is
>>> fragile and can break in some cases. Since you need elevated rights on
>>> both
>>> orgs to move the repo (say airbnb and apache) and that both parties
>>> aren't
>>> ok with that, it's typical to use a middleman org like `apacheinfra`.
>>> They
>>> grant you admin right to that org and you move the repo to there, and
>>> they
>>> do the second part. If, post move, the middleman was to fork a repo with
>>> the same name, or create one, it would break the redirect chain.
>>> Something
>>> INFRA should be aware of at this point and cautious around it. Also note
>>> that some GH integrations work through the redirect chain and some don't
>>> and require re-pointing/configuring the service to the new location.
>>>
>>> Thing you can do to prepare and mitigate risks:
>>> * make merge hook checks optional, so that if coverage, travis, or code
>>> quality checks do not prevent merging, since it's likely those check
>>> won't
>>> trigger and as a non-admin you won't be able to force-merge
>>> * consider unprotecting protected branches so that you can push to master
>>> if controlling master is important in your workflow. This way you can
>>> effectively merge PRs without clicking the button on GH.
>>> * make sure co