Re: Have we ever made a Kibble release?

[Kaxil Naik]

Yup, this is what we do at Apache Airflow too.

And I did the same for the recently released Apache Airflow Helm Chart.

Example:
https://lists.apache.org/x/thread.html/rd2a0a6386c22bb0942c28746d59d105adc0364715c15ea772e23e105@%3Cdev.airflow.apache.org%3E

I have releasing Airflow versions from past 3 or so years, so have good
experience with ASF guidelines around it.

Regards,
Kaxil



On Sun, May 23, 2021, 15:24 Sharan Foga  wrote:

>
>
> On 2021/05/23 14:15:32, Kaxil Naik  wrote:
> > Yeah, as far as I know we don't have a release. For Kibble 2 we can
> release
> > Python wheel/binary artifacts.
>
> I know there is some discussion going on about binary artifacts and Apache
> releases as generally Apache releases need to be source code, though they
> have been supplemented with binary artifacts. We just need to make sure we
> adhere to the current policy.
>
> Thanks
> Sharan
>
> >
> > Regards,
> > Kaxil
> >
> > On Sun, May 23, 2021, 15:10 Tomasz Urbaszek 
> wrote:
> >
> > > As far as I know Apache Kibble was never released. I would propose
> > > using Github milestones to track future releases of new Kibble:
> > > https://github.com/apache/kibble/milestone/1
> > >
> > > Tomek
> > >
> > > On Sun, 23 May 2021 at 16:00, Sharan Foga  wrote:
> > > >
> > > > Hi All
> > > >
> > > > Thanks to all who helped review our Board report. We got some
> feedback
> > > comment to include details of our last release in our next report. I've
> > > been searching through our mailing archives and can't find any details
> or
> > > Vote of the project ever doing a release.
> > > >
> > > > If this is the case then we need to think about planning one as part
> of
> > > our roadmap.
> > > >
> > > > Thanks
> > > > Sharan
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: Have we ever made a Kibble release?

[Kaxil Naik]

Yeah, as far as I know we don't have a release. For Kibble 2 we can release
Python wheel/binary artifacts.

Regards,
Kaxil

On Sun, May 23, 2021, 15:10 Tomasz Urbaszek  wrote:

> As far as I know Apache Kibble was never released. I would propose
> using Github milestones to track future releases of new Kibble:
> https://github.com/apache/kibble/milestone/1
>
> Tomek
>
> On Sun, 23 May 2021 at 16:00, Sharan Foga  wrote:
> >
> > Hi All
> >
> > Thanks to all who helped review our Board report. We got some feedback
> comment to include details of our last release in our next report. I've
> been searching through our mailing archives and can't find any details or
> Vote of the project ever doing a release.
> >
> > If this is the case then we need to think about planning one as part of
> our roadmap.
> >
> > Thanks
> > Sharan
> >
> >
> >
> >
>

Re: It's Board Report Time!

[Kaxil Naik]

Looks good to me too, thanks Sharan

On Fri, May 7, 2021 at 10:00 PM Tomasz Urbaszek 
wrote:

> Thanks Sharan for putting this together! Looks good to me
>
> Thanks,
> Tomek
>
> On Fri, 7 May 2021 at 19:38, Sharan Foga  wrote:
> >
> > Hi All
> >
> > Here is my initial draft for the report and all feedback welcome!
> >
> > Please let me know if you think there is something important that I've
> missed.
> >
> > Thanks
> > Sharan
> >
> > -
> >
> > ## Description:
> > The mission of Apache Kibble is the creation and maintenance of software
> > related to an interactive project activity analyzer and aggregator
> >
> > ## Issues:
> > There are no issues requiring board attention at this time.
> >
> > ## Membership Data:
> > Apache Kibble was founded 2017-10-18 (4 years ago)
> > There are currently 15 committers and 13 PMC members in this project.
> > The Committer-to-PMC ratio is roughly 8:7.
> >
> > Community changes, past quarter:
> > - Michał Słowikowski was added to the PMC on 2021-05-03
> > - Tomasz Urbaszek was added to the PMC on 2021-05-03
> > - Kaxil Naik was added as committer on 2021-02-21
> >
> > ## Project Activity:
> > Since the last report we have welcomed a new committer, a new PMC Chair
> and
> > two new PMC members.
> >
> > A new git Kibble repo has been created for the code rewrite and the
> existing
> > repo has been renamed Kibble-1. Current users are being re-directed to
> > Kibble-1 as this is the code source for the Kibble demo.
> >
> > Progress has been slow but we hope to kick off activity again by working
> on a
> > roadmap that will help focus the work. We have had some discussions on
> some
> > potential features of vendor neutrality [3] that could be interesting.
> Also
> > confirmed was the importance of Kibble as crucial part of the ASF
> > reporter.apache.org tool [4]
> >
> > We have received a potential bug report [1] which we are looking into. A
> > Kibble talk [2] has also been submitted for ApacheCon@Home as a basic
> > introduction for projects looking for information about their community.
> >
> > ## Community Health:
> > While development activity has been slow, discussions have been more
> active
> > this quarter as we were waiting on the new repo to be created. Now that
> it is
> > - we need to focus on having a clear roadmap that can be communicated
> within
> > the community.
> >
> > It's good to see that our mailing list activity has increased this
> quarter
> > mainly due to the discussions around planning the re-write.
> >
> > [1] https://s.apache.org/vgnbe
> > [2] https://s.apache.org/rbx97
> > [3] https://s.apache.org/a5iux
> > [4] https://s.apache.org/9tr8h
> >
> > On 2021/05/07 16:44:11, Sharan Foga  wrote:
> > > Hi All
> > >
> > > Our quarterly board report is due this month so I will work on putting
> something together and and then post it here for any community feedback.
> > >
> > > Thanks
> > > Sharan
> > >
>

Re: [ANNOUNCE] Please welcome Michal Slowikowski as a new PMC Member

[Kaxil Naik]

Congratulations Michael 

On Mon, May 3, 2021, 15:11 Sharan Foga  wrote:

> Hi Everyone
>
> I'd like to announce that the PMC has invited Michal to become a member of
> the PMC and once again I am happy to confirm that he has accepted.
>
> Michal has a positive attitude and has been active and collaborative,
> which are key strengths in helping our project grow and evolve.
>
> Please join me in welcoming Michal and congratulating him on his new role.
>
> Thanks
> Sharan
>
>
>

Re: [ANNOUNCE] Please welcome Tomasz Urbaszek as a new PMC Member

[Kaxil Naik]

Congratulations Tomek, very well deserved 

On Mon, May 3, 2021, 15:09 Sharan Foga  wrote:

> Hi Everyone
>
> I'd like to announce that the PMC has invited Tomek to become a member of
> the PMC and I am happy to confirm that he has accepted.
>
> We are keen to grow the project and Tomek has shown his enthusiasm and
> motivation to help us move the project forward.
>
> Please join me in congratulating Tomek on his new role.
>
> Thanks
> Sharan
>
>

Re: New Kibble PMC Chair

[Kaxil Naik]

Congratulations Sharan 

On Sat, May 1, 2021, 15:22 Sharan Foga  wrote:

> Hi All
>
> A quick note to let you know that I have been recently been elected as the
> new PMC Chair for Kibble. (Huge thanks to Rich Bowen as the outgoing Chair
> who has been taking care of the project since it's been a TLP!)
>
> I'd really like to explore and see how we can make the project and
> community grow so I plan to be a bit more active in moving the project
> forward as we work on the roadmap for the new version, as well as
> supporting Kibble-1... so watch out for a few more posts from me :-)
>
> Thanks
> Sharan
>
>

Re: Apache Kibble 2.0

[Kaxil Naik]

Thanks Tomek,

I agree.

Maybe we should start developing new Kibble and do the decision making
> as we go to avoid overhead of excessive planning?


Regards,
Kaxil

On Sat, Mar 6, 2021 at 2:40 PM Tomasz Urbaszek  wrote:

> Hey all,
>
> I'm wondering if anyone had time to review those documents?
>
> Maybe we should start developing new Kibble and do the decision making
> as we go to avoid overhead of excessive planning?
>
> Cheers,
> Tomek
>
>
> On Sat, 27 Feb 2021 at 18:43, Tomasz Urbaszek 
> wrote:
> >
> > After thinking a bit I think we may consider using Graph QL instead of
> REST... I know that Daniel mentioned that we may consider using GQL and now
> (after doing some brainstorming) I think I start understand why. The only
> thing I'm afraid is that GQL has a steep learning curve and it may make it
> harder to grow the community and develop the app. But I'm not sure if this
> should block us from using GQL.
> >
> > T.
> >
> > On Sat, 27 Feb 2021 at 17:07, Tomasz Urbaszek 
> wrote:
> >>
> >> Hi all,
> >>
> >> I prepared a draft of Open API specification that we may use to build
> new Kibble API:
> >> https://app.swaggerhub.com/apis/turbaszek/kibble/1.0.0
> >>
> >> My idea was that inside single kibble deployment, there can be many
> organizations and each organization has its own users and configured
> scanners.
> >>
> >> So this proposal basically has 3 CRUD endpoints for organizations,
> users and scanners. And additionally there's the crucial
> /organization/{organization_id}/scanners/{scanner_id}/data endpoint which
> returns data collected by a given scanner for a given organization.
> >>
> >> The data returned by this endpoint is an object containing an array of
> data points (the real data) and additional information about the structure
> of a single data point. This makes the response little ambiguous but it
> gives us a big elasticity - each scanner can return data in different
> formats. This proposal does not have any filters or aggregation options
> which we definitely would like to add.
> >>
> >> Please let me know what you think! I'm really open to any suggestion ;)
> >>
> >> Cheers,
> >> Tomek
> >>
> >> On Mon, 22 Feb 2021 at 21:10, Fourat ZOUARI  wrote:
> >>>
> >>> Thx Sharan !
> >>>
> >>> I am excited to discover the great work done on kibble, the project
> >>> addresses the need to visualize information around raw source code
> >>> repositories and this is something i've been working on for some years
> now,
> >>> i'm willing to contribute with my experience to this project, as a
> user and
> >>> as a developer.
> >>>
> >>>
> >>> On Sun, Feb 21, 2021 at 10:55 AM Sharan Foga 
> wrote:
> >>>
> >>> > Hi Fourat
> >>> >
> >>> > Welcome to the community! It is great to see you here and thanks
> very much
> >>> > for your contribution :-)
> >>> >
> >>> > Thanks
> >>> > Sharan
> >>> >
> >>> > On 2021/02/20 22:07:36, Fourat ZOUARI  wrote:
> >>> > > Hi Tomek, Sharan, Added some suggests to the doc
> >>> > >
> >>> > > Thx, Fourat
> >>> > >
> >>> > >
> >>> > > On Sat, Feb 20, 2021 at 1:53 PM Tomasz Urbaszek <
> turbas...@apache.org>
> >>> > > wrote:
> >>> > >
> >>> > > > Thanks Sharan!
> >>> > > >
> >>> > > > I already prepared the project structure together with some
> automation
> >>> > > > (pre-commits, CI, etc) on my fork:
> >>> > > > https://github.com/turbaszek/kibble
> >>> > > >
> >>> > > > Best,
> >>> > > > Tomek
> >>> > > >
> >>> > > >
> >>> > > > On Sat, 20 Feb 2021 at 09:57, Sharan Foga 
> wrote:
> >>> > > > >
> >>> > > > > Hi Tomek
> >>> > > > >
> >>> > > > > Great initiative! I will take a look.
> >>> > > > >
> >>> > > > > I am still waiting on feedback from infra regarding the repo
> rename
> >>> > so
> >>> > > > will follow up on that  again today as it would be good to get
> started
> >>> > with
> >>> > > > the new version.
> >>> > > > >
> >>> > > > > And yes - interesting to see popularity increasing :-)
> >>> > > > >
> >>> > > > > Thanks
> >>> > > > > Sharan
> >>> > > > >
> >>> > > > > On 2021/02/20 08:11:10, Tomasz Urbaszek 
> >>> > wrote:
> >>> > > > > > Hi all,
> >>> > > > > >
> >>> > > > > > I drafted some proposal of Kibble 2.0 design decisions:
> >>> > > > > > https://s.apache.org/kibble-2-0
> >>> > > > > >
> >>> > > > > > This is a google docs - I found those much easier to work
> with than
> >>> > > > > > confluence. But once we have something that we accept I will
> move
> >>> > it
> >>> > > > > > to cwiki.
> >>> > > > > >
> >>> > > > > > Please feel free to add all your ideas and concerns!
> >>> > > > > >
> >>> > > > > > I think the crucial point (not covered in doc) is to decide
> if we
> >>> > want
> >>> > > > > > to reuse the existing Kibble database or not. In my opinion,
> >>> > breaking
> >>> > > > > > backward compatibility may improve development speed.
> >>> > > > > >
> >>> > > > > > And as an interesting fact I think Kibble is getting
> interestingly
> >>> > > > "popular":
> >>> > > > > > https://imgur.com/JssRETL
> >>> > > > > >
> >>> > > > > > Cheers,
> 

Re: Renaming Existing kibble Repo to kibble-old

[Kaxil Naik]



On Mon, Mar 1, 2021 at 10:49 AM Sharan Foga  wrote:

> Hi All
>
> The rename is done and a new blank repo is created! Let's get moving :-)
>
> Thanks
> Sharan
>
> On 2021/02/09 20:07:55, Sharan Foga  wrote:
> > Hi Tomek
> >
> > I am separating this out as a new thread as I think it got lost in the
> previous discussion.  So has anything happened regarding this?
> >
> > I took a quick look and I don't have access to the repo settings to do
> the rename so maybe it is something we need to ask infra to do as the
> apache organisation owner.
> >
> > Thanks
> > Sharan
> >
> >
> > -
> >
> >
> > From: Tomasz Urbaszek 
> > Subject: Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
> > Date: 2021/01/16 11:30:56
> > List: dev@kibble.apache.org
> >
> > Hi folks,
> >
> > I created a 2.0 branch in apache/kibble (just for sanity sake if we would
> > like to reuse some code) and I restored the main branch to some commit
> > before breaking the refactor.
> >
> > Can someone of you rename apache/kibble -> apache/kibble-old and then
> > create apache/kibble repo (as discussed earlier)? I don't have the right
> to
> > rename repo.
> >
> > Cheers,
> > Tomek
> >
>

Re: Announcing a New Committer Kaxil Naik

[Kaxil Naik]

Thank you all for the warm welcome.

On Sun, Feb 21, 2021 at 12:53 PM Michał Słowikowski <
michalslowikowsk...@gmail.com> wrote:

> Congrats Kaxil!
>
> Best regards
> Michał
>
> > On 21 Feb 2021, at 10:53, Sharan Foga  wrote:
> >
> > Hi All
> >
> > The Kibble PMC has invited Kaxil Naik to become a committer on the
> project and we are happy to announce that he has accepted.
> >
> > Please join me in congratulating Kaxil on his new role.
> >
> > Thanks
> > Sharan
> > On behalf of the Apache Kibble PMC
>

Re: Apache Kibble 2.0

[Kaxil Naik]

Added my comments to the doc too.

Regards,
Kaxil

On Sat, Feb 20, 2021 at 10:07 PM Fourat ZOUARI  wrote:

> Hi Tomek, Sharan, Added some suggests to the doc
>
> Thx, Fourat
>
>
> On Sat, Feb 20, 2021 at 1:53 PM Tomasz Urbaszek 
> wrote:
>
> > Thanks Sharan!
> >
> > I already prepared the project structure together with some automation
> > (pre-commits, CI, etc) on my fork:
> > https://github.com/turbaszek/kibble
> >
> > Best,
> > Tomek
> >
> >
> > On Sat, 20 Feb 2021 at 09:57, Sharan Foga  wrote:
> > >
> > > Hi Tomek
> > >
> > > Great initiative! I will take a look.
> > >
> > > I am still waiting on feedback from infra regarding the repo rename so
> > will follow up on that  again today as it would be good to get started
> with
> > the new version.
> > >
> > > And yes - interesting to see popularity increasing :-)
> > >
> > > Thanks
> > > Sharan
> > >
> > > On 2021/02/20 08:11:10, Tomasz Urbaszek  wrote:
> > > > Hi all,
> > > >
> > > > I drafted some proposal of Kibble 2.0 design decisions:
> > > > https://s.apache.org/kibble-2-0
> > > >
> > > > This is a google docs - I found those much easier to work with than
> > > > confluence. But once we have something that we accept I will move it
> > > > to cwiki.
> > > >
> > > > Please feel free to add all your ideas and concerns!
> > > >
> > > > I think the crucial point (not covered in doc) is to decide if we
> want
> > > > to reuse the existing Kibble database or not. In my opinion, breaking
> > > > backward compatibility may improve development speed.
> > > >
> > > > And as an interesting fact I think Kibble is getting interestingly
> > "popular":
> > > > https://imgur.com/JssRETL
> > > >
> > > > Cheers,
> > > > Tomek
> > > >
> >
>

Re: Triage role for contributors

[Kaxil Naik]

No feedback yet, still waiting.

On Tue, Feb 9, 2021 at 7:32 PM Sharan Foga  wrote:

> Hi Kaxil
>
> Did you get any feedback on this or are you still waiting?
>
> Thanks
> Sharan
>
> On 2021/01/08 21:59:58, Kaxil Naik  wrote:
> > I can help with some Issue Triage too in the current repo and what will
> > become the Kibble version2 repo.
> >
> > My Github username: kaxil (https://github.com/kaxil/)
> >
> > Regards,
> > Kaxil
> >
> > On 2020/10/11 18:22:33, Sharan Foga  wrote:
> > > +1 from me :-)>
> > >
> > > Thanks>
> > > Sharan>
> > >
> > > On 2020/10/11 18:21:35, Sharan Foga  wrote: >
> > > > Hi All>
> > > > >
> > > > In order to request this role for Tomek, we need to show community
> > approval for the triage role and also reference it in the Infra Jira
> ticket
> > request.>
> > > > >
> > > > So please can you respond if you support giving Tomek a triage role.>
> > > > >
> > > > Thanks>
> > > > Sharan>
> > > > >
> > > > On 2020/10/11 16:45:56, Sharan Foga  wrote: >
> > > > > Hi Tomek>
> > > > > >
> > > > > Thanks for volunteering to help with triage. I took a look at the
> > kibble github repo and I dont have authority to grant access. It sounds
> > like a request to infra so I will request it for you.>
> > > > > >
> > > > > Thanks>
> > > > > Sharan>
> > > > > >
> > > > > On 2020/10/11 11:37:35, Tomasz Urbaszek  wrote:
> >
> > > > > > Hello,>
> > > > > > >
> > > > > > Quite recently Github introduced a new repository access level
> > called>
> > > > > > "triage role":>
> > > > > > https://infra.apache.org/github-roles.html>
> > > > > > >
> > > > > > As described on that page:>
> > > > > > > It is a good role for people who are interested in supporting
> and
> > contributing to the project, but who are not ready to be Committers.>
> > > > > > >
> > > > > > I would like to ask you if it would be possible to grant me this
> > role.>
> > > > > > I'm happy to manage the issues and PRs and ping people for
> > reviews.>
> > > > > > My Github handle is: turbaszek>
> > > > > > >
> > > > > > Best wishes,>
> > > > > > Tomek>
> > > > > > >
> > > > > >
> > > > >
> > >
> >
>

Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021

[Kaxil Naik]

Thanks Sharan

On 2021/01/10 17:10:12, Sharan Foga  wrote:
> Hi Kaxil>
>
> I have copied the meeting notes onto the Kibble Wiki for reference  >
>
>
https://cwiki.apache.org/confluence/display/KIBBLE/Kibble+Development+Call%3A+7th+January+2021
>
>
> with a link to this thread.>
>
> Thanks>
> Sharan>
>
> On 2021/01/08 21:52:32, Kaxil Naik  wrote: >
> > Hi all,>
> > >
> > Here are the notes from our Kibble dev call from yesterday. Please
feel>
> > free to add anything I have missed.>
> > >
> > *Attendees:*>
> > >
> >- Kaxil Naik>
> >- Tomek Urbaszek>
> >- Michał Słowikowski>
> >- Sharan Foga>
> >- Daniel Gruno>
> > >
> > >
> > Here is the summary of the call.>
> > >
> >- *Kibble v2*:>
> >   - Rewrite the implementation from Scratch for v2 to improve code
and>
> >   to cater for new ideas>
> >   - Cherrypick / copy any code from the current Repo>
> >   - New repo allows us to work with a TDD approach. The current
repo>
> >   does not have tests.>
> >   - *Supporting more DBs*>
> >   - For now, we would just support Elasticsearch.>
> >   - Supporting just one DB allows us to maintain it in a better
way>
> >   - Would be good to find some kind of ORM to talk to
Elasticsearch>
> >   which will allow static Schema and use Class attributes instead>
> > of dealing>
> >   with dicts>
> >   - *Migration*>
> >   - Make migration tool or make whatever we build new as>
> >   backwards-compatible so that we don’t need to re-scan all the>
> > data for ASF>
> >   Kibble -- which can take aleast a week if started from scratch.>
> >   (contains 50million records)>
> >- *Scanners*>
> >   - Build Base Scanner for v2 that will allow easily creating new>
> >   Scanners and encourage more contributions>
> >   - Write docs around how to build a new scanner inheriting>
> >   *BaseScanner*>
> >   - Some of the new Scanners that users have requested are:>
> >  - Gitlab>
> >  - Social Media Scanners: Twitter, Discord, Slack>
> >   - Create a Github Issue template for Scanners so users can
request>
> >   new Scanners>
> >- *Dashboard*>
> >   - Do a POC to see if Apache Superset can replace our current UI>
> >- Make Apache Superset or the existing UI optional as some Users
just>
> >   rely on the API>
> >   - That will allow us to focus on the CORE (Scanner, Server and
ES)>
> >   ->
> >   - What happens to the PR that introduces breaking changes??>
> >  - We should not merge that PR until it is a breaking change,>
> >  instead we should make the PR backwards compatible>
> >  - Or wait until next major release (assign appropriate Github>
> >  Milestone)>
> > >
> > >
> > Let me know if anyone has any thoughts on any of the items listed
above.>
> > >
> > Best regards,>
> > Kaxil>
> > >
>

Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021

[Kaxil Naik]

Please ignore the last bullet around "breaking changes" -- I mixed that up
with the Airflow Dev call summary ;)

Regards,
Kaxil

On Fri, Jan 8, 2021 at 9:52 PM Kaxil Naik  wrote:

> Hi all,
>
> Here are the notes from our Kibble dev call from yesterday. Please feel
> free to add anything I have missed.
>
> *Attendees:*
>
>- Kaxil Naik
>- Tomek Urbaszek
>- Michał Słowikowski
>- Sharan Foga
>- Daniel Gruno
>
>
> Here is the summary of the call.
>
>- *Kibble v2*:
>   - Rewrite the implementation from Scratch for v2 to improve code
>   and to cater for new ideas
>   - Cherrypick / copy any code from the current Repo
>   - New repo allows us to work with a TDD approach. The current repo
>   does not have tests.
>   - *Supporting more DBs*
>   - For now, we would just support Elasticsearch.
>   - Supporting just one DB allows us to maintain it in a better way
>   - Would be good to find some kind of ORM to talk to Elasticsearch
>   which will allow static Schema and use Class attributes instead of 
> dealing
>   with dicts
>   - *Migration*
>   - Make migration tool or make whatever we build new as
>   backwards-compatible so that we don’t need to re-scan all the data for 
> ASF
>   Kibble -- which can take aleast a week if started from scratch.
>   (contains 50million records)
>- *Scanners*
>   - Build Base Scanner for v2 that will allow easily creating new
>   Scanners and encourage more contributions
>   - Write docs around how to build a new scanner inheriting
>   *BaseScanner*
>   - Some of the new Scanners that users have requested are:
>  - Gitlab
>  - Social Media Scanners: Twitter, Discord, Slack
>   - Create a Github Issue template for Scanners so users can request
>   new Scanners
>- *Dashboard*
>   - Do a POC to see if Apache Superset can replace our current UI
>- Make Apache Superset or the existing UI optional as some Users just
>   rely on the API
>   - That will allow us to focus on the CORE (Scanner, Server and ES)
>   -
>   - What happens to the PR that introduces breaking changes??
>  - We should not merge that PR until it is a breaking change,
>  instead we should make the PR backwards compatible
>  - Or wait until next major release (assign appropriate Github
>  Milestone)
>
>
> Let me know if anyone has any thoughts on any of the items listed above.
>
> Best regards,
> Kaxil
>

[Meeting Notes] Kibble Dev Call - 7 Jan 2021

[Kaxil Naik]

Hi all,

Here are the notes from our Kibble dev call from yesterday. Please feel
free to add anything I have missed.

*Attendees:*

   - Kaxil Naik
   - Tomek Urbaszek
   - Michał Słowikowski
   - Sharan Foga
   - Daniel Gruno


Here is the summary of the call.

   - *Kibble v2*:
  - Rewrite the implementation from Scratch for v2 to improve code and
  to cater for new ideas
  - Cherrypick / copy any code from the current Repo
  - New repo allows us to work with a TDD approach. The current repo
  does not have tests.
  - *Supporting more DBs*
  - For now, we would just support Elasticsearch.
  - Supporting just one DB allows us to maintain it in a better way
  - Would be good to find some kind of ORM to talk to Elasticsearch
  which will allow static Schema and use Class attributes instead
of dealing
  with dicts
  - *Migration*
  - Make migration tool or make whatever we build new as
  backwards-compatible so that we don’t need to re-scan all the
data for ASF
  Kibble -- which can take aleast a week if started from scratch.
  (contains 50million records)
   - *Scanners*
  - Build Base Scanner for v2 that will allow easily creating new
  Scanners and encourage more contributions
  - Write docs around how to build a new scanner inheriting
  *BaseScanner*
  - Some of the new Scanners that users have requested are:
 - Gitlab
 - Social Media Scanners: Twitter, Discord, Slack
  - Create a Github Issue template for Scanners so users can request
  new Scanners
   - *Dashboard*
  - Do a POC to see if Apache Superset can replace our current UI
   - Make Apache Superset or the existing UI optional as some Users just
  rely on the API
  - That will allow us to focus on the CORE (Scanner, Server and ES)
  -
  - What happens to the PR that introduces breaking changes??
 - We should not merge that PR until it is a breaking change,
 instead we should make the PR backwards compatible
 - Or wait until next major release (assign appropriate Github
 Milestone)


Let me know if anyone has any thoughts on any of the items listed above.

Best regards,
Kaxil

Re: Dev Call to discuss future Roadmap for Apache Kibble in 2021

[Kaxil Naik]

I have sent the invite to dev list and to you all individually too.

Regards,
Kaxil

On Thu, Dec 24, 2020 at 5:37 PM Tomasz Urbaszek 
wrote:

> Great! FYI the calendar link seems to not work :<
>
> Tomek
>
> On Thu, Dec 24, 2020 at 6:34 PM Kaxil Naik  wrote:
>
> > Let's do it then :) -- Looks like Thursday - Jan 7, 2021 at 2-3 PM GMT is
> > suitable for all those who replied on Doodle.
> >
> > Call Link: https://meet.apache.org/BetterHalvesSenseAround
> > Event:
> >
> >
> https://calendar.google.com/event?action=TEMPLATE=NGwxYzlxajMzNmo0ZGloNW9mZm8xbDMzNzAga2F4aWxAYXN0cm9ub21lci5pbw=kaxil%40astronomer.io
> >
> > Regards,
> > Kaxil
> >
> > On Mon, Dec 21, 2020 at 9:26 AM Daniel Gruno 
> wrote:
> >
> > > Good initiative :), I've put my availability in the doodle.
> > >
> > > I'm leaning towards a complete refactoring/rewrite, a "kibble v2" where
> > > we use the general architecture/design of Kibble but write the python
> > > logic from scratch.
> > >
> > > On 20/12/2020 22.55, Kaxil Naik wrote:
> > > > Hi folks,
> > > >
> > > > I was talking to Daniel and Tomek on Slack that it would be awesome
> to
> > > have
> > > > a DEV call to discuss the future of Apache Kibble.
> > > >
> > > > I am one of the committer, PMC Member and Release Manager of the
> Apache
> > > > Airflow project. I organized regular dev calls for Airflow until we
> > > > released Airflow 2.0 last week. And it really helped with aligning
> the
> > > > goals, scoping the items for the release etc.
> > > >
> > > > I really think this would also be useful for Apache Kibble. I
> recently
> > > > started contributing to Kibble (commits
> > > > <https://github.com/apache/kibble/commits?author=kaxil>) and looks
> > like
> > > it
> > > > needs more love and a bit of refactor. Whether we want to refactor
> > > existing
> > > > codebase or start fresh can be discussed on the call.
> > > >
> > > > I propose let's start with one call and then we will see if a regular
> > > call
> > > > is needed or we can do async.
> > > >
> > > > Following is the link of the doodle I created, please add a vote to
> > what
> > > > day and time suits you best in the new year.
> > > >
> > > > Doodle: https://doodle.com/poll/g39kwv6e87gcvfst
> > > >
> > > > Let me know what you all think.
> > > >
> > > > Regards,
> > > > Kaxil
> > > >
> > >
> > >
> >
>

Re: Dev Call to discuss future Roadmap for Apache Kibble in 2021

[Kaxil Naik]

Let's do it then :) -- Looks like Thursday - Jan 7, 2021 at 2-3 PM GMT is
suitable for all those who replied on Doodle.

Call Link: https://meet.apache.org/BetterHalvesSenseAround
Event:
https://calendar.google.com/event?action=TEMPLATE=NGwxYzlxajMzNmo0ZGloNW9mZm8xbDMzNzAga2F4aWxAYXN0cm9ub21lci5pbw=kaxil%40astronomer.io

Regards,
Kaxil

On Mon, Dec 21, 2020 at 9:26 AM Daniel Gruno  wrote:

> Good initiative :), I've put my availability in the doodle.
>
> I'm leaning towards a complete refactoring/rewrite, a "kibble v2" where
> we use the general architecture/design of Kibble but write the python
> logic from scratch.
>
> On 20/12/2020 22.55, Kaxil Naik wrote:
> > Hi folks,
> >
> > I was talking to Daniel and Tomek on Slack that it would be awesome to
> have
> > a DEV call to discuss the future of Apache Kibble.
> >
> > I am one of the committer, PMC Member and Release Manager of the Apache
> > Airflow project. I organized regular dev calls for Airflow until we
> > released Airflow 2.0 last week. And it really helped with aligning the
> > goals, scoping the items for the release etc.
> >
> > I really think this would also be useful for Apache Kibble. I recently
> > started contributing to Kibble (commits
> > <https://github.com/apache/kibble/commits?author=kaxil>) and looks like
> it
> > needs more love and a bit of refactor. Whether we want to refactor
> existing
> > codebase or start fresh can be discussed on the call.
> >
> > I propose let's start with one call and then we will see if a regular
> call
> > is needed or we can do async.
> >
> > Following is the link of the doodle I created, please add a vote to what
> > day and time suits you best in the new year.
> >
> > Doodle: https://doodle.com/poll/g39kwv6e87gcvfst
> >
> > Let me know what you all think.
> >
> > Regards,
> > Kaxil
> >
>
>

Dev Call to discuss future Roadmap for Apache Kibble in 2021

[Kaxil Naik]

Hi folks,

I was talking to Daniel and Tomek on Slack that it would be awesome to have
a DEV call to discuss the future of Apache Kibble.

I am one of the committer, PMC Member and Release Manager of the Apache
Airflow project. I organized regular dev calls for Airflow until we
released Airflow 2.0 last week. And it really helped with aligning the
goals, scoping the items for the release etc.

I really think this would also be useful for Apache Kibble. I recently
started contributing to Kibble (commits
) and looks like it
needs more love and a bit of refactor. Whether we want to refactor existing
codebase or start fresh can be discussed on the call.

I propose let's start with one call and then we will see if a regular call
is needed or we can do async.

Following is the link of the doodle I created, please add a vote to what
day and time suits you best in the new year.

Doodle: https://doodle.com/poll/g39kwv6e87gcvfst

Let me know what you all think.

Regards,
Kaxil