Please ignore the last bullet around "breaking changes" -- I mixed that up
with the Airflow Dev call summary ;)
Regards,
Kaxil
On Fri, Jan 8, 2021 at 9:52 PM Kaxil Naik wrote:
> Hi all,
>
> Here are the notes from our Kibble dev call from yesterday. Please feel
> free to add anything I have missed.
>
> *Attendees:*
>
>- Kaxil Naik
>- Tomek Urbaszek
>- Michał Słowikowski
>- Sharan Foga
>- Daniel Gruno
>
>
> Here is the summary of the call.
>
>- *Kibble v2*:
> - Rewrite the implementation from Scratch for v2 to improve code
> and to cater for new ideas
> - Cherrypick / copy any code from the current Repo
> - New repo allows us to work with a TDD approach. The current repo
> does not have tests.
> - *Supporting more DBs*
> - For now, we would just support Elasticsearch.
> - Supporting just one DB allows us to maintain it in a better way
> - Would be good to find some kind of ORM to talk to Elasticsearch
> which will allow static Schema and use Class attributes instead of
> dealing
> with dicts
> - *Migration*
> - Make migration tool or make whatever we build new as
> backwards-compatible so that we don’t need to re-scan all the data for
> ASF
> Kibble -- which can take aleast a week if started from scratch.
> (contains 50million records)
>- *Scanners*
> - Build Base Scanner for v2 that will allow easily creating new
> Scanners and encourage more contributions
> - Write docs around how to build a new scanner inheriting
> *BaseScanner*
> - Some of the new Scanners that users have requested are:
> - Gitlab
> - Social Media Scanners: Twitter, Discord, Slack
> - Create a Github Issue template for Scanners so users can request
> new Scanners
>- *Dashboard*
> - Do a POC to see if Apache Superset can replace our current UI
>- Make Apache Superset or the existing UI optional as some Users just
> rely on the API
> - That will allow us to focus on the CORE (Scanner, Server and ES)
> -
> - What happens to the PR that introduces breaking changes??
> - We should not merge that PR until it is a breaking change,
> instead we should make the PR backwards compatible
> - Or wait until next major release (assign appropriate Github
> Milestone)
>
>
> Let me know if anyone has any thoughts on any of the items listed above.
>
> Best regards,
> Kaxil
>
