Finally I started working on new Kibble: https://github.com/apache/kibble/pull/8
I would like to propose introducing the following new concepts in new Apache Kibble: 1. DataSource - Github, Gitlab, Jira etc. - each data source would be a python package that would include logic for getting data from the external source as well as some metric and authentication mechanisms. 2. DataType - in case of Github: issues, PRs, commits etc - each DataSource can define its own data types. Single data type should define methods for getting relevant data and way to persist them in Kibble - this would replace the concept of scanners. Why? In this way Apache Kibble main aim and value would be building a framework for collecting projects information and serving them in useful, aggregated way. Having such a framework would make adding new data sources as simple as plug and play, thus allowing easy extension and possibility to build custom data source. Let me know what do you think. I will definitely try to play with this idea in my PR. Thanks, Tomek On 2021/03/22 19:49:28, Fourat ZOUARI <fou...@gmail.com> wrote: > Hey Sharan, > > I've already commented on some parts of the doc and to date, it seems > fine/sufficient for kicking off the project, more reflections may come over > through the process. > > I suggest we have a kanban board on gh containing all pending tasks as a > projection for the project. > > Fourat > > > On Mon, Mar 22, 2021 at 7:10 PM Sharan Foga <sha...@apache.org> wrote: > > > Hi All > > > > Does anyone have any feedback on the work Tomek has done around this so > > far? Are people happy with the suggestions or do you have anything else in > > mind? > > > > Thanks > > Sharan > > > > On 2021/03/06 14:39:43, Tomasz Urbaszek <turbas...@apache.org> wrote: > > > Hey all, > > > > > > I'm wondering if anyone had time to review those documents? > > > > > > Maybe we should start developing new Kibble and do the decision making > > > as we go to avoid overhead of excessive planning? > > > > > > Cheers, > > > Tomek > > > > > > > > > On Sat, 27 Feb 2021 at 18:43, Tomasz Urbaszek <turbas...@apache.org> > > wrote: > > > > > > > > After thinking a bit I think we may consider using Graph QL instead of > > REST... I know that Daniel mentioned that we may consider using GQL and now > > (after doing some brainstorming) I think I start understand why. The only > > thing I'm afraid is that GQL has a steep learning curve and it may make it > > harder to grow the community and develop the app. But I'm not sure if this > > should block us from using GQL. > > > > > > > > T. > > > > > > > > On Sat, 27 Feb 2021 at 17:07, Tomasz Urbaszek <turbas...@apache.org> > > wrote: > > > >> > > > >> Hi all, > > > >> > > > >> I prepared a draft of Open API specification that we may use to build > > new Kibble API: > > > >> https://app.swaggerhub.com/apis/turbaszek/kibble/1.0.0 > > > >> > > > >> My idea was that inside single kibble deployment, there can be many > > organizations and each organization has its own users and configured > > scanners. > > > >> > > > >> So this proposal basically has 3 CRUD endpoints for organizations, > > users and scanners. And additionally there's the crucial > > /organization/{organization_id}/scanners/{scanner_id}/data endpoint which > > returns data collected by a given scanner for a given organization. > > > >> > > > >> The data returned by this endpoint is an object containing an array > > of data points (the real data) and additional information about the > > structure of a single data point. This makes the response little ambiguous > > but it gives us a big elasticity - each scanner can return data in > > different formats. This proposal does not have any filters or aggregation > > options which we definitely would like to add. > > > >> > > > >> Please let me know what you think! I'm really open to any suggestion > > ;) > > > >> > > > >> Cheers, > > > >> Tomek > > > >> > > > >> On Mon, 22 Feb 2021 at 21:10, Fourat ZOUARI <fou...@gmail.com> wrote: > > > >>> > > > >>> Thx Sharan ! > > > >>> > > > >>> I am excited to discover the great work done on kibble, the project > > > >>> addresses the need to visualize information around raw source code > > > >>> repositories and this is something i've been working on for some > > years now, > > > >>> i'm willing to contribute with my experience to this project, as a > > user and > > > >>> as a developer. > > > >>> > > > >>> > > > >>> On Sun, Feb 21, 2021 at 10:55 AM Sharan Foga <sha...@apache.org> > > wrote: > > > >>> > > > >>> > Hi Fourat > > > >>> > > > > >>> > Welcome to the community! It is great to see you here and thanks > > very much > > > >>> > for your contribution :-) > > > >>> > > > > >>> > Thanks > > > >>> > Sharan > > > >>> > > > > >>> > On 2021/02/20 22:07:36, Fourat ZOUARI <fou...@gmail.com> wrote: > > > >>> > > Hi Tomek, Sharan, Added some suggests to the doc > > > >>> > > > > > >>> > > Thx, Fourat > > > >>> > > > > > >>> > > > > > >>> > > On Sat, Feb 20, 2021 at 1:53 PM Tomasz Urbaszek < > > turbas...@apache.org> > > > >>> > > wrote: > > > >>> > > > > > >>> > > > Thanks Sharan! > > > >>> > > > > > > >>> > > > I already prepared the project structure together with some > > automation > > > >>> > > > (pre-commits, CI, etc) on my fork: > > > >>> > > > https://github.com/turbaszek/kibble > > > >>> > > > > > > >>> > > > Best, > > > >>> > > > Tomek > > > >>> > > > > > > >>> > > > > > > >>> > > > On Sat, 20 Feb 2021 at 09:57, Sharan Foga <sha...@apache.org> > > wrote: > > > >>> > > > > > > > >>> > > > > Hi Tomek > > > >>> > > > > > > > >>> > > > > Great initiative! I will take a look. > > > >>> > > > > > > > >>> > > > > I am still waiting on feedback from infra regarding the repo > > rename > > > >>> > so > > > >>> > > > will follow up on that again today as it would be good to get > > started > > > >>> > with > > > >>> > > > the new version. > > > >>> > > > > > > > >>> > > > > And yes - interesting to see popularity increasing :-) > > > >>> > > > > > > > >>> > > > > Thanks > > > >>> > > > > Sharan > > > >>> > > > > > > > >>> > > > > On 2021/02/20 08:11:10, Tomasz Urbaszek < > > turbas...@apache.org> > > > >>> > wrote: > > > >>> > > > > > Hi all, > > > >>> > > > > > > > > >>> > > > > > I drafted some proposal of Kibble 2.0 design decisions: > > > >>> > > > > > https://s.apache.org/kibble-2-0 > > > >>> > > > > > > > > >>> > > > > > This is a google docs - I found those much easier to work > > with than > > > >>> > > > > > confluence. But once we have something that we accept I > > will move > > > >>> > it > > > >>> > > > > > to cwiki. > > > >>> > > > > > > > > >>> > > > > > Please feel free to add all your ideas and concerns! > > > >>> > > > > > > > > >>> > > > > > I think the crucial point (not covered in doc) is to > > decide if we > > > >>> > want > > > >>> > > > > > to reuse the existing Kibble database or not. In my > > opinion, > > > >>> > breaking > > > >>> > > > > > backward compatibility may improve development speed. > > > >>> > > > > > > > > >>> > > > > > And as an interesting fact I think Kibble is getting > > interestingly > > > >>> > > > "popular": > > > >>> > > > > > https://imgur.com/JssRETL > > > >>> > > > > > > > > >>> > > > > > Cheers, > > > >>> > > > > > Tomek > > > >>> > > > > > > > > >>> > > > > > > >>> > > > > > >>> > > > > > > >
