Hi Tomek Thanks for working on this.
I like the idea. From the first time we had the demo running and showed the list of possible sources - someone immediately asked about adding one that we didn't have, so I think making adding them in a simple yet flexible way will be a good thing! Thanks Sharan On 2021/05/22 13:50:55, Tomasz Urbaszek <turbas...@apache.org> wrote: > Finally I started working on new Kibble: > https://github.com/apache/kibble/pull/8 > > I would like to propose introducing the following new concepts in new Apache > Kibble: > > 1. DataSource - Github, Gitlab, Jira etc. - each data source would be a > python package that would include logic for getting data from the external > source as well as some metric and authentication mechanisms. > > 2. DataType - in case of Github: issues, PRs, commits etc - each DataSource > can define its own data types. Single data type should define methods for > getting relevant data and way to persist them in Kibble - this would replace > the concept of scanners. > > Why? In this way Apache Kibble main aim and value would be building a > framework for collecting projects information and serving them in useful, > aggregated way. Having such a framework would make adding new data sources as > simple as plug and play, thus allowing easy extension and possibility to > build custom data source. > > Let me know what do you think. I will definitely try to play with this idea > in my PR. > > Thanks, > Tomek > > > > On 2021/03/22 19:49:28, Fourat ZOUARI <fou...@gmail.com> wrote: > > Hey Sharan, > > > > I've already commented on some parts of the doc and to date, it seems > > fine/sufficient for kicking off the project, more reflections may come over > > through the process. > > > > I suggest we have a kanban board on gh containing all pending tasks as a > > projection for the project. > > > > Fourat > > > > > > On Mon, Mar 22, 2021 at 7:10 PM Sharan Foga <sha...@apache.org> wrote: > > > > > Hi All > > > > > > Does anyone have any feedback on the work Tomek has done around this so > > > far? Are people happy with the suggestions or do you have anything else in > > > mind? > > > > > > Thanks > > > Sharan > > > > > > On 2021/03/06 14:39:43, Tomasz Urbaszek <turbas...@apache.org> wrote: > > > > Hey all, > > > > > > > > I'm wondering if anyone had time to review those documents? > > > > > > > > Maybe we should start developing new Kibble and do the decision making > > > > as we go to avoid overhead of excessive planning? > > > > > > > > Cheers, > > > > Tomek > > > > > > > > > > > > On Sat, 27 Feb 2021 at 18:43, Tomasz Urbaszek <turbas...@apache.org> > > > wrote: > > > > > > > > > > After thinking a bit I think we may consider using Graph QL instead of > > > REST... I know that Daniel mentioned that we may consider using GQL and > > > now > > > (after doing some brainstorming) I think I start understand why. The only > > > thing I'm afraid is that GQL has a steep learning curve and it may make it > > > harder to grow the community and develop the app. But I'm not sure if this > > > should block us from using GQL. > > > > > > > > > > T. > > > > > > > > > > On Sat, 27 Feb 2021 at 17:07, Tomasz Urbaszek <turbas...@apache.org> > > > wrote: > > > > >> > > > > >> Hi all, > > > > >> > > > > >> I prepared a draft of Open API specification that we may use to build > > > new Kibble API: > > > > >> https://app.swaggerhub.com/apis/turbaszek/kibble/1.0.0 > > > > >> > > > > >> My idea was that inside single kibble deployment, there can be many > > > organizations and each organization has its own users and configured > > > scanners. > > > > >> > > > > >> So this proposal basically has 3 CRUD endpoints for organizations, > > > users and scanners. And additionally there's the crucial > > > /organization/{organization_id}/scanners/{scanner_id}/data endpoint which > > > returns data collected by a given scanner for a given organization. > > > > >> > > > > >> The data returned by this endpoint is an object containing an array > > > of data points (the real data) and additional information about the > > > structure of a single data point. This makes the response little ambiguous > > > but it gives us a big elasticity - each scanner can return data in > > > different formats. This proposal does not have any filters or aggregation > > > options which we definitely would like to add. > > > > >> > > > > >> Please let me know what you think! I'm really open to any suggestion > > > ;) > > > > >> > > > > >> Cheers, > > > > >> Tomek > > > > >> > > > > >> On Mon, 22 Feb 2021 at 21:10, Fourat ZOUARI <fou...@gmail.com> wrote: > > > > >>> > > > > >>> Thx Sharan ! > > > > >>> > > > > >>> I am excited to discover the great work done on kibble, the project > > > > >>> addresses the need to visualize information around raw source code > > > > >>> repositories and this is something i've been working on for some > > > years now, > > > > >>> i'm willing to contribute with my experience to this project, as a > > > user and > > > > >>> as a developer. > > > > >>> > > > > >>> > > > > >>> On Sun, Feb 21, 2021 at 10:55 AM Sharan Foga <sha...@apache.org> > > > wrote: > > > > >>> > > > > >>> > Hi Fourat > > > > >>> > > > > > >>> > Welcome to the community! It is great to see you here and thanks > > > very much > > > > >>> > for your contribution :-) > > > > >>> > > > > > >>> > Thanks > > > > >>> > Sharan > > > > >>> > > > > > >>> > On 2021/02/20 22:07:36, Fourat ZOUARI <fou...@gmail.com> wrote: > > > > >>> > > Hi Tomek, Sharan, Added some suggests to the doc > > > > >>> > > > > > > >>> > > Thx, Fourat > > > > >>> > > > > > > >>> > > > > > > >>> > > On Sat, Feb 20, 2021 at 1:53 PM Tomasz Urbaszek < > > > turbas...@apache.org> > > > > >>> > > wrote: > > > > >>> > > > > > > >>> > > > Thanks Sharan! > > > > >>> > > > > > > > >>> > > > I already prepared the project structure together with some > > > automation > > > > >>> > > > (pre-commits, CI, etc) on my fork: > > > > >>> > > > https://github.com/turbaszek/kibble > > > > >>> > > > > > > > >>> > > > Best, > > > > >>> > > > Tomek > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > On Sat, 20 Feb 2021 at 09:57, Sharan Foga <sha...@apache.org> > > > wrote: > > > > >>> > > > > > > > > >>> > > > > Hi Tomek > > > > >>> > > > > > > > > >>> > > > > Great initiative! I will take a look. > > > > >>> > > > > > > > > >>> > > > > I am still waiting on feedback from infra regarding the repo > > > rename > > > > >>> > so > > > > >>> > > > will follow up on that again today as it would be good to get > > > started > > > > >>> > with > > > > >>> > > > the new version. > > > > >>> > > > > > > > > >>> > > > > And yes - interesting to see popularity increasing :-) > > > > >>> > > > > > > > > >>> > > > > Thanks > > > > >>> > > > > Sharan > > > > >>> > > > > > > > > >>> > > > > On 2021/02/20 08:11:10, Tomasz Urbaszek < > > > turbas...@apache.org> > > > > >>> > wrote: > > > > >>> > > > > > Hi all, > > > > >>> > > > > > > > > > >>> > > > > > I drafted some proposal of Kibble 2.0 design decisions: > > > > >>> > > > > > https://s.apache.org/kibble-2-0 > > > > >>> > > > > > > > > > >>> > > > > > This is a google docs - I found those much easier to work > > > with than > > > > >>> > > > > > confluence. But once we have something that we accept I > > > will move > > > > >>> > it > > > > >>> > > > > > to cwiki. > > > > >>> > > > > > > > > > >>> > > > > > Please feel free to add all your ideas and concerns! > > > > >>> > > > > > > > > > >>> > > > > > I think the crucial point (not covered in doc) is to > > > decide if we > > > > >>> > want > > > > >>> > > > > > to reuse the existing Kibble database or not. In my > > > opinion, > > > > >>> > breaking > > > > >>> > > > > > backward compatibility may improve development speed. > > > > >>> > > > > > > > > > >>> > > > > > And as an interesting fact I think Kibble is getting > > > interestingly > > > > >>> > > > "popular": > > > > >>> > > > > > https://imgur.com/JssRETL > > > > >>> > > > > > > > > > >>> > > > > > Cheers, > > > > >>> > > > > > Tomek > > > > >>> > > > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > > > > > >
