Setting Up Github Project Board for Kibble Foundations work
Hi All So as Tomek mentioned in his post [1] - we can track our potential release versions using the milestones functionality. We can use the project board functionality to help us manage and group the different pieces of work. Tomek mentioned 4 areas [2] that we could use as a high level grouping to organise the work. These were Foundations, Additional Critical Data Sources, API and UI, each of which build on each other. So let's start with setting up a board for the first one - the Apache Kibble Foundations work: Suggested objectives so far are: 1, Setup up scanner for a limited set of datasources (E.g If we fix this at 2 then we could have Github and Jira) 2. Make data accessible via a Database (From our call I think we are starting with Elasticsearch - is this still valid?) 3. Setup some basic metrics (We need to decide how basic this is - maybe it's simply number of records or aggregations or we can take a look at Kibble-1 and select something simple but useful) 4. Linking Apache Superset as a dashboard (We need to do some investigation of how to integrate this and also connect with the Superset community to see if they will be willing to help us in this effort) We need to break these down into more detailed tasks and the plan is that we can see more easily what the key areas we need to focus on. If we find things are not working the we can adjust and evolve as needed. Happy to get any suggestions for any of the points mentioned above as well as other comments or feedback. Thanks Sharan [1] https://s.apache.org/a9b6r [2] https://s.apache.org/02xud
Re: Have we ever made a Kibble release?
Excellent! :-) On 2021/05/23 14:29:43, Kaxil Naik wrote: > Yup, this is what we do at Apache Airflow too. > > And I did the same for the recently released Apache Airflow Helm Chart. > > Example: > https://lists.apache.org/x/thread.html/rd2a0a6386c22bb0942c28746d59d105adc0364715c15ea772e23e105@%3Cdev.airflow.apache.org%3E > > I have releasing Airflow versions from past 3 or so years, so have good > experience with ASF guidelines around it. > > Regards, > Kaxil > > > > On Sun, May 23, 2021, 15:24 Sharan Foga wrote: > > > > > > > On 2021/05/23 14:15:32, Kaxil Naik wrote: > > > Yeah, as far as I know we don't have a release. For Kibble 2 we can > > release > > > Python wheel/binary artifacts. > > > > I know there is some discussion going on about binary artifacts and Apache > > releases as generally Apache releases need to be source code, though they > > have been supplemented with binary artifacts. We just need to make sure we > > adhere to the current policy. > > > > Thanks > > Sharan > > > > > > > > Regards, > > > Kaxil > > > > > > On Sun, May 23, 2021, 15:10 Tomasz Urbaszek > > wrote: > > > > > > > As far as I know Apache Kibble was never released. I would propose > > > > using Github milestones to track future releases of new Kibble: > > > > https://github.com/apache/kibble/milestone/1 > > > > > > > > Tomek > > > > > > > > On Sun, 23 May 2021 at 16:00, Sharan Foga wrote: > > > > > > > > > > Hi All > > > > > > > > > > Thanks to all who helped review our Board report. We got some > > feedback > > > > comment to include details of our last release in our next report. I've > > > > been searching through our mailing archives and can't find any details > > or > > > > Vote of the project ever doing a release. > > > > > > > > > > If this is the case then we need to think about planning one as part > > of > > > > our roadmap. > > > > > > > > > > Thanks > > > > > Sharan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: Have we ever made a Kibble release?
Yup, this is what we do at Apache Airflow too. And I did the same for the recently released Apache Airflow Helm Chart. Example: https://lists.apache.org/x/thread.html/rd2a0a6386c22bb0942c28746d59d105adc0364715c15ea772e23e105@%3Cdev.airflow.apache.org%3E I have releasing Airflow versions from past 3 or so years, so have good experience with ASF guidelines around it. Regards, Kaxil On Sun, May 23, 2021, 15:24 Sharan Foga wrote: > > > On 2021/05/23 14:15:32, Kaxil Naik wrote: > > Yeah, as far as I know we don't have a release. For Kibble 2 we can > release > > Python wheel/binary artifacts. > > I know there is some discussion going on about binary artifacts and Apache > releases as generally Apache releases need to be source code, though they > have been supplemented with binary artifacts. We just need to make sure we > adhere to the current policy. > > Thanks > Sharan > > > > > Regards, > > Kaxil > > > > On Sun, May 23, 2021, 15:10 Tomasz Urbaszek > wrote: > > > > > As far as I know Apache Kibble was never released. I would propose > > > using Github milestones to track future releases of new Kibble: > > > https://github.com/apache/kibble/milestone/1 > > > > > > Tomek > > > > > > On Sun, 23 May 2021 at 16:00, Sharan Foga wrote: > > > > > > > > Hi All > > > > > > > > Thanks to all who helped review our Board report. We got some > feedback > > > comment to include details of our last release in our next report. I've > > > been searching through our mailing archives and can't find any details > or > > > Vote of the project ever doing a release. > > > > > > > > If this is the case then we need to think about planning one as part > of > > > our roadmap. > > > > > > > > Thanks > > > > Sharan > > > > > > > > > > > > > > > > > > > > > >
Re: Have we ever made a Kibble release?
On 2021/05/23 14:15:32, Kaxil Naik wrote: > Yeah, as far as I know we don't have a release. For Kibble 2 we can release > Python wheel/binary artifacts. I know there is some discussion going on about binary artifacts and Apache releases as generally Apache releases need to be source code, though they have been supplemented with binary artifacts. We just need to make sure we adhere to the current policy. Thanks Sharan > > Regards, > Kaxil > > On Sun, May 23, 2021, 15:10 Tomasz Urbaszek wrote: > > > As far as I know Apache Kibble was never released. I would propose > > using Github milestones to track future releases of new Kibble: > > https://github.com/apache/kibble/milestone/1 > > > > Tomek > > > > On Sun, 23 May 2021 at 16:00, Sharan Foga wrote: > > > > > > Hi All > > > > > > Thanks to all who helped review our Board report. We got some feedback > > comment to include details of our last release in our next report. I've > > been searching through our mailing archives and can't find any details or > > Vote of the project ever doing a release. > > > > > > If this is the case then we need to think about planning one as part of > > our roadmap. > > > > > > Thanks > > > Sharan > > > > > > > > > > > > > > >
Re: Have we ever made a Kibble release?
On 2021/05/23 14:09:47, Tomasz Urbaszek wrote: > As far as I know Apache Kibble was never released. I would propose > using Github milestones to track future releases of new Kibble: > https://github.com/apache/kibble/milestone/1 +1 > > Tomek > > On Sun, 23 May 2021 at 16:00, Sharan Foga wrote: > > > > Hi All > > > > Thanks to all who helped review our Board report. We got some feedback > > comment to include details of our last release in our next report. I've > > been searching through our mailing archives and can't find any details or > > Vote of the project ever doing a release. > > > > If this is the case then we need to think about planning one as part of our > > roadmap. > > > > Thanks > > Sharan > > > > > > > > >
Re: Have we ever made a Kibble release?
Yeah, as far as I know we don't have a release. For Kibble 2 we can release Python wheel/binary artifacts. Regards, Kaxil On Sun, May 23, 2021, 15:10 Tomasz Urbaszek wrote: > As far as I know Apache Kibble was never released. I would propose > using Github milestones to track future releases of new Kibble: > https://github.com/apache/kibble/milestone/1 > > Tomek > > On Sun, 23 May 2021 at 16:00, Sharan Foga wrote: > > > > Hi All > > > > Thanks to all who helped review our Board report. We got some feedback > comment to include details of our last release in our next report. I've > been searching through our mailing archives and can't find any details or > Vote of the project ever doing a release. > > > > If this is the case then we need to think about planning one as part of > our roadmap. > > > > Thanks > > Sharan > > > > > > > > >
Re: Have we ever made a Kibble release?
On 23/05/2021 16.00, Sharan Foga wrote: Hi All Thanks to all who helped review our Board report. We got some feedback comment to include details of our last release in our next report. I've been searching through our mailing archives and can't find any details or Vote of the project ever doing a release. If this is the case then we need to think about planning one as part of our roadmap. We never hit a point where a release was feasible. So no, no releases ever, just like Whimsy and STeVe ;) Thanks Sharan
Re: Have we ever made a Kibble release?
As far as I know Apache Kibble was never released. I would propose using Github milestones to track future releases of new Kibble: https://github.com/apache/kibble/milestone/1 Tomek On Sun, 23 May 2021 at 16:00, Sharan Foga wrote: > > Hi All > > Thanks to all who helped review our Board report. We got some feedback > comment to include details of our last release in our next report. I've been > searching through our mailing archives and can't find any details or Vote of > the project ever doing a release. > > If this is the case then we need to think about planning one as part of our > roadmap. > > Thanks > Sharan > > > >
Have we ever made a Kibble release?
Hi All Thanks to all who helped review our Board report. We got some feedback comment to include details of our last release in our next report. I've been searching through our mailing archives and can't find any details or Vote of the project ever doing a release. If this is the case then we need to think about planning one as part of our roadmap. Thanks Sharan
Re: [DISCUSSION] Kibble Roadmap
Hi Tomek Thanks for the feedback. I've included some comments inline. On 2021/05/12 08:37:56, Tomasz Urbaszek wrote: > > 1) How far do we support Kibble-1 while the new Kibble is being developed? > > To be honest I think we as a project don't have enough capacity to > both maintain old Kibble and build new. The old code is hard to read, > not to mention to change. I tried it a few times. Good point. What I am trying to uncover here is about setting a common understanding and expectation about what we will and wont do. We need to send a consistent message to people that either uncover problems or bugs to Kibble-1 while the new Kibble is under construction. Do we allow people to report their issue and then use that list as a cross check to make sure that the issue is eitherno longer relevant or has been fixed in the new Kibble? > > > 2) What can we do we ensure that the Kibble dependency in > > reporter.apache.org tool continues to be supported? > > I think once we have a new working Kibble (together with tests > ensuring data quality) we should integrate it into reporter tool - how > I don't know yet. Until that time we reporter should be ok with old > Kibble. This sounds reasonable. The reporter tool is working well as it is and we are only getting comments around clarifications on how the metrics are calculated. > > > 3) How do we plan for the development of the new Kibble? > > Not sure but I would say we should just do it. Once we have some base > on which multiple people can work then we can split the work and so > on. There was no decision if we want to use REST or Graph QL so the > API part is still undefined (thuse I put it out after 1 milestone in > my proposal). But we seemed to be aligned to use Elasticsearch for > database, so the first step is definitely to define some basic classes > and interfaces around it. Just doing it has always been the way :-). I don't the big picture yet and was thinking of putting together something high level that is easy to follow so the information is open and clear for anyone who wants to contribute. Maybe it is as simple as creating the Github Kanban board and a Wiki page link... > > > 4) What can we do to grow Kibble usage and the community? > > Apart from active community we need well-structured project with > enough documentation and developer tools to make onboarding smooth and > easy. > Another good point and I'd be happy to spend some on the documentation. Thanks Sharan > Thanks, > Tomek > > On Mon, 10 May 2021 at 21:32, Sharan Foga wrote: > > > > Hi Tomek > > > > I think this is a great start. I was thinking about Kibble from a few other > > perspectives too especially around questions or problems that we might want > > to solve. > > > > 1) How far do we support Kibble-1 while the new Kibble is being developed? > > 2) What can we do we ensure that the Kibble dependency in > > reporter.apache.org tool continues to be supported? > > 3) How do we plan for the development of the new Kibble? (And I think you > > have some good responses for this in your message) > > 4) What can we do to grow Kibble usage and the community? (One of the > > things I'd like to include here is CHAOSS and getting Kibble setup as an > > implementation of some their metrics...) > > > > My understanding is that first three each have different repos so perhaps > > we could think about managing the roadmap at that level.. > > > > I will have a bit more time later this week - so let me see if I can refine > > or iterate on this a bit more. > > > > Thanks > > Sharan > > > > > > On 2021/05/02 16:33:04, Tomasz Urbaszek wrote: > > > +1 for using Github issues and projects for managing/tracking - it > > > works pretty well in Apache Airflow. > > > > > > Also I'm definitely in favour of defining a roadmap. I think top level > > > points on such a roadmap should correspond to some big milestones we > > > would like to achieve as a community but they also should be small > > > enough to make executing possible. If I may suggest some starting > > > point I would say: > > > bit more > > > 1. Apache Kibble foundations - we have scanners for few data sources > > > and some basic metrics. Data is accessible via underlying database + > > > Apache Superset as dashboard > > > 2. Crucial data sources - more scanners, covering all required data > > > sources (ASF crucial ones). At this point we should have quite good > > > documentation and understanding of the data we are collecting allowing > > > us for better decision making for next steps. > > > 3. Kibble API - we have an extensible API that allows users access the > > > data collected by our scanners. The API can be used for data sources > > > management (for example for configuring credentials). > > > 4. Kibble UI - there's an web app that can be used for accessing the data. > > > > > > What do you think? > > > > > > Cheers, > > > Tomek > > > > > > On Sat, 10 Apr 2021 at 18:48, Sharan Foga wrote: > > > >
Re: Apache Kibble 2.0
Hi Tomek
Thanks for working on this.
I like the idea. From the first time we had the demo running and showed the
list of possible sources - someone immediately asked about adding one that we
didn't have, so I think making adding them in a simple yet flexible way will be
a good thing!
Thanks
Sharan
On 2021/05/22 13:50:55, Tomasz Urbaszek wrote:
> Finally I started working on new Kibble:
> https://github.com/apache/kibble/pull/8
>
> I would like to propose introducing the following new concepts in new Apache
> Kibble:
>
> 1. DataSource - Github, Gitlab, Jira etc. - each data source would be a
> python package that would include logic for getting data from the external
> source as well as some metric and authentication mechanisms.
>
> 2. DataType - in case of Github: issues, PRs, commits etc - each DataSource
> can define its own data types. Single data type should define methods for
> getting relevant data and way to persist them in Kibble - this would replace
> the concept of scanners.
>
> Why? In this way Apache Kibble main aim and value would be building a
> framework for collecting projects information and serving them in useful,
> aggregated way. Having such a framework would make adding new data sources as
> simple as plug and play, thus allowing easy extension and possibility to
> build custom data source.
>
> Let me know what do you think. I will definitely try to play with this idea
> in my PR.
>
> Thanks,
> Tomek
>
>
>
> On 2021/03/22 19:49:28, Fourat ZOUARI wrote:
> > Hey Sharan,
> >
> > I've already commented on some parts of the doc and to date, it seems
> > fine/sufficient for kicking off the project, more reflections may come over
> > through the process.
> >
> > I suggest we have a kanban board on gh containing all pending tasks as a
> > projection for the project.
> >
> > Fourat
> >
> >
> > On Mon, Mar 22, 2021 at 7:10 PM Sharan Foga wrote:
> >
> > > Hi All
> > >
> > > Does anyone have any feedback on the work Tomek has done around this so
> > > far? Are people happy with the suggestions or do you have anything else in
> > > mind?
> > >
> > > Thanks
> > > Sharan
> > >
> > > On 2021/03/06 14:39:43, Tomasz Urbaszek wrote:
> > > > Hey all,
> > > >
> > > > I'm wondering if anyone had time to review those documents?
> > > >
> > > > Maybe we should start developing new Kibble and do the decision making
> > > > as we go to avoid overhead of excessive planning?
> > > >
> > > > Cheers,
> > > > Tomek
> > > >
> > > >
> > > > On Sat, 27 Feb 2021 at 18:43, Tomasz Urbaszek
> > > wrote:
> > > > >
> > > > > After thinking a bit I think we may consider using Graph QL instead of
> > > REST... I know that Daniel mentioned that we may consider using GQL and
> > > now
> > > (after doing some brainstorming) I think I start understand why. The only
> > > thing I'm afraid is that GQL has a steep learning curve and it may make it
> > > harder to grow the community and develop the app. But I'm not sure if this
> > > should block us from using GQL.
> > > > >
> > > > > T.
> > > > >
> > > > > On Sat, 27 Feb 2021 at 17:07, Tomasz Urbaszek
> > > wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> I prepared a draft of Open API specification that we may use to build
> > > new Kibble API:
> > > > >> https://app.swaggerhub.com/apis/turbaszek/kibble/1.0.0
> > > > >>
> > > > >> My idea was that inside single kibble deployment, there can be many
> > > organizations and each organization has its own users and configured
> > > scanners.
> > > > >>
> > > > >> So this proposal basically has 3 CRUD endpoints for organizations,
> > > users and scanners. And additionally there's the crucial
> > > /organization/{organization_id}/scanners/{scanner_id}/data endpoint which
> > > returns data collected by a given scanner for a given organization.
> > > > >>
> > > > >> The data returned by this endpoint is an object containing an array
> > > of data points (the real data) and additional information about the
> > > structure of a single data point. This makes the response little ambiguous
> > > but it gives us a big elasticity - each scanner can return data in
> > > different formats. This proposal does not have any filters or aggregation
> > > options which we definitely would like to add.
> > > > >>
> > > > >> Please let me know what you think! I'm really open to any suggestion
> > > ;)
> > > > >>
> > > > >> Cheers,
> > > > >> Tomek
> > > > >>
> > > > >> On Mon, 22 Feb 2021 at 21:10, Fourat ZOUARI wrote:
> > > > >>>
> > > > >>> Thx Sharan !
> > > > >>>
> > > > >>> I am excited to discover the great work done on kibble, the project
> > > > >>> addresses the need to visualize information around raw source code
> > > > >>> repositories and this is something i've been working on for some
> > > years now,
> > > > >>> i'm willing to contribute with my experience to this project, as a
> > > user and
> > > > >>> as a developer.
> > > > >>>
> > > > >>>
> > > > >>> On Sun, Feb 21, 2021 at 10:55 AM
