Re: Kibble Read the Docs account config ?

[Daniel Gruno]

On 3/31/24 10:26, sharanf wrote:

Hi All

I am finally getting some cycles to start doing some work :-). I am 
starting to take a look around and do some minor tidy up.  Took an 
initial look at our website and did some small changes - just to get 
back into the swing of things. I'd like to do a bit more on the 
documentation but there seems to be a bit of a mismatch with the Read 
the Docs github details.  When trying to edit our docs on Github it's 
coming up with a 404 page not found as the docs directory doesnt exist 
in the 'Kibble' repository anymore  and instead it is part of the 
Kibble-1 github repo which is correct (since it's that code that we are 
running in our demo).


So a quick question - does anyone know the Read the Docs account details 
or config so that we can change it to refer to Kibble-1 instead of 
Kibble and so that I can start getting that documentation updated. I 
know it's a little outdated but users are still using it to download and 
test Kibble so even though it's Kibble-1 - I think it makes sense to 
ensure it includes the right info so they can get up and running.


Thanks
Sharan


RTD had me as an admin for the kibble page. I can add you if you have an 
RTD account. In the meantime, I've changed the repo source to kibble-1.

Re: [DISCUSS] Clearing Down the Kibble Github Repo

[Daniel Gruno]

On 2023-07-10 22:02, sharanf wrote:

Hi All

When we separated out the repos into Kibble-1 and Kibble the plan was to 
start again from scratch. Some work was done in the new Kibble repo but 
that has been stalled for quite a while. I know Daniel has some ideas 
for either kickstarting some new work but that could mean clearing down 
the Kibble repo once again.


We don't have a working version on things in the Kibble repo as its 
Kibble-1 that is the working version. I've also seen that we are getting 
some PRs Inlcuding this one (https://github.com/apache/kibble/pull/17) 
coming through on the Kibble repo that are more documentation focussed 
changes.


So I'd like to discuss whether we continue to merge these PRs or decide 
to clear the Kibble repo down and begin again.


What do people think?


I think, if the aim is to sort of start afresh, perhaps the best 
compromise there is to move the current main branch to the side (rename 
it, keep its history) and make a new main branch from scratch.


And yeah, I do have some WIP stuff that grabs ideas and prototypes from 
both old and newer kibble code, I should polish it up and commit, when 
time allows :)




Thanks
Sharan

Re: Draft Kibble Board Report

[Daniel Gruno]

Looks good to me.

Well aware (painfully aware?) of the low activity, and I do have some 
prototypes almost ready to start pushing, along with some basic CI 
testing. I am currently trying to figure out the best approach here, but 
will follow up in a separate thread.



On 2023-06-11 21:11, sharanf wrote:

Hi All

See below - I've put together a draft board report for the quarter. 
Please take a look and let me know if you have any feedback, comments or 
updates.


Thanks
Sharan



## Description:
The mission of Apache Kibble is the creation and maintenance of software
related to an interactive project activity analyzer and aggregator

## Project Status:
Current project status: Ongoing, low activity.
Issues for the board: None.

## Membership Data:
Apache Kibble was founded 2017-10-17 (6 years ago)
There are currently 15 committers and 13 PMC members in this project.
The Committer-to-PMC ratio is roughly 8:7.

Community changes, past quarter:
- No new PMC members. Last addition was Michał Słowikowski on 2021-05-02.
- No new committers. Last addition was Kaxil Naik on 2021-02-21.

## Project Activity:
The project has not yet made a release.

Very little activity again during the quarter and still trying to find free
cycles to work on a release of previous Kibble version and a roadmap for 
what

the new Kibble could look like.

We've had a documentation contribution [1] from a new contributor and also
some interest from someone trying out an install [2].

## Community Health:
The community is small and we know that we need to start focussing on 
building

it up to be more sustainable but we first need to have work on defining the
project vision and roadmap of where we want the project to go. I still 
think

putting out a Kibble-1 release will help separate the previous version from
the new one that we'd like to build.

[1] https://github.com/apache/kibble/pull/16
[2] https://s.apache.org/zm5ej

Re: Help needed

[Daniel Gruno]

On 2023-05-08 08:09, Rothenbucher, Paul wrote:

Hello Developers,

I am currently searching for help. After cloning Kibble.git in 
/var/www/kibble I am not getting all the required folders.
As you can see in the screenshot. The Following commands from the 
documentation cant be done.



There are no /api/ and /ui/ folders which is stopping me from installing 
kibble any further. Is there a more detailed installation guide and what 
is the Error on why not everything is getting cloned.

We can set up a call if you want.

Thank you and regards.
Paul Rothenbucher


Hi, Paul,
You probably want to use https://github.com/apache/kibble-1 instead of 
the other kibble repository, as the current main kibble repository is 
being used to prototype a next generation platform.


For scanner clients, you'll want to use 
https://github.com/apache/kibble-scanners





*Paul Rothenbucher*

MES MOM & Automation New Associate

Accenture GmbH

Schweinfurter Strasse 11

97080, Würzburg

Tel: +49 931 460862311

Mobile: +49 151 12985539

*accenture.com* **






This message is for the designated recipient only and may contain 
privileged, proprietary, or otherwise confidential information. If you 
have received it in error, please notify the sender immediately and 
delete the original. Any other use of the e-mail by you is prohibited. 
Where allowed by local law, electronic communications with Accenture and 
its affiliates, including e-mail and instant messaging (including 
content), may be scanned by our systems for the purposes of information 
security and assessment of internal compliance with Accenture policy. 
Your privacy is important to us. Accenture uses your personal data only 
in compliance with data protection laws. For further information on how 
Accenture processes your personal data, please see our privacy statement 
at https://www.accenture.com/us-en/privacy-policy.

__

www.accenture.com

Re: Kibble Board Report Due

[Daniel Gruno]

On 1/29/23 15:05, sharanf wrote:

Hi All

Our board report is due soon so I will work on drafting something. I 
know it's been another quiet quarter for the project but we have had 
some significant holidays as part of it. As we move further in 2023 
let's see how we can start taking some small steps to generate some 
meaningful project activity.


Thanks
Sharan


Hi Sharan,

I am having some "unofficial" brainstorming chats with some other 
interested folks (some are on the kibble project, some are not (yet?)) 
about how to best turn the proposed roadmap into something more 
tangible, and it is my hope that it'll result in some basic scaffolding 
in the coming weeks.


I will note that these conversations are not for defining a new 
direction or such - that will always be brought to the mailing list in 
some form or the other - but rather just some informal chats about how 
one could syntactically approach some of the goals set out in the 
roadmap from November last year.


This will probably end up resulting in some prototype code that we can 
look at and discuss wherever applicable (mailing lists, issues/PRs etc).


There are also a few yet-to-be-answered questions about the repository 
layout for this, but I have not gotten that far yet :)


With regards,
Daniel.

Re: Rebooting the Kibble roadmap

[Daniel Gruno]

On 2022-09-15 13:56, sharanf wrote:

Hi Daniel

+1

I am extremely happy to see this thread about rebooting Kibble :-)

I think anything that makes it easier to contribute is a good thing. We 
have been talking about making the visualisations better (and more 
modern!) so adding the ability to improve it and make it more flexible 
is great, They say a picture says a 1000 words so while discussion is 
good - a perhaps an architecture diagram similar to what our friends at 
DevLake <https://devlake.apache.org/docs/Overview/Architecture>have done 
would help.


I am working on some mock-ups of the overall design idea, you can see it 
at https://imgur.com/a/ANUsie7




The data part did need some re-organisation and I remember the 
discussion we had previously <https://github.com/apache/kibble/pull/8> 
about trying to separate the source and the types so am happy to see 
this is still part of the roadmap to tidy up.


I  like the idea of not abandoning the mailing list, subversion and 
other data sources in favour of only Git and Github as there are many 
projects out there that still use them.


The mono repo idea is good - we already tried keeping two repos so let's 
see if one works better.


The only other thing I would bring up now is the possibility of making a 
release of Kibble-1 to give people coming along something to look at, 
work with and download. So will start a new thread for that. If 
necessary I'd be willing to have a go at working on organising that 
(with any other help I can get!) :-)


Thanks
Sharan


On 2022-09-11 18:36, Daniel Gruno wrote:

Hi folks,
a while back we attempted a complete redesign of the Kibble platform, 
which unfortunately fizzled out. I'd like to restart this process, but 
perhaps simplify and condense our goals a bit, so as to lower the bar 
for participation and implementation.


I'd like to propose we divide Kibble into three components:

- A management service that purely exists to manage sources, data 
access, and delegate jobs
- A scanner service that uses the sources/jobs from the above and 
gathers data points
- An optional visualization service that can latch onto the database 
and visualize the data gathered by the scanners. This would be 
optional in that the base server and scanners would work independently 
from this, and any other visualization platform (jupyter, devlake?? 
kibana??) could be utilized instead of the default option.


For the data part of it all, I'd also like to propose we do a split 
between source types and data types. That is to say, a source type 
could be a git repo, a github repo, a subversion repo, a pony mail 
list, a jira instance etc, whereas a data type could be a commit, an 
email, a ticket, etc. data types could have one or more associated 
sources, and these sources would themselves have individual ways of 
obtaining the required data. Thus, an issue scanner could essentially 
be source-type-agnostic, as the data type plugin itself would supply 
the API for grabbing a pre-determined base set of common data for that 
data type. As an example, a ticket scanner could work off both GitHub 
Issues, JIRA, and BugZilla. The scanner module would not need to know 
how to handle these individual calls, as the data type module would 
abstract that part and provide a standardized interface. This would 
also mean we could expand into subversion/mercurial/etc territory by 
abstracting the repository calls and providing a unified way of 
interacting with commits of any repository type.


Provided this is agreeable, I am willing to spend both time and 
resources on this (both that of my own, and that of my company).


WDYT?

With regards,
Daniel.

PS: I'd propose we start off with a mono-repo strategy, to ease the 
deployment and release workflow. If we later feel that a split into 
server/client is better, then we can do that.

Re: How to add daffodil-vscode repo so it is included in our board report stats?

[Daniel Gruno]

On 2022-09-06 10:29, Mike Beckerle wrote:

Can someone help me with this?

I *think* our board report wizard is including only daffodil and
daffodil-site repositories in the stats.

Is there an action to add the newer (this year) daffodil-vscode repository
so that the stats include it, or to check if this is already the case?


I have added daffodil-vscode to our scans, and it should already be 
showing up in your reports.




This repo has been very active of late, and I think the stats do not
include it.

Any help greatly appreciated.

Mike Beckerle
Apache Daffodil PMC | daffodil.apache.org
OGF DFDL Workgroup Co-Chair | www.ogf.org/ogf/doku.php/standards/dfdl/dfdl
Owl Cyber Defense | www.owlcyberdefense.com

Re: Have we ever made a Kibble release?

[Daniel Gruno]

On 23/05/2021 16.00, Sharan Foga wrote:

Hi All

Thanks to all who helped review our Board report. We got some feedback comment 
to include details of our last release in our next report. I've been searching 
through our mailing archives and can't find any details or Vote of the project 
ever doing a release.

If this is the case then we need to think about planning one as part of our 
roadmap.


We never hit a point where a release was feasible. So no, no releases 
ever, just like Whimsy and STeVe ;)




Thanks
Sharan

Re: Kibble Talk for ApacheCon?

[Daniel Gruno]

On 23/04/2021 17.37, Sharan Foga wrote:

Hi All

I think it could good to submit a Kibble talk for ApacheCon. It would be based 
on Kibble-1 as that is where we have all the Apache project data.

A lot of projects may not be aware of Kibble, what it show them and how they 
can use it to understand what is and has been happening in their community.

There could be 2 possible angles

1) Kibble 101 for beginners - go through the basics of and maybe open it up to 
the audience to look at their projects

2) How PMCs could use Kibble to help them understand how their community is 
made up, are they attracting new committers, are people staying, are they 
attracting new contributors and do they have non coding contributors whose 
contributions may not have been recognised etc


I think a "community health reporting 101" would be a welcome addition 
to the conference!




I know the CFP closing date is coming up pretty quickly but would be good to 
get some feedback. What do people think?

Thanks
Sharan

Re: Is Kibble being used in reporter.apache.org tool?

[Daniel Gruno]

On 07/03/2021 15.32, Sharan Foga wrote:

Hi All

Some of the stats that are being used in the reporter.apache.org tool look like 
or are very similar to the stats coming from Kibble. Does anyone know if Kibble 
is the main source?


Yes, reporter.a.o uses the Kibble demo at demo.kibble.a.o for its 
statistics :)




I was trying to find the reporter tool source code and found this in Github 
repo under Comdev

https://github.com/apache/comdev-reporter

but I also saw a note that Daniel had created or updated the tool with a new 
wizard https://s.apache.org/if8b6

Have I got the correct repo for the reporter tool source?


https://svn.apache.org/repos/asf/comdev/reporter.apache.org/trunk/



As we go through the Kibble rewrite we need to understand where we have 
anything dependent on our existing solution (Kibble-1) , so we can plan a 
strategy for updating it if necessary.


I think the Kibble 1 instance will be online for probably another year 
or so without any changes to it, as it is indeed being actively used.
Once we have a kibble 2, we can look at how old queries could be sent to 
the new version, and then switch over.




Thanks
Sharan

Re: Renaming Existing kibble Repo to kibble-old

[Daniel Gruno]

On 09/02/2021 21.07, Sharan Foga wrote:

Hi Tomek

I am separating this out as a new thread as I think it got lost in the previous 
discussion.  So has anything happened regarding this?

I took a quick look and I don't have access to the repo settings to do the 
rename so maybe it is something we need to ask infra to do as the apache 
organisation owner.


yeah, renames require a jira ticket, I believe. Feel free to file one :)



Thanks
Sharan


-


From: Tomasz Urbaszek 
Subject: Re: [Meeting Notes] Kibble Dev Call - 7 Jan 2021
Date: 2021/01/16 11:30:56
List: dev@kibble.apache.org

Hi folks,

I created a 2.0 branch in apache/kibble (just for sanity sake if we would
like to reuse some code) and I restored the main branch to some commit
before breaking the refactor.

Can someone of you rename apache/kibble -> apache/kibble-old and then
create apache/kibble repo (as discussed earlier)? I don't have the right to
rename repo.

Cheers,
Tomek

Re: Dev Call to discuss future Roadmap for Apache Kibble in 2021

[Daniel Gruno]

Good initiative :), I've put my availability in the doodle.

I'm leaning towards a complete refactoring/rewrite, a "kibble v2" where 
we use the general architecture/design of Kibble but write the python 
logic from scratch.


On 20/12/2020 22.55, Kaxil Naik wrote:

Hi folks,

I was talking to Daniel and Tomek on Slack that it would be awesome to have
a DEV call to discuss the future of Apache Kibble.

I am one of the committer, PMC Member and Release Manager of the Apache
Airflow project. I organized regular dev calls for Airflow until we
released Airflow 2.0 last week. And it really helped with aligning the
goals, scoping the items for the release etc.

I really think this would also be useful for Apache Kibble. I recently
started contributing to Kibble (commits
) and looks like it
needs more love and a bit of refactor. Whether we want to refactor existing
codebase or start fresh can be discussed on the call.

I propose let's start with one call and then we will see if a regular call
is needed or we can do async.

Following is the link of the doodle I created, please add a vote to what
day and time suits you best in the new year.

Doodle: https://doodle.com/poll/g39kwv6e87gcvfst

Let me know what you all think.

Regards,
Kaxil

Refreshing ASF demo today

[Daniel Gruno]

As subject says, I'm refreshing the repository list at the ASF demo, 
adding a few hundred new ones to the mix. This should temporarily mark 
all repositories as not having been scanned till the first scan runs, as 
I've changed some meta-data for all repos (using a token instead of a 
password for scanning issues).


I expect everything to be back to normal within a day or so.

With regards,
Daniel.

Re: [DISCUSS] Restructure the Apache Kibble project

[Daniel Gruno]

On 21/10/2020 10.42, Tomasz Urbaszek wrote:

Yesterday I took a deeper look at the api. And if I correctly
understand, currently we are generating open API spec from python
code.

If we would like to preserve it, I would like to suggest considering
the FastAPI framework https://fastapi.tiangolo.com . It has many
advantages including using native type hints from python 3 for data
validation as well as generating OpenAPI spec.


mhmm, I was even hoping to go as far as py3.9 for annotations to help 
"documenting as you code" :) I am very much for a type hint solution, 
especially as we could use mypy for tests then.




Tomek

On Mon, Oct 19, 2020 at 9:02 PM Tomasz Urbaszek  wrote:


Thanks for the replies!


I'm not too concerned about breaking API, we _could_ make the next one a 
version 2 with a version 2 API, a clean break from version 1


My main point in preserving the compatibiliy is to be sure that the ui
will work as expected. This will also limit the scope of the changes.


Whichever framework we pick, I would prefer if we could auto-generate the 
JSON/GraphQL API from the source.


+1 for automating as much as possible, probably I would lean to first
do the JSON as it is now and the consider adding GraphQL.

If there is no objections to merging scanners with server I will do
this as a first step. Here is a Github issue so we can manage all the
work:
https://github.com/apache/kibble/issues/64

As per the ideas - I created issues to keep it for future reference :)

Bests,
Tomek


On Mon, Oct 19, 2020 at 7:54 PM Sharan Foga  wrote:


Hi Tomek

Thanks for the initiative and ideas!

Anything that makes things simpler or easier is OK with me :-)  and 
consolidating things seems like a good idea. I am guessing that a lot of the 
changes would probably be behind the scenes rather than at the user end - right?

On the wishlist - I saw a talk by Myrle at ApacheCon@Home and she had done some 
initial work linking contributions to affiliation and I mentioned seeing if 
that could be something we could look at including into Kibble. I think we 
tried to do it with Meta Pony Factor but dont think it is complete. Anyway lots 
of ideas to keep us going!

Thanks
Sharan


On 2020/10/18 14:28:33, Tomasz Urbaszek  wrote:

Hello all,

I would like to propose a few things that I think are worth considering:

1. Merge all three repositories (kibble, kibble-scanners,
kibble-docker) into one. In this way I think we may simplify
development, dependency management, and distribution. Our project is
rather small and increasing places needing management probably won't
help.

2. Once we have the API server and scanner in one repo we can make
Kibble a python package. This will not only simplify installation and
setup but also may help us create a nice CLI to manage all components
(setup, server, scanners). By doing this users will be able to run
something like `kibble server` or `kibble scanners ponymail`.

3. Rewrite kibble API server to use a framework (for example FastAPI
which supports OpenAPI specs). This point is probably the hardest one
to implement as it will require adding tests to preserve compatibility
with kibble ui.

What do you think? Do you have anything that you would love to see in
Apache Kibble? :)

Cheers,
Tomek





--

Tomasz Urbaszek
Polidea | Software Engineer

M: +48 505 628 493
E: tomasz.urbas...@polidea.com

Unique Tech
Check out our projects!

Re: [DISCUSS] Restructure the Apache Kibble project

[Daniel Gruno]

On 18/10/2020 16.28, Tomasz Urbaszek wrote:

Hello all,

I would like to propose a few things that I think are worth considering:

1. Merge all three repositories (kibble, kibble-scanners,
kibble-docker) into one. In this way I think we may simplify
development, dependency management, and distribution. Our project is
rather small and increasing places needing management probably won't
help.

2. Once we have the API server and scanner in one repo we can make
Kibble a python package. This will not only simplify installation and
setup but also may help us create a nice CLI to manage all components
(setup, server, scanners). By doing this users will be able to run
something like `kibble server` or `kibble scanners ponymail`.

3. Rewrite kibble API server to use a framework (for example FastAPI
which supports OpenAPI specs). This point is probably the hardest one
to implement as it will require adding tests to preserve compatibility
with kibble ui.


I'm not too concerned about breaking API, we _could_ make the next one a 
version 2 with a version 2 API, a clean break from version 1. One thing 
I would perhaps love to see is if we could move to or add a GraphQL 
server component, if such would be possible.


Whichever framework we pick, I would prefer if we could auto-generate 
the JSON/GraphQL API from the source. Part of why I built the current 
"FauxpenAPI" server is to that it would do validation and autodocs. But 
that was nearly four years ago, technology changes :)




What do you think? Do you have anything that you would love to see in
Apache Kibble? :)


One thing we have been forgetting for a long time is embeds, akin to 
what Snoot.io uses on the projects.apache.org site. It needs to be 
possible to grab a chart from the standard UI, get an embed link, and 
publish it somewhere (with the appropriate caching in place for such a 
request)




Cheers,
Tomek

Re: New Contributor Pull Request to extend gitignore files

[Daniel Gruno]

On 05/10/2020 19.38, Sharan Foga wrote:

Hi All

We have a first time contributor who has submitted a change to extend the 
gitignore files.

https://github.com/apache/kibble/pull/33

It has no conflcts but I don't know what the consequences of the change so 
please can someone take a look and review it to see if it is OK to merge.


Looks fine to me.



Thanks
Sharan

Re: Github Issues not coming through in Kibble Issue Statistics

[Daniel Gruno]

On 01/09/2020 20.20, Sharan Foga wrote:

Hi

I may have found a potential bug. I have some Github sources that use and have 
github issues attached to them but nothing is coming through on the  Kibble 
issue statistics.

Thanks
Sharan



FWIW, I've discussed this and the previous email with Sharan off-list. 
It appears to be a documentation issue with github sources, and we'll 
work to get that fixed. In short, GitHub has deprecated certain 
authentication mechanisms, and so we need to tweak our requirements in 
the source setup page.

Re: Problem with Kibble Demo?

[Daniel Gruno]

On 18/05/2020 13.16, Sharan Foga wrote:

Hi

I tried logging into the Kibble demo with my own login and it's no longer 
working. The guest login isnt working either. Has anyone else noticed any 
problems with the demo?

Thanks
Sharan



The database had gone to a party without telling us.
I've restarted it and things should be back to normal.

Re: Kibble and Key Phrase Extraction

[Daniel Gruno]

On 05/01/2020 17.43, Sharan Foga wrote:

Hi All

In a previous Kibble version, although not on the menu, we did have a link for 
Key Phrase Extraction (KPE). Has that disappeared or is it still working?

I'm going to be doing a Kibble related talk at CHAOSSCon so wanted to re-run 
some of the KPE reports to use that I ran last year and the year before.

Thanks
Sharan



Hi Sharan,
I thnk we did a clean reinstall at some point back, and must have 
forgotten to sset up the PicoAPI key for the text analysis (we use 
PicoAPI for key phrases and sentiment analysis). I'll get that 
re-enabled. Unsure how long it'll take to scan stuff, could be weeks.


With regards,
Daniel.

Re: Mention Kibble in CHAOSS metrics?

[Daniel Gruno]

On 27/11/2019 21.58, Georg Link wrote:

Hi Kibble folks,

In the CHAOSS metrics, we list software that implement a specific metric.
I am sure that Kibble provides many of the metrics we are describing in CHAOSS.
I would love to recognize Kibble accordingly within each metric.

Is this something you’d be interested in?


Naturally :)
Sharan and I did a mapping of CHAOSS <-> Kibble for FOSDEM 2019, with a 
bunch of the metrics and how they are basically the same. I'll go look 
for that slideshow.




Best,
Georg


--
Georg Link
(he/him)

Re: What does the project use for issue management?

[Daniel Gruno]

On 17/09/2019 23.51, sebb wrote:

I've scanned the website, but cannot find any reference to the issue
management system.

In particular, where should website bugs be reported?



Sorry for the late post,
github issues would be the preferred place for this.

Kibble powering the new ASF Board Report Wizard

[Daniel Gruno]

As subject says, Kibble is now used to pull in metrics for the Community 
Health section of the new report wizard (see dev@community for details).


Hopefully, this will increase the interest in our project and provide 
some feedback :) Still working on what exactly to pull from Kibble, but 
it proved easier than I thought, by using simple API queries using the 
Kibble-Token header.


For instance, grabbing all PRs/issues from GitHub for a specific project 
can be done as simple as:


issues = requests.post('https://demo.kibble.apache.org/api/issue/issues',
  headers = {
'Content-Type': 'application/json',
'Kibble-Token': TOKEN,
  },
  json = {
"page":"issues",
"quick":True,
"interval": "week",
"subfilter":"/" + project,
"distinguish":True
}
 ).json();

If anyone has good ideas on which metrics we could use (or improve upon) 
for assessing a project's overall health and activity, I'd appreciate 
feedback :)


With regards,
Daniel.

Re: Projects refresh on kibble.dev?

[Daniel Gruno]

Update:

- 24 new repos were added today
- Fixes have been put in place to work around GitHub's rate limits by 
checking said limits and waiting for them to reset, then continuing.
- Additional workarounds for abiding by abuse detection rules have been 
made.


I am happy to report that as of today, all GitHub Issues/PRs are 
properly scanned on a daily basis :)



On 10/07/2019 16.25, Robert Munteanu wrote:

On Wed, 2019-07-10 at 15:48 +0200, Daniel Gruno wrote:

On 7/10/19 3:44 PM, Robert Munteanu wrote:

Hi,

Can anyone please run an import of the new GitHub projects on
https://kibble.dev ? Currently some repositories are missing.


I can get around to that this week, sure thing!


Thanks!




Also, I see that on [1] some sources have not been synced due to
"API
rate limit exceeded for user ID 51666742". Does this impact the
analysis of projects on the long run or does it only mean that it
will
be delayed by some days?


That depends...we have nearly 2000 repos on github now, and scanning
them all for changes is one of the bottlenecks we're working on (keep
in
mind, github allows for 5,000 API calls per hour only). It _should_
only
mean that some (randomly ordered) github issue/pr scans are delayed
for
a day or two, but I'll take a look at whether we can somehow avoid
this
more easily, when time permits.


Thanks for clarifying. For me personally it is perfectly fine to get
data with some delay (1-2-5 days), as I'm not looking at it in real
time, maybe once every month.

Thanks!

Robert

Re: Projects refresh on kibble.dev?

[Daniel Gruno]

On 7/10/19 3:44 PM, Robert Munteanu wrote:

Hi,

Can anyone please run an import of the new GitHub projects on
https://kibble.dev ? Currently some repositories are missing.


I can get around to that this week, sure thing!



Also, I see that on [1] some sources have not been synced due to "API
rate limit exceeded for user ID 51666742". Does this impact the
analysis of projects on the long run or does it only mean that it will
be delayed by some days?


That depends...we have nearly 2000 repos on github now, and scanning 
them all for changes is one of the bottlenecks we're working on (keep in 
mind, github allows for 5,000 API calls per hour only). It _should_ only 
mean that some (randomly ordered) github issue/pr scans are delayed for 
a day or two, but I'll take a look at whether we can somehow avoid this 
more easily, when time permits.




Thanks,

Robert

[1]: https://kibble.dev/organisations.html?page=sources

Re: New demo server, ALL THE PROJECTS!

[Daniel Gruno]

On 6/12/19 11:39 AM, Robert Munteanu wrote:

Hi,

On Mon, 2019-06-10 at 17:52 +0200, Daniel Gruno wrote:

the dogs and ponies at Quenda have donated a much beefier machine
for
our tests, with more than a terabyte of super fast NVMe storage and
twice the compute capacity, meaning we're going to aim at having the
ENTIRE ASF in our demo! This might not work very well at all, but
we're
going to try!


That is great news :-) . Do you also plan to keep this instance
periodically updates with new repositories that are created in the
apache github org?


The hopeful plan is to keep it up to date with everything we have at 
ASF. I will, however, stress that it's a demo, and not a production 
system, so there will be no guarantees that it stays up or that the data 
isn't wiped now and then, for science reasons. First and foremost, it's 
meant as a test for developing the software :)




Thanks,

Robert

Re: New demo server, ALL THE PROJECTS!

[Daniel Gruno]

On 6/12/19 12:14 AM, Sharan Foga wrote:

HI Daniel

I have re-registered for the demo and need to be added to an organisation, so 
please can you do that?

Do we have some docs on this process and probably need others to be able to do 
it - so am happy to volunteer. I am guessing once we have the whole ASF repos 
there - our user base could significantly increase.


I followed 
https://apache-kibble.readthedocs.io/en/latest/setup.html#installing-the-server 
for setting up the new instance, fixing small bugs as I went along. As 
for adding users, you go to organization -> users and add people there. 
The user/org management tools are in need of rework in general, with a 
few features still missing. Hopefully I'll have time over the summer to 
address that :)




Thanks
Sharan

On 2019/06/10 15:52:05, Daniel Gruno  wrote:

Hi folks,
the dogs and ponies at Quenda have donated a much beefier machine for
our tests, with more than a terabyte of super fast NVMe storage and
twice the compute capacity, meaning we're going to aim at having the
ENTIRE ASF in our demo! This might not work very well at all, but we're
going to try!

1748 repositories (1528 git, 220 svn) have been input in the machine,
and it's not chewing through a million lines of code per minute.

Later on I'll get mail, jira etc added...

Anyway, the new box is live at https://kibble.dev/ and you should sign
up for a new account, as this is a clean database (trying elasticsearch
7.1 for improvements). Ping me and I'll get you added to the Apache
organisation there.

If people need access to the box, let me know.
I'll also file a JIRA to have the demo.kibble.apache.org DNS record
changed later on, when things are working.


With regards,
Daniel.

New demo server, ALL THE PROJECTS!

[Daniel Gruno]

Hi folks,
the dogs and ponies at Quenda have donated a much beefier machine for 
our tests, with more than a terabyte of super fast NVMe storage and 
twice the compute capacity, meaning we're going to aim at having the 
ENTIRE ASF in our demo! This might not work very well at all, but we're 
going to try!


1748 repositories (1528 git, 220 svn) have been input in the machine, 
and it's not chewing through a million lines of code per minute.


Later on I'll get mail, jira etc added...

Anyway, the new box is live at https://kibble.dev/ and you should sign 
up for a new account, as this is a clean database (trying elasticsearch 
7.1 for improvements). Ping me and I'll get you added to the Apache 
organisation there.


If people need access to the box, let me know.
I'll also file a JIRA to have the demo.kibble.apache.org DNS record 
changed later on, when things are working.



With regards,
Daniel.

Idea: Contribution categories in Kibble

[Daniel Gruno]

Hi folks,
I had a late night idea (based on a tweet from Shane) that I think will 
be great, and so I've started working on it a bit: (code) Contributions 
by contribution type. The basic idea would be to start by looking for 
commits based on the type of files changed, and sort it into categories 
such as code development, documentation, graphic design, road-mapping 
(if I can figure that one out :p), and then visualize how contributions 
have been made based on which category they fall into. This could later 
be coupled with tags on issues/prs and so forth, to create some 
visibility into the various ways of contributing to FLOSS.


WDYT?

With regards,
Daniel.

Re: Password reset not working?

[Daniel Gruno]

On 01/05/2019 09.05, Sharan Foga wrote:

Hi Robert

Taking a look at the password reset code, there is nothing behind it except a 
re-display of the index page. I'm setup as Admin but cant see anywhere to 
manage or reset paswords for existing user profiles.

Daniel do you have any tips how we can get Robert back up and running?


I don't think the password reset works at all right now :\.
I might have time to make it do stuff within a few days, but for now, 
it'd be best to just use the guest account if needed.




Thanks
Sharan

On 2019/04/30 11:15:44, Robert Munteanu  wrote:

Hi Sharan,

On Sun, 2019-04-28 at 10:45 +, Sharan Foga wrote:

Hi Robert

Did anyone fix this for you yet?



Nope, not yet :-)

Robert


Thanks
Sharan

On 2019/04/16 13:13:31, Robert Munteanu  wrote:

Hi,

I tried to reset my password on http://demo.kibble.apache.org/ . I
enter my email (romb...@apache.org) and I just get redirected to
http://demo.kibble.apache.org/index.html . Nothing reaches my
inbox.

Can anyone please advise how to solve this?

Thanks,

Robert

Re: Patch method in the sources.py file.

[Daniel Gruno]

On 30/04/2019 08.20, Esra Karakaş wrote:

Hi all,

A while ago I sent a pull request for editing the names of the
organization. Now I want to create patch method about editing sources
in the source.py file.
I would like to know what you want this method to do exactly.

Thanks!
Esra



https://demo.kibble.apache.org/apidoc.html#patch-api-sources is what we 
would expect the request to accept.


It should work much like the PUT method, except:
 - it should only accept one source definition, not an array of them
 - it should check that an object with the source ID already exists
 - it should do an update rather than an override of the object.
 - it should *not* allow you to change the source type.

Once the API handler is done, we would then look into how we're going to 
handle this in the UI.

Re: New Sources Added to Kibble Demo

[Daniel Gruno]

On 28/04/2019 05.50, Sharan Foga wrote:

Hi All

I've added a few new sources to the Kibble demo as part of my research (Kylin, 
Weex, Skywalking and Dubbo). These are podlings and projects that have been 
initiated by Chinese contributors.

I have also added Trafodion at the request of Pierre Smits. BTW I've seen 
something recently about the size of the trafodion repo -- will that be an 
issue for Kibble yet? (as I know Daniel mentioned that we have fixed space 
available for the demo)

Thanks
Sharan



We should have plenty of space in the scratch directory for many more 
projects (we're using 20GB out of 100). We're not (yet) limited by disk 
space, I suspect concurrency is going to be an issue before disk space is.

Re: Kibble Demo – Refreshing the Source Data

[Daniel Gruno]

On 13/04/2019 09.40, Sharan Foga wrote:

Hi All

I’ve been taking a look around the Kibble demo as I’d like to add some new 
sources to help with my research paper. I’ve just added Apache Kylin (repo, 
mailing lists and issue tracker) and might add a few more from Incubator as I'd 
like to take a look at the evolution over time of project metrics and behaviour 
as they progress from incoming podling to Top Level Project.


As an aside, if you add GitHub as a source, you will need to add a 
username/password, or it can't fetch most of the data from there. I can 
share some bot credentials with you if need be.




Is there a way to get the data refreshed on the fly or do I need to wait for 
the regular job to run?

Thanks
Sharan

Re: Kibble Demo – Refreshing the Source Data

[Daniel Gruno]

On 13/04/2019 09.40, Sharan Foga wrote:

Hi All

I’ve been taking a look around the Kibble demo as I’d like to add some new 
sources to help with my research paper. I’ve just added Apache Kylin (repo, 
mailing lists and issue tracker) and might add a few more from Incubator as I'd 
like to take a look at the evolution over time of project metrics and behaviour 
as they progress from incoming podling to Top Level Project.

Is there a way to get the data refreshed on the fly or do I need to wait for 
the regular job to run?


the cron job runs around midnight every day. You can get access to the 
box and manually run the scans. If you'd like that, I can set you up 
with access to the demo box.




Thanks
Sharan

Re: Adding more projects to Kibble Demo

[Daniel Gruno]

On 25/02/2019 22.57, Sharan Foga wrote:

Hi All

I’m planning to use Kibble again for my MBA Thesis – this time I’m planning to 
use it to look at incubating projects and ones that have graduated, so would it 
be OK if I add some more sources to the demo instance?


Sure, that's what it's there for!



I remember when we were setting it up – there was a limit to how many projects 
we could accept – is that still the case?


There is a technical limit, yeah, defined by the hardware. We're not 
quite there yet (I'd think we could double what we have now), you can 
probably add quite a few repos, especially if they're not 20 year old 
ones with 50k commits each :)




Thanks
Sharan

Re: Contributing Apache Kibble

[Daniel Gruno]

On 1/10/19 12:01 AM, Esra Karakaş wrote:

In fact, I used Kibble and I understood how it works. As I said
before, I am familiar with the python but I couldn't find a feature to
add. If there is anything you can suggest, I would like to start from
that point.


A good starting point would be the open issues and pull requests on 
GitHub. Some of them are just missing simple code tweaks before they can 
be merged in.




Daniel Gruno , 8 Oca 2019 Sal, 21:42 tarihinde şunu yazdı:


On 1/8/19 3:17 PM, Esra Karakaş wrote:

Dear all,

I am the fourth grade student at university. As a thesis, I would to
contribute to Kibble. I can allow time at least 10 hours a week for
that. I examined Kibble. I am familiar with Python programming
language. I would like to work on it with you if there is anything
that I can add something or correct any issues.
I am waiting for your
respond to start working on it.

Thanks!




Hi Esra,
thanks for volunteering to work with us on improving Kibble!
If you are familiar with python, and you have a grasp of how Kibble
works, you can basically suggest any improvement or new feature you
like. And even if you don't fully understand how the bits and pieces fit
together yet, you can help improve by looking at the feature sets we
have, and maybe think about what we could add or improve. Kibble is very
modular and malleable, so you can quite quickly write a scanner or a new
visualization, and have it all line up and just work.

So, the short of the long; figure out the essentials of how Kibble
works, think of new ideas to add or ways to improve what we have - you
don't have to program it all by yourself, even writing down some
thought-out suggestions will help tremendously.

With regards,
Daniel.

Re: Kibble Lightning Talk at CHAOSSCon ?

[Daniel Gruno]

On 1/13/19 5:28 PM, Georg Link wrote:

+1

I am happy to hear that you will be sharing your Kibble work at CHAOSScon.
One aspect that I would find interesting is whether and how Kibble is
benefiting from the CHAOSS work and/or contributing to it. It might be
powerful if you could include 30 seconds to touch on that.


Definitely. I would probably admit that the coupling is very loose at 
the moment, but it is there, and perhaps CHAOSSCon can help bring the 
project closer to the CHAOSS community!


I could see us touch on:

- Areas where Kibble follows/uses CHAOSS standards
- Areas where we diverge
- Call to action to help bring us closer together and leverage the 
strengths in both places.


Kibble is stable and works, but it's still largely a work/thought in 
progress, and if we can align some of our ideas with established 
standards, that would be awesome.


Circling back to the lightning talk; what would we have to do to get a 
slot there?




Georg

On Sun, Jan 13, 2019 at 9:41 AM Daniel Gruno  wrote:


On 1/13/19 4:29 PM, Sharan Foga wrote:

Hi All

We missed the CHAOSSCon CFP deadline but I see from their published

agenda there might be a slot available in the lightning talks sections for
us to do something on Kibble.




https://urldefense.proofpoint.com/v2/url?u=https-3A__chaoss.community_chaosscon-2D2019-2Deu_=DwIDaQ=Cu5g146wZdoqVuKpTNsYHeFX_rg6kWhlkLF8Eft-wwo=P2SSJg6nDRqSw6MK6ZVW76NaDtv69G3wvyOnOxwzqv0=4Fv7NFgoMzVH3WIMhZrwAUK4uMxZ0O5QXMJfWgEFKSU=iDZ9SmP7zma4QoyZ5UlGz17F8d6PvC5fTGlnn-wVYYY=


Myself and Daniel are going to be at CHAOSSCon  so if there is space in

that section of the schedule, I’m sure we could easily fill 5 minutes and
probably more!. Daniel – would you be keen to do something if we could get
a slot? (I’d help too obviously :-)

Yeah, I'm sure we could do a Kibble Flash Crash Course!
I'd be happy to work on that.



Thanks
Sharan

Re: Kibble Lightning Talk at CHAOSSCon ?

[Daniel Gruno]

On 1/13/19 4:29 PM, Sharan Foga wrote:

Hi All

We missed the CHAOSSCon CFP deadline but I see from their published agenda 
there might be a slot available in the lightning talks sections for us to do 
something on Kibble.

https://chaoss.community/chaosscon-2019-eu/

Myself and Daniel are going to be at CHAOSSCon  so if there is space in that 
section of the schedule, I’m sure we could easily fill 5 minutes and probably 
more!. Daniel – would you be keen to do something if we could get a slot? (I’d 
help too obviously :-)


Yeah, I'm sure we could do a Kibble Flash Crash Course!
I'd be happy to work on that.



Thanks
Sharan

Re: Contributing Apache Kibble

[Daniel Gruno]

On 1/8/19 3:17 PM, Esra Karakaş wrote:

Dear all,

I am the fourth grade student at university. As a thesis, I would to
contribute to Kibble. I can allow time at least 10 hours a week for
that. I examined Kibble. I am familiar with Python programming
language. I would like to work on it with you if there is anything
that I can add something or correct any issues.
I am waiting for your
respond to start working on it.

Thanks!




Hi Esra,
thanks for volunteering to work with us on improving Kibble!
If you are familiar with python, and you have a grasp of how Kibble 
works, you can basically suggest any improvement or new feature you 
like. And even if you don't fully understand how the bits and pieces fit 
together yet, you can help improve by looking at the feature sets we 
have, and maybe think about what we could add or improve. Kibble is very 
modular and malleable, so you can quite quickly write a scanner or a new 
visualization, and have it all line up and just work.


So, the short of the long; figure out the essentials of how Kibble 
works, think of new ideas to add or ways to improve what we have - you 
don't have to program it all by yourself, even writing down some 
thought-out suggestions will help tremendously.


With regards,
Daniel.

Re: Anyone going to CHAOSSCon?

[Daniel Gruno]

On 12/22/18 11:23 AM, Sharan Foga wrote:

Hi All

I’ve decided that I’m going to go to CHAOSSCon to meet people from the CHAOSS 
workgroups and see how we can get more involved with what is happening. I think 
there are important topics for Kibble as well as Community Development in 
general so will post on ComDev too.

  It’s being held on the 1st February 2019 in Brussels, the day before FOSDEM – 
so if you are going to FOSDEM then please consider coming in a day earlier to 
attend CHAOSSCon. See the link below for more details

https://chaoss.community/chaosscon-2019-eu/

If I’d known I was definitely going then I might have submitted something, 
perhaps a Kibble overview – although I think their CFP is now closed.


I'm planning to attend as well. Unfortunately, like you, I had intended 
to submit a paper, but was traveling abroad during the CFP window, and 
missed it by a day due to health related issues :\. I still plan on 
attending, and it'll be great for not only catching up with users of 
kibble and other like-minded pieces of software, but also to figure out 
what the consumers of this sort of software is looking for; what's 
working, what's not, and what can we do more of.




Anyway – if anyone else is planning to go then please let me know!

Thanks
Sharan

Re: Index mailing list subscriptions and Twitter activity?

[Daniel Gruno]

On 10/04/2018 09:24 AM, Christofer Dutz wrote:

Hi all,

it would be great if we could also track the number of subscriptions to each 
mailinglist … would that be possible?


There is generally speaking no way to fetch this in a standardized way, 
and pretty much no one has those numbers publicly available.




Also as Twitter is becoming quite an interesting communication channel for 
projects … how about also tracking the number of tweets, retweets, likes, 
followers for a Projects twitter account?


I am (albeit slowly!) working on a twitter scanner for Kibble. See 
https://github.com/apache/kibble-scanners/blob/master/src/plugins/scanners/twitter.py 
- I think I have some changes pending that I need to commit, and then 
test it.




Just thinking ;-)


Chris

PS: Now jira and commits are correctly monitored for PLC4X … what’s “FORUMS” … 
do we have Apache Forums?



Forums are stuff like discourse and so on. We rarely use that at the ASF.

Re: API access available?

[Daniel Gruno]

On 09/18/2018 05:52 PM, Robert Munteanu wrote:

One things which threw me off a bit is that even existing repositories
now have a status of "Source hasn't been processed yet..." . That looks
mostly cosmetic though since the old data is still there.


Aaah, yeah, I think the logic here is "this source object was changed, 
so we haven't technically scanned it _as is_ yet". Just caught that 
myself :)


With regards,
Daniel.



Thanks,

Robert

Re: API access available?

[Daniel Gruno]

On 09/18/2018 05:58 PM, Robert Munteanu wrote:

On Tue, 2018-09-18 at 17:55 +0200, Daniel Gruno wrote:

One thing to note; For GitHub repos, you will need to specify a
user/pass if you want github issues/PRs to be looked at - this can be
an
anonymous account if you like. I'll see if I can't work our regular
kibble user into the repos you added, assuming you did not specify
any user.


Interesting, I did not know that (maybe add it as a hint on the page?).
I did not specify any user as I saw no need :-) - GitHub repos are
public but it would be interesting to see PR information as well.

Thanks,

Robert




It should say that the user/pass is required, but perhaps we need to 
tweak the wording there.


WTR API access, there's now a WIP token system in place.
Go to https://demo.kibble.apache.org/api/session in a browser, and 
you'll see a 'token' element if you're logged in. You can pass that to 
curl as a 'Kibble-Token' header to gain access to things.


curl -H 'Kibble-Token: abcdef1234' https://

With regards,
Daniel.

Re: API access available?

[Daniel Gruno]

On 09/18/2018 05:52 PM, Robert Munteanu wrote:

On Thu, 2018-09-13 at 16:45 +0200, Daniel Gruno wrote:

On 09/13/2018 04:42 PM, Daniel Gruno wrote:

On 09/13/2018 04:38 PM, Robert Munteanu wrote:

Hi,

I am trying to simplify my job of keeping the list of sources up-
to-
date for the Kibble demo instance. Basically we add git
repositories
periodically and I want to add them to Kibble as well.

I was thinking of using the Kibble API to retrieve the list of
sources
and checking which of our repositories are not there, so I can
easily
paste them in Kibble.


It's a bit convoluted at the moment, apologies.
First, you have to obtain a session cookie by logging in;
https://demo.kibble.apache.org/apidoc.html#put-api-session
when you have that cookie, pass it to the API end point you wish to
use,
and it should work just fine.

I'll be working later next week on API tokens for use here, it's
one of
the things that are still missing.

With regards,
Daniel.



I should add one important/easy thing here:

If you want to keep the list of sources up to date with e.g. github,
you
can just PUT a request with all the sources you have, and it will
both
update existing ones and add the ones it doesn't have, there are
checks
against adding duplicates, so only sources that aren't in the DB
would
get added even if you post old sources to the API.



That is nice! Since I'm having some issues with the API (see earlier
emails) I used the web form to paste in all Sling repos again. It seems
to have worked - the repos are added (though I see not yet processed )
and the old date is still there.


I'll look into the API issue - I think it might be an issue with the 
session maker, which MAY need a cookie before you PUT to session...which 
obviously shouldn't be a thing :D maybe I should just look into proper 
API tokens instead.




One things which threw me off a bit is that even existing repositories
now have a status of "Source hasn't been processed yet..." . That looks
mostly cosmetic though since the old data is still there.


I think some of it is a remnant of me copying over a bunch of it 
directly from the old database, not through the API.


One thing to note; For GitHub repos, you will need to specify a 
user/pass if you want github issues/PRs to be looked at - this can be an 
anonymous account if you like. I'll see if I can't work our regular 
kibble user into the repos you added, assuming you did not specify any user.




Thanks,

Robert

Re: Contributors page tests

[Daniel Gruno]

On 09/17/2018 10:16 AM, Robert Munteanu wrote:

On Mon, 2018-09-17 at 10:04 +0200, Daniel Gruno wrote:

I also see that the contributor is linked to an email address so we
will
probably will have people that are contributing under different
email
addresses, though not sure if that is an issue at this stage.


There are, and it's not an easy task to solve. email addresses are
the
best we can reliably guess is a single person. There could be two
Sharans or Daniels, but they would have different email addresses,
so
that's our best guess for what constitutes a person. We should make
some
way of merging people.


One suggestions that was floated around when presenting Kibble was the
GitHub API - since Kibble can pull data from GitHub it can potentially
use that API _if_ a someone defines multiple email aliases for the same
account.


Yeah, provided they make it public - sadly, most people don't make all 
their emails public, and GitHub does a good job of hiding it :\




Thanks,

Robert

Re: Contributors page tests

[Daniel Gruno]

On 09/17/2018 09:58 AM, sha...@apache.org wrote:

Hi Daniel

Great work - it looks very nice!

I know it is a draft so may not be all there yet so how are you counting 
contributions? For example for Kibble - it's showing you have 74 
contributions and when I click to look at the detail, I see the number 
of emails and code commits but it doesn't add up to 74.


Buggo! I'll get that fixed right away :)



I also see that the contributor is linked to an email address so we will 
probably will have people that are contributing under different email 
addresses, though not sure if that is an issue at this stage.


There are, and it's not an easy task to solve. email addresses are the 
best we can reliably guess is a single person. There could be two 
Sharans or Daniels, but they would have different email addresses, so 
that's our best guess for what constitutes a person. We should make some 
way of merging people.




Thanks
Sharan

On 17.9.2018 09:38, Daniel Gruno wrote:

Hi folks,
I've started building the contributors page that was missing in kibble.
It's a very early draft, but for instance, you can see all kibble 
contributors for the past year at:
https://demo.kibble.apache.org/contributors.html?page=people=kibble=1505599200=1537221599 



You can filter by date and sources (including quick filter), so you 
can, for instance, see everyone who's interacted with a project, or 
just committers, just email authors and so on.


I'll get it more interactive later on.

With regards,
Daniel.

Contributors page tests

[Daniel Gruno]

Hi folks,
I've started building the contributors page that was missing in kibble.
It's a very early draft, but for instance, you can see all kibble 
contributors for the past year at:

https://demo.kibble.apache.org/contributors.html?page=people=kibble=1505599200=1537221599

You can filter by date and sources (including quick filter), so you can, 
for instance, see everyone who's interacted with a project, or just 
committers, just email authors and so on.


I'll get it more interactive later on.

With regards,
Daniel.

Re: Update from CHAOSS Meetings 11-12 September 2018

[Daniel Gruno]

On 09/14/2018 12:37 PM, Sharan Foga wrote:

- Sample implementation of metrics: They are using Jupyter notebooks as a way 
to show examples of the GMD metrics have been implemented. (Can we setup a 
notebook for some Kibble data and logic?)



I don't know that we *need* this part. Jupyter, as I understand it, is 
about aggregating, formulating and graphing data-sets - kibble already 
does this in the UI via python3 and C3/D3 (P3C3D3!), so I don't know 
what we would need another visualization method for right now...


We do exports as data tables, PNG images and SVG files out of the box.
What would Jupyter add to this?

With regards,
Daniel.

Re: Update from CHAOSS Meetings 11-12 September 2018

[Daniel Gruno]

On 09/14/2018 12:37 PM, Sharan Foga wrote:

Hi All

I attended a couple of CHAOSS meetings this week and mentioned that I would 
send a brief update to this list. The main ones I picked up were:

- CHAOSSCon Europe is happening on Friday 1st February 2019 in Brussels (Note 
that it is the day before FOSDEM!). They are currently looking for a location.

- Metrics in Practice: They are looking for some more examples of how community 
metrics are used in practice (I think we will be able to provide some feedback 
here especially with the feedback we have coming in from the pilot projects)

- Re-organisation and re-naming of of the Growth, Maturity and Decline (GMD) 
Github repo: https://github.com/chaoss/wg-gmd (Changes have been suggested and 
this is currently out for feedback before updating)

- The GMD workgroup would like to focus on Use Cases related to the current 
metrics that have been defined (I think we will be able to provide some 
feedback here based on what has been implemented in Kibble as well as 
information from the pilot projects)

- Sample implementation of metrics: They are using Jupyter notebooks as a way 
to show examples of the GMD metrics have been implemented. (Can we setup a 
notebook for some Kibble data and logic?)

Anyone is welcome to join the CHAOSS meetings and the next meetings are as 
follows:
- The next weekly meeting is Tuesday 18th September 2018
- The next GMD Workgroup meeting is 26th September 2018

Thanks
Sharan



The GMD stuff makes me think of our community retention charts, like 
this one; 
https://demo.kibble.apache.org/engagement.html?page=retention=httpd 
where you can clearly see growth, maturity and eventual (slow, on-going) 
decline of httpd as other web servers become popular. This isn't to say 
httpd is dying, but that other projects are now having a bite of the web 
server cake as well. We _could_ grab some attic'ed projects and look at 
the retention charts for them, it is usually very visible there when a 
project has peaked.


With regards,
Daniel.

Re: API access available?

[Daniel Gruno]

On 09/13/2018 04:38 PM, Robert Munteanu wrote:

Hi,

I am trying to simplify my job of keeping the list of sources up-to-
date for the Kibble demo instance. Basically we add git repositories
periodically and I want to add them to Kibble as well.

I was thinking of using the Kibble API to retrieve the list of sources
and checking which of our repositories are not there, so I can easily
paste them in Kibble.


It's a bit convoluted at the moment, apologies.
First, you have to obtain a session cookie by logging in;
https://demo.kibble.apache.org/apidoc.html#put-api-session
when you have that cookie, pass it to the API end point you wish to use, 
and it should work just fine.


I'll be working later next week on API tokens for use here, it's one of 
the things that are still missing.


With regards,
Daniel.

Re: Possible to exclude directories from analysis?

[Daniel Gruno]

top posting, yaay!
I have a new server sort of set up now. I'll have infra redirect to that 
one instead, and it'll rebuild most of the database during the night 
(and the next night, and...).


there's a new option, a path filter in the repos tab, which filters 
commits, line changes, trends, top contributors etc by paths affected, 
so you can enter either 'jbake' to get everything touching jbake, or 
'!jbake' to get everything that doesn't touch those files.


The CNAME should switch over some time today :)

With regards,
Daniel.

PS: We're also switching to elasticsearch 6 with this move, which is 
going to be great, as that allows us to test on a modern ES, instead of 
the old 5.x installation we're currently running on.


On 09/12/2018 12:24 PM, Daniel Gruno wrote:

On 09/12/2018 12:22 PM, Robert Munteanu wrote:


If you look at the sling-site repository at [1] we have the actual
documentation under src/main/jbake, with
- content being markdown files
- templates being ... well ... templates
- and assets being static files

Some of those static files are generated (javadoc, Maven plugin sites)
and should not be recorded by Kiddle. Those are the ones that are
problematic, especially since we have a large number of javadocs
committed.

$ find src/main/jbake/assets/apidocs -type f | wc -l
7127

Those are the ones we'd like excluded, if at all possible.

Thanks,

Robert

[1]: https://github.com/apache/sling-site/tree/master/src/main/jbake




I've added a change to the scanners, so they will put a list of files 
changes into each commit object we record. This is likely going to 
require a complete re-scan of all things sling...which in theory is 
fine, as I _was_ planning on moving the demo server to a new box anyway. 
I'll let y'all know more when I have that worked out in my mind :)

After the move and re-scan, it should be possible to exclude by file paths.

Re: Possible to exclude directories from analysis?

[Daniel Gruno]

On 09/12/2018 12:22 PM, Robert Munteanu wrote:


If you look at the sling-site repository at [1] we have the actual
documentation under src/main/jbake, with
- content being markdown files
- templates being ... well ... templates
- and assets being static files

Some of those static files are generated (javadoc, Maven plugin sites)
and should not be recorded by Kiddle. Those are the ones that are
problematic, especially since we have a large number of javadocs
committed.

$ find src/main/jbake/assets/apidocs -type f | wc -l
7127

Those are the ones we'd like excluded, if at all possible.

Thanks,

Robert

[1]: https://github.com/apache/sling-site/tree/master/src/main/jbake




I've added a change to the scanners, so they will put a list of files 
changes into each commit object we record. This is likely going to 
require a complete re-scan of all things sling...which in theory is 
fine, as I _was_ planning on moving the demo server to a new box anyway. 
I'll let y'all know more when I have that worked out in my mind :)

After the move and re-scan, it should be possible to exclude by file paths.

Re: Feedback Requested from Kibble Demo Projects

[Daniel Gruno]

On 09/12/2018 12:14 PM, Robert Munteanu wrote:

3. I was unable to find what the Pony Factor is. Some documentation
(inline in the app or in the manual) would be great.


https://ke4qqq.wordpress.com/2015/02/08/pony-factor-math/

We should put a link to that or add the description to the 
documentation, yeah. it's basically a risk factor.




-

Guess that about covers it. Overall I'm happy Kibble exists I can
extract stats about our projects so thanks for that!

Robert

[1]: 
https://demo.kibble.apache.org/contributors.html?page=mvp=adb6da82a758f0cc0d3b650b9e839c15c0e0f7d92ff5f074821d7e98

Re: Possible to exclude directories from analysis?

[Daniel Gruno]

On 09/12/2018 12:00 PM, Robert Munteanu wrote:

On Sat, 2018-09-08 at 12:54 +0200, Daniel Gruno wrote:

On 09/05/2018 08:38 PM, Robert Munteanu wrote:

Hi,

I'm using the demo Kibble instance to visualise code contributions
for
the Apache Sling project. One thing I noticed is that Kibble things
we're 75% HTML, which is not right - we're a Java project.

I think it's due to the fact that we use gitpubsub and have
registered
our github.com/apache/sling-site repository with kibble. That
repository's master branch holds all the HTML we publish, including
lots of Javadocs, Maven plug-in documentation, etc.


The easiest path would be to simply exclude the sling-site repository
in
your reports. If you're using a quick filter, instead of filtering
on
'sling', you could do a negative lookahead and filter on
'sling(?!-site)' as the quick filter accepts regular expressions.


Thanks for the suggestestion. I ended up excluding the sling-site
repository completely from the 'Apache Sling' view. It's not ideal as
it does not capture documentation contributions, which are quite
important as well.

It would be great if in the future we would have a more fine-grained
solution.


Ideal solutions are rare :)
Could you elaborate on exactly *what* you want to see, and what you want 
to filter away? Some things may be possible, but when you have to do 
aggregations on something like 3 million commits in real-time, it gets 
tricky to exclude paths and individual files without throwing a huge lag 
spike into the mix.




Thanks,

Robert

Re: Relative and Comparative Mood Analysis

[Daniel Gruno]

On 09/11/2018 08:03 PM, Sharan Foga wrote:

Hi Daniel

Thanks for the response. I've included some comments inline.

On 2018/09/11 10:40:46, Daniel Gruno  wrote:

On 09/11/2018 12:23 PM, Sharan Foga wrote:

Hi All

On the mood analysis screen  https://s.apache.org/YImU

I’ve noticed the toggle on the mood analysis so would like to know exactly what 
this means.

Also if you click the box to toggle between the two modes then it’s not clear 
from the screen when you are using ‘Relative’ mode and when you are using 
‘Comparative’ mode.


The text in the charts change when you tick the box, and should explain
what it is.


Ah.. I see that now but am glad to hear the elaboration as it wasn't clear to 
me.

  If that's not clear enough, let me elaborate:


Standard (unticked) mode


So does standard mode = relative mode?


"relative/comparative" is *one* mode, it's either off or on :)
two words for the same mode.

If enabled, it compares moods to *all* lists it has, regardless of 
filtering.

If off, it only shows how that list (or those lists) are.



shows the moods as an 'absolute' value

unrelated to the average across lists. Moods are still relative to each
other (in that the median is close to 100), but they only reflect the
list(s) you have chosen to analyze.


Ok so to me this means that the relative mode shows the mood analysis say 
across one project if that is the filter being used. So with my Httpd example - 
in relative mode this gives me an indicator of the general mood profile for 
Httpd based on and in relation to the other moods that the Httpd project 
generates.

And the mood gauge in relative mode gives me an overall indication of the 
filtered project or view (e.g Httpd) based on the moods found in the sources 
for the filter. So the more positive the moods found, the higher the rating 
will be.

If my understanding is wrong then please correct me as I really need to get 
this as it is going to be part of my research :-)



Relative mode (ticked)


Should this be Comparative mode?

uses the same values, but compares them to the

overall moods found in *all* lists in the database.


So does this mean that you are comparing against all the sources for the 
organisation or if you have multiple organisations will it compare against all 
organisations?

 From what you say, it sounds like comparative mode (if that is what it is) 
compares the mood profile against everything. So Kibbletest will compare 
Httpd's mood profile against all the other projects loaded into Kibbletest.


Thus, on the right

side, a score of 50% or above means you are doing as well or better than
other lists in terms of happy moods, and a score lower than that would
mean the list(s) don't quite reflect the overall mood of all lists. On
the left side, the values are relative to overall mood as well, in that
100 means 'same as all the other lists', anything higher means a more
prevalent occurrence on the list(s) you picked, whereas a value lower
than 100 means it's less frequent there compared to the overall average.



OK this is good and this is clearer

So now I'm after more :-) Is there a way to compare the mood profiles of two 
different projects. For example, if I wanted to compare the profiles of Httpd 
and Beam, how could I do that?


I hope that helps, and if we need to work on the UI explanations, maybe
add a document to the documentation, then so be it :)


It does - thanks. I'd probably be happier with clearer explanations so will 
document what I can as I'm going through.

Thanks
Sharan



With regards,
Daniel.



Thanks
Sharan

Re: Feedback Requested from Kibble Demo Projects

[Daniel Gruno]

On 09/10/2018 10:46 AM, Sharan wrote:

Hi

The quick filter seems to need a view, so I've created a Netbeans view 
under the Kibble guest login.


the sub-filter is 'sticky', so you don't really need to create permanent 
filters much. If you type 'netbeans' in the sub-filter box, it should 
stick when you navigate to other pages on the site.




Thanks
Sharan

On 10.9.2018 10:24, Daniel Gruno wrote:

On 09/10/2018 10:16 AM, Geertjan Wielenga wrote:
Not sure what Kibble is, this is the first time I've heard of it -- 
but very interested to know what the statistics are that it shows 
about Apache NetBeans.


https://demo.kibble.apache.org/
log in, go to 'data points', type in 'netbeans' in the quick filter 
and click the sub-filter button :) and...look around!


Thanks,

Gj

On Mon, Sep 10, 2018 at 10:07 AM, <mailto:sha...@apache.org>> wrote:


    Hi All

    Thanks very much for being one of the projects that is part of the
    Kibble demo instance. We hope that you are making use of the
    information Kibble shows about your project. We have had our demo
    instance up and running for most of this year so now we'd like to
    get some feedback from you about how you are using Kibble.

    Some example questions that we’d be interested in are as follows:

  * How often are you using Kibble and what are you using it for?
  * How does information from Kibble inform decisions or is useful
    in other ways?
  * What questions about your community can you currently not answer
    and what information could Kibble provide to help answer these
    questions?
  * What problems do you have with using Kibble?

    Please note that even if you are not using Kibble then it would be
    good to understand why (e.g perhaps the documentation for using it
    isn’t clear etc). Also if you have any other feedback about Kibble
    then we’d be happy to hear it so that we can work on improving it
    for the future.

    If you have any questions then please feel free to ask them on our
    Kibble dev list (dev@kibble.apache.org
    <mailto:dev@kibble.apache.org>). We're looking forward to receiving
    your feedback.

    Thanks
    Sharan

Re: Feedback Requested from Kibble Demo Projects

[Daniel Gruno]

On 09/10/2018 10:16 AM, Geertjan Wielenga wrote:
Not sure what Kibble is, this is the first time I've heard of it -- but 
very interested to know what the statistics are that it shows about 
Apache NetBeans.


https://demo.kibble.apache.org/
log in, go to 'data points', type in 'netbeans' in the quick filter and 
click the sub-filter button :) and...look around!


Thanks,

Gj

On Mon, Sep 10, 2018 at 10:07 AM, > wrote:


Hi All

Thanks very much for being one of the projects that is part of the
Kibble demo instance. We hope that you are making use of the
information Kibble shows about your project. We have had our demo
instance up and running for most of this year so now we'd like to
get some feedback from you about how you are using Kibble.

Some example questions that we’d be interested in are as follows:

  * How often are you using Kibble and what are you using it for?
  * How does information from Kibble inform decisions or is useful
in other ways?
  * What questions about your community can you currently not answer
and what information could Kibble provide to help answer these
questions?
  * What problems do you have with using Kibble?

Please note that even if you are not using Kibble then it would be
good to understand why (e.g perhaps the documentation for using it
isn’t clear etc). Also if you have any other feedback about Kibble
then we’d be happy to hear it so that we can work on improving it
for the future.

If you have any questions then please feel free to ask them on our
Kibble dev list (dev@kibble.apache.org
). We're looking forward to receiving
your feedback.

Thanks
Sharan

Re: Possible to exclude directories from analysis?

[Daniel Gruno]

On 09/05/2018 08:38 PM, Robert Munteanu wrote:

Hi,

I'm using the demo Kibble instance to visualise code contributions for
the Apache Sling project. One thing I noticed is that Kibble things
we're 75% HTML, which is not right - we're a Java project.

I think it's due to the fact that we use gitpubsub and have registered
our github.com/apache/sling-site repository with kibble. That
repository's master branch holds all the HTML we publish, including
lots of Javadocs, Maven plug-in documentation, etc.


The easiest path would be to simply exclude the sling-site repository in 
your reports. If you're using a quick filter, instead of filtering on 
'sling', you could do a negative lookahead and filter on 
'sling(?!-site)' as the quick filter accepts regular expressions.


With regards,
Daniel.



Is it possible to exclude a certain directory from analysis, to make
the statistics more relevant?

Thanks,

Robert

Re: Do we have a Wiki?

[Daniel Gruno]

On 08/09/2018 01:21 PM, Sharan Foga wrote:

Hi All

The question is in the subject line - do we have a wiki? I was looking for a 
place to start putting together the list of pilot projects using Kibble in 
preparation for gathering their feedback.


You can put in a request for a cwiki space on selfserve.a.o


Thanks
Sharan

Goal of Kibble [was Re: Personality and Value Indicators]

[Daniel Gruno]

On 08/03/2018 07:55 PM, Vaibhav Kumar wrote:

  Hi,
I'm new on this open source can someone try explaining what we are trying to 
achieve here because reading a single line (Apache Kibble is a suite of tools 
for collecting, aggregating and visualizing activity in software projects) from 
READ.md doesn't give me any clarity.
Any help on this would be appreciated
Regards,Vaibhav


Hi Vaibhav, let me expand on the text:

Apache Kibble is a suite of tools (a master server and nodes with 
various scanner programs) designed to:


- collect: scan code repositories, mailing lists, forums, issue/bug 
trackers, continuous integration tools and more, and put the data into a 
database (in other words; what's going on??)
- aggregate: use the collected data to generate statistics, trends, 
analyses and more (in other words; what can we deduct from the data we 
have gathered?)
- visualize: via the web, you can access the kibble master and see the 
collected data and analysed stats in a multitude of charts and lists, 
with the option to filter, search and more.


We have a demo at https://demo.kibble.apache.org/ if you want to see it 
in action.


I hope this answers some of your questions.

With regards,
Daniel.

Re: Personality and Value Indicators

[Daniel Gruno]

On 08/03/2018 01:16 PM, Daniel Gruno wrote:

On 07/16/2018 08:13 PM, sha...@apache.org wrote:



On 16.7.2018 19:32, Daniel Gruno wrote:

On 07/15/2018 07:40 AM, Sharan Foga wrote:

Hi All

I seem to remember that there is a parameter or a graphic that 
highlights the mood of the community based on some emotional 
indicators.  Could kibble be used to also pull out personality or 
value indicators etc ? (a bit like the Watson API does….)


Absolutely!
The demo uses PicoAPI for sentiment analysis, and it (IIRC) does also 
supply these indicators. There is probably a lot of indicators buried 
in the database that we aren't properly showing just yet. As with the 
other topic (pw reset), I can look at this in about two weeks time :)


That's great news! OK when you get back let's start a discussion about 
it.


Looks like our current provider uses the following traits:

anger
anticipation
disgust
fear
joy
negative
neutral
positive
sadness
surprise
trust


Correction, the complete list is:

analytical
anger
anticipation
confident
disgust
fear
joy
negative
neutral
positive
sadness
surprise
tentative
trust




There are some slight variations between providers (Kibble itself does 
not currently provide analysis itself), but they seem to have roughly 
the same traits, albeit worded differently.




Thanks
Sharan




With regards,
Daniel.



The reason I’m interested is that I’m doing some research into the 
Apache Way, ASF culture and values etc for an academic paper so 
wanted to see if we could somehow try and measure it.


Thanks
Sharan

Re: Personality and Value Indicators

[Daniel Gruno]

On 07/15/2018 07:40 AM, Sharan Foga wrote:

Hi All

I seem to remember that there is a parameter or a graphic that highlights the 
mood of the community based on some emotional indicators.  Could kibble be used 
to also pull out personality or value indicators etc ? (a bit like the Watson 
API does….)


Absolutely!
The demo uses PicoAPI for sentiment analysis, and it (IIRC) does also 
supply these indicators. There is probably a lot of indicators buried in 
the database that we aren't properly showing just yet. As with the other 
topic (pw reset), I can look at this in about two weeks time :)


With regards,
Daniel.



The reason I’m interested is that I’m doing some research into the Apache Way, 
ASF culture and values etc for an academic paper so wanted to see if we could 
somehow try and measure it.

Thanks
Sharan

Re: Problem with Kibble Demo Password Reset?

[Daniel Gruno]

On 07/16/2018 07:01 AM, Sharan Foga wrote:

Hi

I'm still haven't received a password reset email yet - but the good news is 
that I remembered my password so am in :-).

For the demo - can we check that the password reset functionality is working 
properly as we have a few people from the pilot projects that are not using the 
guest login? (And I'm happy to be part of the testing if it helps)


I think it's in a WIP state at the moment, I can look into it when I get 
back from the states (I am traveling at the moment, so my main focus is 
just getting work done, and a bit of FLOSS if I have the time).


Thanks
Sharan

On 2018/07/15 13:38:49, sha...@apache.org wrote:

Hi

I’ve been trying to log into the kibble demo today and realised that
I’ve forgotten my password so used the password reset. I haven’t
received anything yet – I was expecting an instantaneous thing – but so
far nothing. Is this normal or is something not working correctly?

Thanks
Sharan

Cron jobs on kibble demo set up

[Daniel Gruno]

Hi folks,
I've set the kibble scanners to run daily at around midnight on the demo
server now (02:30 CET). Jenkins, Buildbot and Travis (when we get that
done) will run every 2 hours continuously, and the other scans will run
at 2:30.

With regards,
Daniel.

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

On 01/16/2018 09:43 PM, Robert Munteanu wrote:
> Hi,
> 
> On Mon, 2017-11-27 at 19:26 +0100, Daniel Gruno wrote:
>> Hi there, fellow Apache projects!
>>
>> The Apache Kibble project serves as a practical implementation of
>> metrics deemed to be helpful for open source projects trying to
>> understand where their project is, was, and is headed.
>>
>> As such, we need help in determining which metrics projects either
>> already use and consider useful for measuring project health or which
>> metrics they would love to have and use.
>>
>> We are looking for projects interested in participating in the Kibble
>> demo instance ( https://demo.kibble.apache.org/ ) and sending
>> feedback
>> to the Kibble project on which parts they find useful, which elements
>> they find useless and which ideas they would love to see implemented
>> to
>> better gauge the health and activity of their project.
> 
> I would be interesting in gathering information about the Apache Sling
> project. Now, I'm aware that we might be a bit special, having in
> access of 250 projects ( canonical listing at [1] ), so I won't be
> shocked if you reject or delay our inclusion.

The actual number of repos is somewhat of a moot factor - what really
matters is the code size and activity. I've been doing some timed scans
on the demo server, and I think we can safely bump the resource object
count to 600-700 without any real difficult (note that this would still
only be about an 1/8th of all of ASF).

If someone wants to add Sling, go right ahead :)

> 
> We are interested in Kibble since we have split our source repositories
> which makes it hard to generate aggregate statistics of our development
> activity. For me personally the most important indicators to track
> would be the number of distinct contributors over the last X months,
> which is a fair indicator of community engagement.
> 
> Thanks,
> 
> Robert
> 
> 
> [1]: https://github.com/apache/sling-aggregator/blob/master/default.xml
> 

Getting started on documentation for Kibble

[Daniel Gruno]

Hi folks,
I've started writing some documentation on setup/usage for Kibble inside
the main repo, under docs/. It's using the RsT format, which means we
can have ReadTheDocs generate the documentation on commit for us,
available at https://apache-kibble.readthedocs.io/

So far I've gotten the installation/setup instructions somewhat done,
and next I'll move onto how to set up an organisation and add sources.
If anyone is willing to 'document as they explore', it would be greatly
appreciated. In particular, I'd be interested in how people go about
using the UI, any secrets/tips they have etc, but also just general
usage documentation.

There seems to be a growing interest in trying out the software, so
having proper documentation would be awesome and helpful.

With regards,
Daniel.

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

On 2018/01/09 18:05:39, Jacques Le Roux <jacques.le.r...@les7arts.com> wrote: 
> Thanks Guys,
> 
> Works perfectly :)

Great to hear :)
As for SVN, it's a difficult thing to scan, so there is no support for it yet.
SVN also lacks a proper committer/author scheme, so the results are not easy to 
interpret.
> 
> Jacques
> 
> 
> Le 09/01/2018 à 16:08, Rafael Weingärtner a écrit :
> > Awesome!
> > I have invited Jacques now. I used his apache email account 
> > *jler...@apache.org
> > <jler...@apache.org>*
> >
> > On Tue, Jan 9, 2018 at 1:04 PM, Daniel Gruno <humbed...@apache.org> wrote:
> >
> >> On 01/09/2018 12:40 PM, Rafael Weingärtner wrote:
> >>> Ah, so you created your own user. You are not using the guest user?
> >>>
> >>> I am still not able to do this process there. someone would need to add
> >> you
> >>> as a user into Apache org there.
> >> I've fixed the member API now, so you should be able to invite Jacques
> >> to the organisation.
> >>
> >>> On Tue, Jan 9, 2018 at 9:37 AM, Jacques Le Roux <
> >>> jacques.le.r...@les7arts.com> wrote:
> >>>
> >>>> Thanks Rafael,
> >>>>
> >>>> But I can't (permission) add them either using my @a.o address or my
> >> main
> >>>> own
> >>>>
> >>>> Jacques
> >>>>
> >>>>
> >>>>
> >>>> Le 09/01/2018 à 12:15, Rafael Weingärtner a écrit :
> >>>>
> >>>>> OFBIZ's SVN was not added. However, we added Github repositories:
> >>>>> https://github.com/apache/ofbiz-framework.git
> >>>>> https://github.com/apache/ofbiz-plugins.git
> >>>>> https://github.com/apache/ofbiz.git
> >>>>>
> >>>>> I am not sure if the integration with SVN is working...
> >>>>>
> >>>>> On Tue, Jan 9, 2018 at 9:12 AM, Jacques Le Roux <
> >>>>> jacques.le.r...@les7arts.com> wrote:
> >>>>>
> >>>>> Hi Daniel,
> >>>>>> When I try to add the trunk svn sources to the OFBiz view I created it
> >>>>>> says I can't despite using my @a.o address
> >>>>>>
> >>>>>> How to add myself to an organisation? At
> >> https://demo.kibble.apache.org
> >>>>>> /organisations.html?page=org I get "You don't seem to belong to any
> >>>>>> organisations just yet."
> >>>>>>
> >>>>>> And when I get to https://demo.kibble.apache.org
> >>>>>> /organisations.html?page=org-users I only read "Loading, hang on
> >>>>>> tight..!
> >>>>>> " even after some minutes (I get a lot of that everywhere ;))
> >>>>>>
> >>>>>> BTW I guess it's based on snoot.io right? (I have got snoot to work
> >>>>>> right)
> >>>>>>
> >>>>>> Thanks
> >>>>>>
> >>>>>> Jacques
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Le 27/11/2017 à 19:26, Daniel Gruno a écrit :
> >>>>>>
> >>>>>> Hi there, fellow Apache projects!
> >>>>>>> The Apache Kibble project serves as a practical implementation of
> >>>>>>> metrics deemed to be helpful for open source projects trying to
> >>>>>>> understand where their project is, was, and is headed.
> >>>>>>>
> >>>>>>> As such, we need help in determining which metrics projects either
> >>>>>>> already use and consider useful for measuring project health or which
> >>>>>>> metrics they would love to have and use.
> >>>>>>>
> >>>>>>> We are looking for projects interested in participating in the Kibble
> >>>>>>> demo instance ( https://demo.kibble.apache.org/ ) and sending
> >> feedback
> >>>>>>> to the Kibble project on which parts they find useful, which elements
> >>>>>>> they find useless and which ideas they would love to see implemented
> >> to
> >>>>>>> better gauge the health and activity of their project.
> >>>>>>>
> >>>>>>> Ini

Kibble and Key Phrase Extraction

[Daniel Gruno]

Hi folks,
before Christmas, I had some very informal talks with some other Kibble
folks in various places about Key Phrase Extraction (KPE) and how we
might be able to use that in Kibble.

KPE is the process of taking a longer text, for instance an email, and
extracting small sentences or words that are at the center of the text -
as an example, taking chapter one from a Christmas Carol would (among
other things) point to Scrooge, Marley and 'Dead as a Door-nail' as key
elements. This isn't to say that you can always grasp exactly what all
10,000 words in that chapter was about, but you get some elements that
play a key role in it. As an aside, KPE is supported by most text
analysis services out there (I've tried it with Watson, Azure and
picoAPI thus far)

My idea with KPE is so generate a list of keywords and sentences to
accompany each email (and possibly issues/tickets?) so that we can
generate a map of "hot topics" both in projects and across them, looking
for commonalities and trends. This _could_ show that most projects have
similar work-flows/activities in common, or it could show the exact
opposite - I don't know yet, but I sure would love to find out! It could
perhaps also tell us (ngram-style) something about how technology
progresses over time by following the occurrences of certain
words/sentences in projects as a time-series.

I would _love_ to get some input on this (especially as far as 'what can
we use this for, if anything' is concerned), and I will probably start
some basic KPE extraction tests in the coming days. As mentioned
elsewhere, the Kibble demo instance has virtually unlimited text
analysis credits at the moment, including KPE, so we might as well make
use of that :)

WDYT?

With regards,
Daniel.

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

ache-sling-commons-cache-portal.git
>> https://github.com/apache/commons-numbers.git
>> https://github.com/apache/commons-functor.git
>> https://github.com/apache/commons-jxpath.git
>> https://github.com/apache/commons-chain.git
>> https://github.com/apache/commons-scxml.git
>> https://github.com/apache/commons-proxy.git
>> https://github.com/apache/commons-rng.git
>> https://github.com/apache/commons-jci.git
>> https://github.com/apache/commons-weaver.git
>> https://github.com/apache/commons-bsf.git
>> https://github.com/apache/commons-jelly.git
>> https://github.com/apache/sling-org-apache-sling-commons-cache-container-test.git
>> https://github.com/apache/sling-org-apache-sling-commons-metrics-rrd4j.git
>>
>> Mailing lists:
>> https://lists.apache.org/list.html?d...@commons.apache.org
>> https://lists.apache.org/list.html?u...@commons.apache.org
>> https://lists.apache.org/list.html?d...@commonsrdf.apache.org
>>
>>
>> Jira:
>> https://issues.apache.org/jira/browse/COMMONSRDF
>> https://issues.apache.org/jira/browse/COMMONSSITE
>> https://issues.apache.org/jira/browse/ATTRIBUTES
>> https://issues.apache.org/jira/browse/BCEL
>> https://issues.apache.org/jira/browse/BEANUTILS
>> https://issues.apache.org/jira/browse/BETWIXT
>> https://issues.apache.org/jira/browse/BSF
>> https://issues.apache.org/jira/browse/CHAIN
>> https://issues.apache.org/jira/browse/CLI
>> https://issues.apache.org/jira/browse/CODEC
>> https://issues.apache.org/jira/browse/COLLECTIONS
>> https://issues.apache.org/jira/browse/COMPRESS
>> https://issues.apache.org/jira/browse/CONFIGURATION
>> https://issues.apache.org/jira/browse/CRYPTO
>> https://issues.apache.org/jira/browse/CSV
>> https://issues.apache.org/jira/browse/DAEMON
>> https://issues.apache.org/jira/browse/DBCP
>> https://issues.apache.org/jira/browse/DBUTILS
>> https://issues.apache.org/jira/browse/DIGESTER
>> https://issues.apache.org/jira/browse/DISCOVERY
>> https://issues.apache.org/jira/browse/DORMANT
>> https://issues.apache.org/jira/browse/EL
>> https://issues.apache.org/jira/browse/EMAIL
>> https://issues.apache.org/jira/browse/EXEC
>> https://issues.apache.org/jira/browse/FEEDPARSER
>> https://issues.apache.org/jira/browse/FILEUPLOAD
>> https://issues.apache.org/jira/browse/FUNCTOR
>> https://issues.apache.org/jira/browse/IMAGING
>> https://issues.apache.org/jira/browse/IO
>> https://issues.apache.org/jira/browse/JCI
>> https://issues.apache.org/jira/browse/JCS
>> https://issues.apache.org/jira/browse/JELLY
>> https://issues.apache.org/jira/browse/JEXL
>> https://issues.apache.org/jira/browse/JXPATH
>> https://issues.apache.org/jira/browse/LANG
>> https://issues.apache.org/jira/browse/LAUNCHER
>> https://issues.apache.org/jira/browse/LOGGING
>> https://issues.apache.org/jira/browse/MATH
>> https://issues.apache.org/jira/browse/MODELER
>> https://issues.apache.org/jira/browse/NET
>> https://issues.apache.org/jira/browse/NUMBERS
>> https://issues.apache.org/jira/browse/OGNL
>> https://issues.apache.org/jira/browse/POOL
>> https://issues.apache.org/jira/browse/PRIMITIVES
>> https://issues.apache.org/jira/browse/PROXY
>> https://issues.apache.org/jira/browse/RESOURCES
>> https://issues.apache.org/jira/browse/RNG
>> https://issues.apache.org/jira/browse/SANDBOX
>> https://issues.apache.org/jira/browse/SANSELAN
>> https://issues.apache.org/jira/browse/SCXML
>> https://issues.apache.org/jira/browse/TEXT
>> https://issues.apache.org/jira/browse/TRANSACTION
>> https://issues.apache.org/jira/browse/VALIDATOR
>> https://issues.apache.org/jira/browse/VFS
>> https://issues.apache.org/jira/browse/WEAVER
>> https://issues.apache.org/jira/browse/MFCOMMONS
>> https://issues.apache.org/jira/browse/WSCOMMONS
>> https://issues.apache.org/jira/browse/XMLCOMMONS
>> https://issues.apache.org/jira/browse/XGC
>>
>> On Tue, Dec 19, 2017 at 10:50 AM, Daniel Gruno <humbed...@apache.org> wrote:
>>
>>> On 12/19/2017 01:09 PM, Rafael Weingärtner wrote:
>>>> Some time has gone by and other projects were not added.
>>>> So, should we add Apache commons?
>>>
>>> sgtm
>>>
>>>>
>>>> On Mon, Dec 4, 2017 at 9:45 AM, Rafael Weingärtner <
>>>> rafaelweingart...@gmail.com> wrote:
>>>>
>>>>> ok, I will do that.
>>>>>
>>>>> On Mon, Dec 4, 2017 at 9:42 AM, Daniel Gruno <humbed...@apache.org>
>>> wrote:
>>>>>
>>>>>> 

Re: Spinning repo picker?

[Daniel Gruno]

On 12/12/2017 11:49 AM, Rafael Weingärtner wrote:
> I might have karma, but I am still a newbie ;P
> So, it seems that I was in the right direction. the problem is that when I
> access that page, I only get this message: "Loading, hang on tight..!" and
> nothing happens.
> I checked the request you guys are doing to retrieve some data. And there
> is a message saying: "Only organisation owners can view this list."

Hm, sounds like a bug then, if you're logged in with your apache.org
account. I'll take a look at that later :) In the meantime, I've added
myrle now.

> 
> On Tue, Dec 12, 2017 at 8:45 AM, Daniel Gruno <humbed...@apache.org> wrote:
> 
>> On 12/12/2017 09:40 AM, Rafael Weingärtner wrote:
>>> Ah, you do not add yourself. That is the problem.
>>> I also have not figured out how to add you to Apache organization there
>> :(
>>>
>>> Let's wait for Daniel or others that have karma for this...
>>
>> You have karma here, Rafael :)
>> https://demo.kibble.apache.org/organisations.html?page=org-users is
>> where you would add new users. As stated earlier, all we really need is
>> the email Myrle used to sign up for an account with, and we can add her
>> as a member.
>>
>>>
>>> On Tue, Dec 12, 2017 at 6:39 AM, Myrle Krantz <my...@apache.org> wrote:
>>>
>>>> None -- I also haven't figured out how to add myself to one yet.
>>>>
>>>> On Tue, Dec 12, 2017 at 9:34 AM, Rafael Weingärtner
>>>> <rafaelweingart...@gmail.com> wrote:
>>>>> When you login at page [1], there is a section called "Your
>>>> organizations";
>>>>> what organizations do you have there?
>>>>>
>>>>> [1] https://demo.kibble.apache.org/organisations.html?page=org
>>>>>
>>>>> On Tue, Dec 12, 2017 at 6:11 AM, Myrle Krantz <my...@apache.org>
>> wrote:
>>>>>
>>>>>> (Sorry Fineract list -- I sent this to the wrong list.  It was
>>>>>> intended for the kibbles list.)
>>>>>>
>>>>>> On Tue, Dec 12, 2017 at 8:36 AM, Myrle Krantz <my...@apache.org>
>> wrote:
>>>>>>> Hey all,
>>>>>>>
>>>>>>> Just FYI: I've announced Fineract's participation in kibbles on the
>>>>>>> fineract dev list.  The first time I looked through it, I was quite
>>>>>>> pleased with the level and kind of information I could get about our
>>>>>>> project.
>>>>>>>
>>>>>>> ...but when I try to look *after* I've logged in as my...@apache.org
>> ,
>>>>>>> the repository picker doesn't load.  I'm using Chrome 63.0.3239.84
>>>>>>> running on Ubuntu.  The guest login seems to work just fine
>> strangely.
>>>>>>> Am I doing something wrong?  Or have a found a bug?
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Myrle
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rafael Weingärtner
>>>>
>>>
>>>
>>>
>>
>>
> 
> 

Re: Spinning repo picker?

[Daniel Gruno]

On 12/12/2017 09:40 AM, Rafael Weingärtner wrote:
> Ah, you do not add yourself. That is the problem.
> I also have not figured out how to add you to Apache organization there :(
> 
> Let's wait for Daniel or others that have karma for this...

You have karma here, Rafael :)
https://demo.kibble.apache.org/organisations.html?page=org-users is
where you would add new users. As stated earlier, all we really need is
the email Myrle used to sign up for an account with, and we can add her
as a member.

> 
> On Tue, Dec 12, 2017 at 6:39 AM, Myrle Krantz  wrote:
> 
>> None -- I also haven't figured out how to add myself to one yet.
>>
>> On Tue, Dec 12, 2017 at 9:34 AM, Rafael Weingärtner
>>  wrote:
>>> When you login at page [1], there is a section called "Your
>> organizations";
>>> what organizations do you have there?
>>>
>>> [1] https://demo.kibble.apache.org/organisations.html?page=org
>>>
>>> On Tue, Dec 12, 2017 at 6:11 AM, Myrle Krantz  wrote:
>>>
 (Sorry Fineract list -- I sent this to the wrong list.  It was
 intended for the kibbles list.)

 On Tue, Dec 12, 2017 at 8:36 AM, Myrle Krantz  wrote:
> Hey all,
>
> Just FYI: I've announced Fineract's participation in kibbles on the
> fineract dev list.  The first time I looked through it, I was quite
> pleased with the level and kind of information I could get about our
> project.
>
> ...but when I try to look *after* I've logged in as my...@apache.org,
> the repository picker doesn't load.  I'm using Chrome 63.0.3239.84
> running on Ubuntu.  The guest login seems to work just fine strangely.
> Am I doing something wrong?  Or have a found a bug?
>
> Best Regards,
> Myrle

>>>
>>>
>>>
>>> --
>>> Rafael Weingärtner
>>
> 
> 
> 

Listing 3rd party support

[Daniel Gruno]

Hi folks,
As Kibble gets more advanced, is starts supporting more and more 3rd
party services and APIs. Should we compile a list of these for the web site?

As I see it, this would be broken into two categories:

1: Source support (GitHub, JIRA, BugZilla, Twitter etc etc)
2: API Service Brokers (IBM Watson, MS Azure, picoAPI etc for analyses)

WDYT?

The idea is two-fold: One, we list which source objects people can work
with and what analysis services we support, and two, it might provide
incentives for 3rd party entities to support us with credits etc for
their services, furthering our capabilities to map/chart community
health/activity. This would of course be done on a neutral basis where
any service/source of value to the project would be listed upon request
in a random order.

With regards,
Daniel.

Re: Guest login or "real" login?

[Daniel Gruno]

On 12/08/2017 11:27 AM, Rafael Weingärtner wrote:
> Hi Myrle,
> What other sources are these? I only found those sources (Github, mailing
> list and Jira) that I listed in my previous email.
> 
> Regarding the account, I believe you can create you own, but it might be a> 
> good idea to wait for Daniel's two cents of wisdom here ;)

You can create your own or use the guest account.
If you create your own, please let us know which email you signed up
with, so we can add you to the org in the demo. Otherwise I'm afraid
you'll see a very blank demo :)

> 
> On Fri, Dec 8, 2017 at 6:59 AM, Myrle Krantz  wrote:
> 
>> Hey Rafael,
>>
>> Thank you.
>>
>> I'm back from my business trip now and finally had a chance to sit
>> down and look at this.  I will definitely be using this in the future.
>> Even in its current early development state, it has provided me with
>> some valuable insights.  The sources are complete for now, but we will
>> be importing the Fineract CN repositories to gitbox over the next
>> week, and I hope we can add them to the analysis set.
>>
>> Fineract CN is over 30 repositories though.  Is that too many?
>>
>> One more question: is everyone using the guest login?  Or can I get a
>> "real" login?
>>
>> Best Regards,
>> Myrle
>>
>> On Thu, Nov 30, 2017 at 2:39 PM, Rafael Weingärtner
>>  wrote:
>>> Hi Myrle,
>>> I will add these source for Fineract:
>>>
>>> GitHub repo(s):
>>> https://github.com/apache/fineract.git
>>> https://github.com/apache/fineract-site.git
>>>
>>> Mailing lists:
>>> https://lists.apache.org/list.html?u...@fineract.apache.org
>>> https://lists.apache.org/list.html?d...@fineract.apache.org
>>>
>>> Jira:
>>> https://issues.apache.org/jira/projects/FINERACT
>>>
>>> If there is something wrong or something else that I left out, please do
>> not
>>> hesitate to tell me.
>>
> 
> 
> 

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

On 12/04/2017 12:41 PM, Rafael Weingärtner wrote:
> So, should we wait a bit more before adding Apache commons?

How about if we hold off for 3-4 days, and if we haven't hit >150ish
sources by then, we can add them?

> 
> On Mon, Dec 4, 2017 at 9:09 AM, Daniel Gruno <humbed...@apache.org> wrote:
> 
>> On 12/03/2017 12:25 PM, Rafael Weingärtner wrote:
>>> Gilles, I actually do not know the reason behind this limit on data
>>> sources. I am fairly new to kibble project. Let´s see what Daniel or
>> others
>>> have to say about this limit.
>>
>> The limit is somewhat arbitrary, but we would favor smaller projects
>> over behemoths like Commons at the moment. We're trying to, at one hand,
>> figure out how many sources we can cram into the single demo box we have
>> at our disposal, while also trying to get as many projects involved as
>> possible. I know from elsewhere that we would need something like 5-6
>> times the resources we currently have, if we were to put the entire ASF
>> into the Kibble demo, and we don't currently have a budget for that.
>>
>> Now, the entire ASF would be around 7,500 source objects, so 100 is a
>> very low limit, granted. If the Commons project has an actual interest
>> in using Kibble for reports and what not, we can get them added - the
>> thing to remember here is merely that we'd favor getting as much bang
>> for our buck as possible.
>>
>> I did some quick checks of available resources, and we're not using a
>> whole bunch yet, so we can probably aim for 200 source objects for now,
>> and still be fine.
>>
>> With regards,
>> Daniel.
>>
>>>
>>> On Sat, Dec 2, 2017 at 10:30 PM, Gilles Sadowski <er...@apache.org>
>> wrote:
>>>
>>>> Hi.
>>>>
>>>> On 2017-12-02 21:27, Rafael Weingärtner <rafaelweingart...@gmail.com>
>>>> wrote:
>>>>> How should I handle adding apache commons?
>>>>>
>>>>> If I add all of the sources we will go over the limit of 100 sources.
>> In
>>>>> Github there are 72 projects, and on Jira there other 70+ projects.
>>>>
>>>> A lot of projects, yes; but most of them are quite small, and many
>>>> have very low activity.
>>>> Would that mitigate usage resources?
>>>> If the number above is a hard limit, I'll ask the ML for a priority
>> list...
>>>>
>>>>> At
>>>>> least the mailing list seems to be only the ones found here:
>>>>> https://lists.apache.org/list.html?u...@commons.apache.org
>>>>
>>>> The "dev" ML should rather be selected.
>>>>
>>>> Gilles
>>>>
>>>>>
>>>>> On Fri, Dec 1, 2017 at 11:44 AM, Gilles Sadowski <er...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Hello.
>>>>>>
>>>>>> On 2017-11-27 19:26, Daniel Gruno <humbed...@apache.org> wrote:
>>>>>>> [...]
>>>>>>>
>>>>>>> PS: Please note, we have limited capacity for these tests. We cannot
>>>>>>> have every single ASF project in the demo, and we reserve the rights
>>>> to
>>>>>>> pick the projects that can participate, should we get a lot of
>>>> requests.
>>>>>>>
>>>>>>
>>>>>> Would you consider adding the "Commons"[1] projects?
>>>>>>
>>>>>> Regards,
>>>>>> Gilles
>>>>>>
>>>>>> [1] http://commons.apache.org/
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Rafael Weingärtner
>>>>>
>>>>
>>>
>>>
>>>
>>
>>
> 
> 

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

On 12/03/2017 12:25 PM, Rafael Weingärtner wrote:
> Gilles, I actually do not know the reason behind this limit on data
> sources. I am fairly new to kibble project. Let´s see what Daniel or others
> have to say about this limit.

The limit is somewhat arbitrary, but we would favor smaller projects
over behemoths like Commons at the moment. We're trying to, at one hand,
figure out how many sources we can cram into the single demo box we have
at our disposal, while also trying to get as many projects involved as
possible. I know from elsewhere that we would need something like 5-6
times the resources we currently have, if we were to put the entire ASF
into the Kibble demo, and we don't currently have a budget for that.

Now, the entire ASF would be around 7,500 source objects, so 100 is a
very low limit, granted. If the Commons project has an actual interest
in using Kibble for reports and what not, we can get them added - the
thing to remember here is merely that we'd favor getting as much bang
for our buck as possible.

I did some quick checks of available resources, and we're not using a
whole bunch yet, so we can probably aim for 200 source objects for now,
and still be fine.

With regards,
Daniel.

> 
> On Sat, Dec 2, 2017 at 10:30 PM, Gilles Sadowski <er...@apache.org> wrote:
> 
>> Hi.
>>
>> On 2017-12-02 21:27, Rafael Weingärtner <rafaelweingart...@gmail.com>
>> wrote:
>>> How should I handle adding apache commons?
>>>
>>> If I add all of the sources we will go over the limit of 100 sources. In
>>> Github there are 72 projects, and on Jira there other 70+ projects.
>>
>> A lot of projects, yes; but most of them are quite small, and many
>> have very low activity.
>> Would that mitigate usage resources?
>> If the number above is a hard limit, I'll ask the ML for a priority list...
>>
>>> At
>>> least the mailing list seems to be only the ones found here:
>>> https://lists.apache.org/list.html?u...@commons.apache.org
>>
>> The "dev" ML should rather be selected.
>>
>> Gilles
>>
>>>
>>> On Fri, Dec 1, 2017 at 11:44 AM, Gilles Sadowski <er...@apache.org>
>> wrote:
>>>
>>>> Hello.
>>>>
>>>> On 2017-11-27 19:26, Daniel Gruno <humbed...@apache.org> wrote:
>>>>> [...]
>>>>>
>>>>> PS: Please note, we have limited capacity for these tests. We cannot
>>>>> have every single ASF project in the demo, and we reserve the rights
>> to
>>>>> pick the projects that can participate, should we get a lot of
>> requests.
>>>>>
>>>>
>>>> Would you consider adding the "Commons"[1] projects?
>>>>
>>>> Regards,
>>>> Gilles
>>>>
>>>> [1] http://commons.apache.org/
>>>>
>>>
>>>
>>>
>>> --
>>> Rafael Weingärtner
>>>
>>
> 
> 
> 

Re: Hello from Apache Streams

[Daniel Gruno]

On 12/03/2017 01:02 PM, Daniel Gruno wrote:
> On 12/02/2017 10:41 PM, Steve Blackmon wrote:
>>  Sorry about that!  Here’s a link to the notebook that doesn’t require
>> registration.
>>
>> https://www.zepl.com/viewer/notebooks/bm90ZTovL3N0ZXZlYmxhY2ttb24vYXBhY2hlLXplcHBlbGluLWRhc2hib2FyZC84YjQ5YmY3MWIxYTU0ZTE2YjlkMDQyMTliMzNlMjQzYS9ub3RlLmpzb24
>>
>> In this notebook we used the %spark interpreter to collect the data, but
>> most of the work is done as scala in the driver process.  The streams code
>> base is java and not dependent on spark or other frameworks external to the
>> jar file.
>>
>> The easiest integration I can think of given the python/java language gap
>> would use docker - Streams could prepare a docker container packaged with
>> all the necessary code, and Kibble installations could use it to run ad-hoc
>> or scheduled data processes.  The data collected could be written as
>> new-line delimited json on container mounted volumes,  or directly to an
>> elasticsearch index.
>>
>> Docker’s not really necessary though, if the system where Kibble’s running
>> has a JRE configured and a streams distribution local that could work too.
> 
> Right, but probably the easiest entry point for people just "wanting to
> get things done" :). I could also imagine us setting up a remote service
> that could handle this via HTTP API as an alternate solution, akin to
> how you would use a GitHub API - that is to say, we'd have a VM that you
> could query and it'd have all the Java in place for speedy access to
> these sort of things. Either or both would work for me, and if streams
> is willing to sort out the actual data gathering, we could have this put
> into ES quickly and get started on using the data gathered.
> 
> I'll have to ponder how we're going to present this, and which charts
> would be most informative here. There is a lot of potential here.
> 
> If Streams can provide us with a "run this" sort of container that can
> spit out JSON, that would be awesome. While ES directly might be easier,
> there's the use-case scenario where ES is not local to the system
> (Kibble is intended to support both local ES and remote-via-json-api
> systems), so a JSON output might be the best for now.
> 
> With regards,
> Daniel.

A request: Could we get this JSON output as a single document per
repost/like? That is to say, every time jane doe does a retweet etc of
one of our tweets, that should be one document with the various data
fields. This would allow for some interesting mappings instead of just
bar charts :)

> 
>>
>> Steve
>>
>> On Dec 2, 2017 at 2:10 PM, Daniel Gruno <humbed...@apache.org> wrote:
>>
>>
>> On 12/02/2017 09:07 PM, Steve Blackmon wrote:
>>
>> Hi Kibble Team,
>>
>> I've been checking out the code and the demo site this weekend.
>>
>> I'm interested in joining the team and integrating some of the data
>> sources maintained in http://streams.apache.org
>>
>> Specifically, activity streams from the social media presences of
>> projects and contributors (who opt in) as well as statistics derived
>> from them could make a nice addition to Kibble.
>>
>> Here's an example: analysis of Twitter accounts of Apache project
>> using Streams and Zeppelin:
>> https://www.zepl.com/UvGWgAZb7/spaces/Sb9ElZuDD/8b49bf71b1a54e16b9d04219b33e243a
>>
>> Cheers,
>>
>> Steve Blackmon
>> sblack...@apache.org
>>
>>
>> Hi Steve,
>> I like the idea, but I am unable to see the link you shared, it shows a
>> 404 for me :(. Having said that, looking into the social media space is
>> definitely something worth doing!
>>
>> With regards,
>> Daniel.
>>
> 

Re: Hello from Apache Streams

[Daniel Gruno]

On 12/02/2017 10:41 PM, Steve Blackmon wrote:
>  Sorry about that!  Here’s a link to the notebook that doesn’t require
> registration.
> 
> https://www.zepl.com/viewer/notebooks/bm90ZTovL3N0ZXZlYmxhY2ttb24vYXBhY2hlLXplcHBlbGluLWRhc2hib2FyZC84YjQ5YmY3MWIxYTU0ZTE2YjlkMDQyMTliMzNlMjQzYS9ub3RlLmpzb24
> 
> In this notebook we used the %spark interpreter to collect the data, but
> most of the work is done as scala in the driver process.  The streams code
> base is java and not dependent on spark or other frameworks external to the
> jar file.
> 
> The easiest integration I can think of given the python/java language gap
> would use docker - Streams could prepare a docker container packaged with
> all the necessary code, and Kibble installations could use it to run ad-hoc
> or scheduled data processes.  The data collected could be written as
> new-line delimited json on container mounted volumes,  or directly to an
> elasticsearch index.
> 
> Docker’s not really necessary though, if the system where Kibble’s running
> has a JRE configured and a streams distribution local that could work too.

Right, but probably the easiest entry point for people just "wanting to
get things done" :). I could also imagine us setting up a remote service
that could handle this via HTTP API as an alternate solution, akin to
how you would use a GitHub API - that is to say, we'd have a VM that you
could query and it'd have all the Java in place for speedy access to
these sort of things. Either or both would work for me, and if streams
is willing to sort out the actual data gathering, we could have this put
into ES quickly and get started on using the data gathered.

I'll have to ponder how we're going to present this, and which charts
would be most informative here. There is a lot of potential here.

If Streams can provide us with a "run this" sort of container that can
spit out JSON, that would be awesome. While ES directly might be easier,
there's the use-case scenario where ES is not local to the system
(Kibble is intended to support both local ES and remote-via-json-api
systems), so a JSON output might be the best for now.

With regards,
Daniel.

> 
> Steve
> 
> On Dec 2, 2017 at 2:10 PM, Daniel Gruno <humbed...@apache.org> wrote:
> 
> 
> On 12/02/2017 09:07 PM, Steve Blackmon wrote:
> 
> Hi Kibble Team,
> 
> I've been checking out the code and the demo site this weekend.
> 
> I'm interested in joining the team and integrating some of the data
> sources maintained in http://streams.apache.org
> 
> Specifically, activity streams from the social media presences of
> projects and contributors (who opt in) as well as statistics derived
> from them could make a nice addition to Kibble.
> 
> Here's an example: analysis of Twitter accounts of Apache project
> using Streams and Zeppelin:
> https://www.zepl.com/UvGWgAZb7/spaces/Sb9ElZuDD/8b49bf71b1a54e16b9d04219b33e243a
> 
> Cheers,
> 
> Steve Blackmon
> sblack...@apache.org
> 
> 
> Hi Steve,
> I like the idea, but I am unable to see the link you shared, it shows a
> 404 for me :(. Having said that, looking into the social media space is
> definitely something worth doing!
> 
> With regards,
> Daniel.
> 

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

I'll also note, that if people create their own accounts, please let us
know at the kibble project, so we can add you to the demo organisation
on the demo - otherwise you won't be able to see much :)

On 12/02/2017 12:24 PM, Daniel Gruno wrote:
> On 12/02/2017 01:33 AM, Denis Magda wrote:
>> + Ignite dev
>>
>> Rafael,
>>
>> That’s a really useful project. I would use it for Ignite metrics analysis.
>>
>> How do you measure the pony-factor (contributors/committers breakdown per 
>> specific organization)?
> 
> You can read an article on the original pony factor math at David
> Nalley's blog; https://ke4qqq.wordpress.com/2015/02/08/pony-factor-math/
> 
>>
>> Is there a way to see user/dev list specific metrics such as most active 
>> participants (senders and responders), number of signs-up, etc.?
> 
> Yes and no, but mostly yes :)
> There are three mail-related pages you can look at (please log in before
> you proceed to the URLs below):
> 
> - The first is the basic email metrics (Data Points -> Mailing Lists),
> such as this URL for ignite:
> https://demo.kibble.apache.org/dashboard.html?page=mail=ignite
> You can zoom in on a specific user/dev list by simply changing the
> sub-filter to dev@ignite or user@ignite and clicking the green
> 'sub-filter' button.
> 
> - The second is the mail mapping, which shows you who is discussing with
> whom, at Relationships -> Discussion author maps. An example of this is:
> https://demo.kibble.apache.org/relationships.html?page=mail-map=dev@ignite
> for dev@ignite over the past (default) 6 months. It's advised that you
> zoom in on the timespan to the last month or so, to not get as cluttered
> a view.
> 
> - Finally, while Kibble doesn't have access to who subscribes or
> unsubscribes (that's infra only!), it can extrapolate activity based off
> when people send email, and approximate how many people are currently
> active as mail authors. This retention/activity chart is found under
> Engagement -> Contributor Retention, such as:
> https://demo.kibble.apache.org/engagement.html?page=retention=ignite
> 
> While we also have the fun and interesting mood mapping, it's not
> currently been done for ignite, as that costs us a bit of money each
> time we do that scan.
> 
> I hope you find something worth looking at :)
> 
> With regards,
> Daniel.
> 
>>
>> Igniters, please log in and provide your feedback. The details are below.
>>
>> —
>> Denis
>>
>>> On Nov 30, 2017, at 4:30 PM, Rafael Weingärtner 
>>> <rafaelweingart...@gmail.com> wrote:
>>>
>>> https://demo.kibble.apache.org/dashboard.html?page=repos=true=d58d1803c1337bc2762059eb4d6bd72cca53c4a3c981dcf48d1f7412
>>>  
>>> <https://demo.kibble.apache.org/dashboard.html?page=repos=true=d58d1803c1337bc2762059eb4d6bd72cca53c4a3c981dcf48d1f7412>
>>>
>>> You need to log in as guest. The username and password are presented in the 
>>> login interface.
>>>
>>> On Thu, Nov 30, 2017 at 9:53 PM, Denis Magda <dma...@apache.org 
>>> <mailto:dma...@apache.org>> wrote:
>>> Rafael,
>>>
>>> Where should I go to see Ignite metrics? Tried to look for data here but 
>>> this seems to be a wrong place:
>>> https://demo.kibble.apache.org/ <https://demo.kibble.apache.org/>
>>>
>>> —
>>> Denis
>>>
>>>> On Nov 30, 2017, at 11:43 AM, Rafael Weingärtner 
>>>> <rafaelweingart...@gmail.com <mailto:rafaelweingart...@gmail.com>> wrote:
>>>>
>>>> sure.
>>>> You can already see Ignite status there ;)
>>>>
>>>> On Thu, Nov 30, 2017 at 5:41 PM, Denis Magda <dma...@apache.org 
>>>> <mailto:dma...@apache.org>> wrote:
>>>> Hi Rafael,
>>>>
>>>> Confirming that the information is correct. Please CC me in all the 
>>>> related discussions directly.
>>>>
>>>> —
>>>> Denis
>>>>
>>>>> On Nov 30, 2017, at 5:30 AM, Rafael Weingärtner 
>>>>> <rafaelweingart...@gmail.com <mailto:rafaelweingart...@gmail.com>> wrote:
>>>>>
>>>>>
>>>>> -- Forwarded message --
>>>>> From: Rafael Weingärtner <rafaelweingart...@gmail.com 
>>>>> <mailto:rafaelweingart...@gmail.com>>
>>>>> Date: Thu, Nov 30, 2017 at 11:29 AM
>>>>> Subject: Re: Request for Participation: The Right Metrics for the Right 
>>>>> Project
>>>>> To: dev@kibble

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

On 12/02/2017 01:33 AM, Denis Magda wrote:
> + Ignite dev
> 
> Rafael,
> 
> That’s a really useful project. I would use it for Ignite metrics analysis.
> 
> How do you measure the pony-factor (contributors/committers breakdown per 
> specific organization)?

You can read an article on the original pony factor math at David
Nalley's blog; https://ke4qqq.wordpress.com/2015/02/08/pony-factor-math/

> 
> Is there a way to see user/dev list specific metrics such as most active 
> participants (senders and responders), number of signs-up, etc.?

Yes and no, but mostly yes :)
There are three mail-related pages you can look at (please log in before
you proceed to the URLs below):

- The first is the basic email metrics (Data Points -> Mailing Lists),
such as this URL for ignite:
https://demo.kibble.apache.org/dashboard.html?page=mail=ignite
You can zoom in on a specific user/dev list by simply changing the
sub-filter to dev@ignite or user@ignite and clicking the green
'sub-filter' button.

- The second is the mail mapping, which shows you who is discussing with
whom, at Relationships -> Discussion author maps. An example of this is:
https://demo.kibble.apache.org/relationships.html?page=mail-map=dev@ignite
for dev@ignite over the past (default) 6 months. It's advised that you
zoom in on the timespan to the last month or so, to not get as cluttered
a view.

- Finally, while Kibble doesn't have access to who subscribes or
unsubscribes (that's infra only!), it can extrapolate activity based off
when people send email, and approximate how many people are currently
active as mail authors. This retention/activity chart is found under
Engagement -> Contributor Retention, such as:
https://demo.kibble.apache.org/engagement.html?page=retention=ignite

While we also have the fun and interesting mood mapping, it's not
currently been done for ignite, as that costs us a bit of money each
time we do that scan.

I hope you find something worth looking at :)

With regards,
Daniel.

> 
> Igniters, please log in and provide your feedback. The details are below.
> 
> —
> Denis
> 
>> On Nov 30, 2017, at 4:30 PM, Rafael Weingärtner 
>> <rafaelweingart...@gmail.com> wrote:
>>
>> https://demo.kibble.apache.org/dashboard.html?page=repos=true=d58d1803c1337bc2762059eb4d6bd72cca53c4a3c981dcf48d1f7412
>>  
>> <https://demo.kibble.apache.org/dashboard.html?page=repos=true=d58d1803c1337bc2762059eb4d6bd72cca53c4a3c981dcf48d1f7412>
>>
>> You need to log in as guest. The username and password are presented in the 
>> login interface.
>>
>> On Thu, Nov 30, 2017 at 9:53 PM, Denis Magda <dma...@apache.org 
>> <mailto:dma...@apache.org>> wrote:
>> Rafael,
>>
>> Where should I go to see Ignite metrics? Tried to look for data here but 
>> this seems to be a wrong place:
>> https://demo.kibble.apache.org/ <https://demo.kibble.apache.org/>
>>
>> —
>> Denis
>>
>>> On Nov 30, 2017, at 11:43 AM, Rafael Weingärtner 
>>> <rafaelweingart...@gmail.com <mailto:rafaelweingart...@gmail.com>> wrote:
>>>
>>> sure.
>>> You can already see Ignite status there ;)
>>>
>>> On Thu, Nov 30, 2017 at 5:41 PM, Denis Magda <dma...@apache.org 
>>> <mailto:dma...@apache.org>> wrote:
>>> Hi Rafael,
>>>
>>> Confirming that the information is correct. Please CC me in all the related 
>>> discussions directly.
>>>
>>> —
>>> Denis
>>>
>>>> On Nov 30, 2017, at 5:30 AM, Rafael Weingärtner 
>>>> <rafaelweingart...@gmail.com <mailto:rafaelweingart...@gmail.com>> wrote:
>>>>
>>>>
>>>> -- Forwarded message --
>>>> From: Rafael Weingärtner <rafaelweingart...@gmail.com 
>>>> <mailto:rafaelweingart...@gmail.com>>
>>>> Date: Thu, Nov 30, 2017 at 11:29 AM
>>>> Subject: Re: Request for Participation: The Right Metrics for the Right 
>>>> Project
>>>> To: dev@kibble.apache.org <mailto:dev@kibble.apache.org>
>>>>
>>>>
>>>> Hi Denis, 
>>>> Can you confirm the Apache Ignite information? so I can add it
>>>>
>>>> GitHub repo(s):
>>>> https://github.com/apache/ignite.git <https://github.com/apache/ignite.git>
>>>>
>>>> Jira:
>>>> https://issues.apache.org/jira/browse/IGNITE 
>>>> <https://issues.apache.org/jira/browse/IGNITE>
>>>>
>>>> Mailing lists:
>>>> https://lists.apache.org/list.html?u...@ignite.apache.org 
>>>> <https://lists.apache.org/list.html?u..

Re: Add Apache Pivot to the mix?

[Daniel Gruno]

On 11/30/2017 11:03 PM, Rafael Weingärtner wrote:
> Good question.The job has not started yet. I do not know the periodicity
> between data gatherings.
> 
> Maybe Daniel can answer this one for you?

I certainly can, and the answer is "when someone gets around to it" :).
Currently, we are not scheduling the scans, as we want to assess the
impact of adding new projects, so scans are run manually and timed. I'll
run a new scan momentarily with pivot and ofbiz added, and see what
we're at. I expect this to become automated within a few days.

With regards,
Daniel.


> 
> On Thu, Nov 30, 2017 at 8:00 PM, Roger Whitcomb  > wrote:
> 
> Quick question:  I now see the Pivot .git and everything (mailing
> lists, JIRA, etc.) in the lists of choices, but no activity is
> showing up in the graphs yet.  How long does it take for the demo
> instance to digest everything?
> 
> Thanks,
> ~Roger
> 
> 
> 
> On 11/30/17 1:25 PM, Roger and Beth Whitcomb wrote:
> 
> Looks good!  Thank you, Rafael.
> 
> 
> On 11/30/17 1:23 PM, Rafael Weingärtner wrote:
> 
> Hello Roger,
> I will add the Apache Pivot.
> 
> Github:
> https://github.com/apache/pivot.git
> 
> 
> Mailing lists:
> https://lists.apache.org/list.html?u...@pivot.apache.org
> 
> https://lists.apache.org/list.html?d...@pivot.apache.org
> 
> 
> Jira:
> https://issues.apache.org/jira/browse/PIVOT
> 
> 
> If I missed something, please do tell me.
> 
> On Thu, Nov 30, 2017 at 7:08 PM, Roger and Beth Whitcomb <
> rogerandb...@rbwhitcomb.com
> > wrote:
> 
> Hi all,
> 
>  I'm the PMC chair for Apache Pivot, and we'd like
> to add Pivot to the
> demo instance, if there is still room.
> 
> Regards,
> ~Roger Whitcomb
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> -- 
> Rafael Weingärtner

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

On 11/30/2017 02:48 PM, Rafael Weingärtner wrote:
> Regarding the addition of ComDev to Kibble experiment,
> I found ComDev Jira page: https://issues.apache.org/jira/browse/COMDEV
> 
> The mailing lists are:
> https://lists.apache.org/list.html?d...@community.apache.org
> https://lists.apache.org/list.html?stude...@community.apache.org
> 
> Does ComDev have a repository for some templates or something else
> somewhere?
> I did not find anything on Github, is it on Apache's SVN only?
> 

I don't think comdev has a lot of publicly accessible data we can use
yet, code-wise. there's the svn repo, but there is no git clone of that
yet (and scanning subversion is a regular PITA, sadly). so perhaps we'll
have to suffice with the mail and jira for now.

> 
> On Thu, Nov 30, 2017 at 6:50 AM, Sharan Foga <sha...@apache.org> wrote:
> 
>> Hi
>>
>> I'd be keen for ComDev to participate too.
>>
>> ComDev is a bit different from other projects and I'd be interested in
>> things like identifying new contributors, seeing how people move on to
>> participate in other ASF projects (and vice versa where people established
>> in other projects join ComDev) and also the relationships ComDev has with
>> other ASF projects.
>>
>> Thanks
>> Sharan
>>
>> On 2017-11-27 19:26, Daniel Gruno <humbed...@apache.org> wrote:
>>> Hi there, fellow Apache projects!
>>>
>>> The Apache Kibble project serves as a practical implementation of
>>> metrics deemed to be helpful for open source projects trying to
>>> understand where their project is, was, and is headed.
>>>
>>> As such, we need help in determining which metrics projects either
>>> already use and consider useful for measuring project health or which
>>> metrics they would love to have and use.
>>>
>>> We are looking for projects interested in participating in the Kibble
>>> demo instance ( https://demo.kibble.apache.org/ ) and sending feedback
>>> to the Kibble project on which parts they find useful, which elements
>>> they find useless and which ideas they would love to see implemented to
>>> better gauge the health and activity of their project.
>>>
>>> Initially we are looking for Apache projects to help out, but we will
>>> later on expand this to other open source organizations and projects.
>>>
>>> Projects that participate will be added to the demo instance and scanned
>>> on a regular basis so the data can be used for reports and analysis.
>>> The Kibble PMC will ensure that the correct sources are added, but you
>>> are of course welcome to help identify which parts need analyzing.
>>>
>>> How to participate:
>>>
>>> - Join the dev@kibble.apache.org mailing list and let us know if your
>>> project is interested in joining the demo (a few projects were added in
>>> advance so you can actually test it). You can also join us on HipChat or
>>> in #kibble on Freenode IRC (IRC and HipChat are bridged).
>>>
>>> - Try out the demo, and send us feedback to the mailing list on what you
>>> like, dislike and would love to see added.
>>>
>>> - In particular: Which metrics do you look for when reviewing the code,
>>> development and community health/trends of your project - which do you
>>> have, which would you love to see added?
>>>
>>> With regards,
>>> Daniel on behalf of the Apache Kibble project.
>>>
>>> PS: Please note, we have limited capacity for these tests. We cannot
>>> have every single ASF project in the demo, and we reserve the rights to
>>> pick the projects that can participate, should we get a lot of requests.
>>>
>>
> 
> 
> 

Status of new projects in the demo

[Daniel Gruno]

Hi folks,
Thanks to Rafael Weingärtner, we'll get the current requests added, and
phoenix + cloudstack are already in the database.

So, the kibble demo will consist of:
- beam
- clerezza
- cloudstack
- fineract
- hadoop
- hbase
- httpd
- jena
- ignite
- kibble
- kudu
- netbeans
- phoenix
- samza
- stanbol
- subversion
- trafficserver

This will probably leave us at around 75 sources in the DB, leaving some
25 left for grabs if other projects want to join.

Re: Request for Participation: The Right Metrics for the Right Project

[Daniel Gruno]

On 11/27/2017 08:15 PM, Rafael Weingärtner wrote:
> What about testing this on Apache CloudStack (ACS) folks?
> 
> Kibble fellows, what would you require from us to start analyzing ACS code
> base?

See my previous email on dev@kibble.

> 
> Are the dashboards public visible? Is it possible to integrate with Sonar,
> Coverity or other code analysis tool, so we can gather information
> regarding technical debt and other code quality related metrics to plot?

In theory, anything is possible.
We don't currently have scanners for coverity or sonar, but if someone
would either work on that or provide us with the API and access to
someplace with the data, we could likely make a scanner and charts for
this quite easily. I imagine this would be useful for many projects.

> 
> On Mon, Nov 27, 2017 at 4:26 PM, Daniel Gruno <humbed...@apache.org> wrote:
> 
>> Hi there, fellow Apache projects!
>>
>> The Apache Kibble project serves as a practical implementation of
>> metrics deemed to be helpful for open source projects trying to
>> understand where their project is, was, and is headed.
>>
>> As such, we need help in determining which metrics projects either
>> already use and consider useful for measuring project health or which
>> metrics they would love to have and use.
>>
>> We are looking for projects interested in participating in the Kibble
>> demo instance ( https://demo.kibble.apache.org/ ) and sending feedback
>> to the Kibble project on which parts they find useful, which elements
>> they find useless and which ideas they would love to see implemented to
>> better gauge the health and activity of their project.
>>
>> Initially we are looking for Apache projects to help out, but we will
>> later on expand this to other open source organizations and projects.
>>
>> Projects that participate will be added to the demo instance and scanned
>> on a regular basis so the data can be used for reports and analysis.
>> The Kibble PMC will ensure that the correct sources are added, but you
>> are of course welcome to help identify which parts need analyzing.
>>
>> How to participate:
>>
>> - Join the dev@kibble.apache.org mailing list and let us know if your
>> project is interested in joining the demo (a few projects were added in
>> advance so you can actually test it). You can also join us on HipChat or
>> in #kibble on Freenode IRC (IRC and HipChat are bridged).
>>
>> - Try out the demo, and send us feedback to the mailing list on what you
>> like, dislike and would love to see added.
>>
>> - In particular: Which metrics do you look for when reviewing the code,
>> development and community health/trends of your project - which do you
>> have, which would you love to see added?
>>
>> With regards,
>> Daniel on behalf of the Apache Kibble project.
>>
>> PS: Please note, we have limited capacity for these tests. We cannot
>> have every single ASF project in the demo, and we reserve the rights to
>> pick the projects that can participate, should we get a lot of requests.
>>
> 
> 
> 

Re: Mood mapping experiment

[Daniel Gruno]

On 10/24/2017 01:56 PM, Sharan Foga wrote:
> Hi
> 
> I've found a nice quiet-esh corner at the summit in Prague and am feeling
> pretty good after an exciting morning of interesting  talks :-D
> 
> I hear there is a great talk on mentoring coming up this afternoon from
> someone called Bowen so am really looking forward to that.
> 
> Thanks
> Sharan

That's the spirit! :)
Unsurprisingly, these emails are registering as happy emails on the box,
which should bump things a bit :). It's always fun and exciting when
things do what you intended, yay :)

With regards,
Daniel.

> 
> On 24 Oct 2017 1:48 pm, "Daniel Gruno" <humbed...@apache.org> wrote:
> 
>> On 10/23/2017 07:44 PM, Daniel Gruno wrote:
>>> Hi folks,
>>> I'm trying out an experiment with mapping moods across mailing lists
>>> using the IBM Watson/BlueMix tone analyzer.
>>>
>>> The results are on display at:
>>> https://demo.kibble.apache.org/dashboard.html?page=mail-mood
>>>
>>> Currently just some bar charts, which I'd like to extend to a timeseries
>>> if possible. Only a few hundred email (the latest ones) have been
>>> processed, as this is a paid service, but I find the results quite
>>> interesting :) Especially the paradigm shifts from user/dev and old/new
>>> projects.
>>>
>>> With regards,
>>> Daniel.
>>>
>>
>> Okay, so this seems to be working.
>> In order to properly test it, I'm gonna be positive in this email and
>> say this is great. I love this email, I'm so happy and full of joy! :)
>>
>> Go Kibble, yay kibble!
>>
>> (PS: If you can muster the energy, reply with some happy thoughts so we
>> can see whether it registers!)
>>
>> With regards,
>> Daniel.
>>
> 

Re: What do we want from Kibble?

[Daniel Gruno]

On 10/21/2017 06:11 PM, Rich Bowen wrote:
> I think it's important that we are neutral in terms of saying that a
> particular metric is important, useful, necessary, so on. That is, while it
> may be obvious to us that a more corporately-diverse developer pool is a
> good thing, that is an opinion based in dogma, not in science. What's a
> "good" value for a particular metric is a matter of philosophy rather than
> science. What constitutes a healthy community is, also.
> 
> The software should be awesome at collecting data and displaying trends.
> The specific community in question is responsible for interpreting what
> they mean, in the context of their own community, and what they want to do
> about it.
> 
> The question of "here's what you should do to make your community more
> healthy" is amazingly complicated, and while that may be a goal some day,
> it's in a much later version.
> 
> This also implies that we should be asking projects (LOTS of them) what
> metrics/trends they wish they had a tool to track, and provide those tools.
> We should also be asking them what correlations they want to see and add
> those tools. Things like "when I make a release I get more downloads" or
> "when I add N new committers my tickets get closed slower/faster" or
> whatever. We don't know what they want to know, and if we assume that we
> do, we'll be missing an opportunity.

Big +1 to this.
I think Hervé was more focused on the practicalities here, and I think
both aspects are important. We both want something that works and is
tangible and accessible, but we also want the more qualitative and
anecdotal information out there, not just cold numbers.

I know you have a hectic schedule, but it would be awesome if we could
come up with some emails to send to the ASF projects for starters, and
gauge what they feel are good metrics to keep an eye on as a community -
maybe even get some "behind the scenes" commentary on how certain
metrics have correlated to ups/downs of various aspects of the projects.
Not so we can say "your project is doing well or bad", but so we can
provide information on certain metrics and leave it to projects to act
based on metrics, information about health/diversity and historical
correlations (aka make an informed opinion).

With regards,
Daniel.

> 
> 
> On Sat, Oct 21, 2017 at 12:57 PM, Hervé BOUTEMY <herve.bout...@free.fr>
> wrote:
> 
>> to me, ideally, Kibble 1.0 would be when it has the features required to
>> replace Snoot in Apache Projects Statistics [1] (Snoot service could use
>> Kibble 1.0 as its code)
>>
>> Looking at the "Data Points" page in Kibble demo [2], it seems we're not so
>> far: release early, release often, adding features not available in Snoot
>> for
>> projects.a.o would be for next versions
>>
>> Regards,
>>
>> Hervé
>>
>> [1] https://projects.apache.org/statistics.html
>>
>> [2] https://demo.kibble.apache.org/dashboard.html?page=repos
>>
>> Le vendredi 20 octobre 2017, 16:17:47 CEST Daniel Gruno a écrit :
>>> I'd like to kick off a larger discussion around what we hope Kibble can
>>> achieve, and how this will come about.
>>>
>>> For starters, what sort of data should we collect and display, what
>>> types of visualizations should we offer, and are there special formulas
>>> or algorithms (like Pony Factor) that we'd like to see. Which internal
>>> features should we be using (such as account linking/collating or
>>> collapsable groups of repos based on regex etc)?
>>>
>>> Then comes the bigger points: At what point would we consider Kibble
>>> good enough for a release? What things MUST we have before we can go out
>>> and say "hey, we've got this amazing tool, check it out!"?
>>>
>>> On a similar note, I (and probably Rich too) will be reaching out to the
>>> CHAOSS project over at LF ( http://chaoss.community/ ) to see if we can
>>> work out some specifications and standards with them, for use in Kibble.
>>>
>>> With regards,
>>> Daniel.
>>
>>
>>
> 

What do we want from Kibble?

[Daniel Gruno]

I'd like to kick off a larger discussion around what we hope Kibble can
achieve, and how this will come about.

For starters, what sort of data should we collect and display, what
types of visualizations should we offer, and are there special formulas
or algorithms (like Pony Factor) that we'd like to see. Which internal
features should we be using (such as account linking/collating or
collapsable groups of repos based on regex etc)?

Then comes the bigger points: At what point would we consider Kibble
good enough for a release? What things MUST we have before we can go out
and say "hey, we've got this amazing tool, check it out!"?

On a similar note, I (and probably Rich too) will be reaching out to the
CHAOSS project over at LF ( http://chaoss.community/ ) to see if we can
work out some specifications and standards with them, for use in Kibble.

With regards,
Daniel.

Re: Ping

[Daniel Gruno]

On 10/19/2017 01:01 PM, Christofer Dutz wrote:
> Hi guys
> 
> So, I just subscribed to 1000 new lists and the “dingongs” of the incoming 
> confirmation mails are making my colleagues throw things at me :)
> 
> But now I should be set.
> 
> Chris
> 

Welcome! I'm gonna wait a little while for people to get settled in, and
then flood the ML with ideas and proposals ;)

There's a lot to discuss and review! :)

With regards,
Daniel.