Re: Standards for mail archive statistics gathering?
Le mercredi 6 mai 2015 12:48:34 Steve Blackmon a écrit : > > For visualization, for sure, json is the current natural format when data > > is consumed from the browser. > > I don't have great experience on this, and what I'm missing with json > > currently is a common practice on documenting a structure: are there > > common > > practices? > > In podling streams [0], we make extensive use of json schema [1] thank you: that's exactly the initial info I was looking for: json schema! > from > which we generate POJOs with a maven > plugin jsonschema2pojo [2] which makes manipulating the objects in > Java/Scala pleasant. I expect other languages have > similar jsonschema-based ORM paradigms as well. As usual Java devloper, your tooling is interesting But in the projects-new.a.o case, it is data extraction is coded in Python: if we create json schema, having Python classes generated could simplify coding. Anyone with Python+json schema experience around? > This pattern supports > inheritance both within > and across projects - for example see how [3] extends [4] which > extends [5]. These schemas are relatively self documenting, > but generating documentation or other artifacts is straight-forward as > they are themselves json documents. yeah, json schema document is easy to read (at least the examples on the site...) > > > Because for simple json structure, documentation is not really necessary, > > but once the structure goes complex, documentation is really a key > > requirement for people to use or extend. And I already see this > > shortcoming with the 11 json files from projects-new.a.o = > > https://projects-new.apache.org/json/foundation/ > Having used these json documents a few weeks ago to build an apache > community visualization [6] yeah, really nice visualization! > IMO the current crop of project-new jsons > are intermediate artifacts rather than a sufficiently cross-purpose > data model, a role currently held by DOAP mbox and misc others all > with some inherent shortcomings most notably lack of navigability > between silos. +1 I'm at a point where I start to really understand the concepts involved and want to code a simple data model: I'll report here once I have a first version available. > I'd like to nominate activity streams [7] with > community-specific extensions (such as those roughly prototyped here: > [8] ) as a potential core data model for this effort going forward I had a first look at it: it is more complex than what I had in mind We'll have to share and see what's the best bet > and > I'm happy to help apply some of the useful tools and connectors within > podling streams toward that end. Converting external structured > sources into normalized documents and indexing those activities to > power data-centric APIs and visualizations are wheelhouse use cases > for this project, as they say. Great, stay tuned: I'll probably work on it this week-end Regards, Hervé > > [0] http://streams.incubator.apache.org/ > [1] http://json-schema.org/documentation.html > [2] http://www.jsonschema2pojo.org/ > [3] > https://github.com/steveblackmon/streams-apache/blob/master/activities/src/ > main/jsonschema/objectTypes/committee.json [4] > https://github.com/apache/incubator-streams/blob/master/streams-pojo/src/ma > in/jsonschema/objectTypes/group.json [5] > https://github.com/apache/incubator-streams/blob/master/streams-pojo/src/ma > in/jsonschema/object.json [6] http://72.182.111.65:3000/workspace/3 > [7] http://activitystrea.ms/ > [8] > https://github.com/steveblackmon/streams-apache/blob/master/activities/src/ > main/jsonschema > > Steve Blackmon > sblack...@apache.org > > On Wed, May 6, 2015 at 2:05 AM, Hervé BOUTEMY wrote: > > Le mardi 5 mai 2015 21:26:36 Shane Curcuru a écrit : > >> On 5/5/15 7:33 AM, Boris Baldassari wrote: > >> > Hi Folks, > >> > > >> > Sorry for the late answer on this thread. Don't know what has been done > >> > since then, but I've some experience to share on this, so here are my > >> > 2c.. > >> > >> No, more input is always appreciated! Hervé is doing some > >> centralization of the projects-new.a.o data capture, which is related > >> but slightly separate. > > > > +1 > > this can give a common place to put code once experiments show that we > > should add a new data source > > > >> But this is going to be a long-term project > > > > +1 > > > >> with > >> plenty of different people helping I bet. > > > > I hope so... > > > >> ... > >> > >> > * Parsing mboxes for software repository data mining: > >> > There is a suite of tools exactly targeted at this kind of duty on > >> > github: Metrics Grimoire [1], developed (and used) by Bitergia [2]. I > >> > don't know how they manage time zones, but the toolsuite is widely used > >> > around (see [3] or [4] as examples) so I believe they are quite robust. > >> > It includes tools for data retrieval as well as visualisation. > >> > >> Drat. Metrics Grimoire looks pretty nifty - essentially a set o
Re: What might an open source class look like?
Daniel, great initiative and I think it will become popular over time (new ideas takes a while). 1) I think you should bring up the difference between "Open Source" a la MySQL, i.e development at a corporation and releases thrown over the wall, versus "Open Collaboration" a la Apache, where everything is expected to be happening in the open, asynchronously and relatively slow pace. There is also a third model, which is the "Lone Wolf GitHubber" who does it in the open, maybe even get a lot of Pull Requests, but doesn't expand into a developer's community. 2) In the "Open Collaboration" model, there is then the need for some type of Governance, which varies from "all volunteers" to "paid membership" to "commercial invite only" or combination of all. I recall people mentioning studies that shows that the dynamics changes dramatically as soon as "paid for" _anything_ is introduced, where _anything" might be membership, influence, developer time, evangelism and so on. 3) Adoption pattern(s). I recall that when Gianugo Rabellino (ASF Member) was a SourceSense, they had an "Adoption Path" on their website which was pretty thorough and something we take for granted, but made a lot of sense to unaware commercial entities, and I think this "road map" of how to move from "commercial-only" to "contributor" or even "project leader" is important, even to students, who might end up spear-heading such changes when they get jobs. Good Luck, and please keep this list in the loop Niclas On Thu, May 7, 2015 at 7:42 AM, Daniel Ruggeri wrote: > Hi, all; >We had an interesting chat during the barcamp at ACNA2015 discussing > ideas for spreading the word about open source. A few folks mentioned > that it would be a good idea to partner with local universities to do > talks/programs/etc. This sounded like an interesting idea so I > squirreled it away in the back of my mind to be revisited after I > settled back into the not-apachecon-routine... Interestingly enough, the > day I got back from ApacheCon, a former professor (and mentor of mine) > had asked if I would be willing to send the head of the IS program a > letter of recommendation to accompany his nomination for an award. I > mentioned the idea of doing something with the university regarding open > source and introducing students to the idea in the P.S. of the email... > Well, one conversation led to another and now I find myself teaching a > credited class about open source in the fall. > >I think this is really neat and exciting but a challenge at the same > time. Since the idea was planted in my head w/ the ASF, I thought it > would be a good idea to float the question here to ask, "What would go > in a college class about open source?" I think I can work through a > syllabus, but I'd love to hear suggestions from those who have been > involved in the ASF longer than my 4-ish years. > > Here are some of the ideas I have in mind for things to cover: > *What IS open source? The history/birth of the movement. > *Source control with Subversion/GIT/? > *Bug tracking > *Mailing lists/IRC/communication tools > *Participating in an open source community > *Lab(s) where we create a repository and commit/work through examples of > using the tools > *Guest speaker: How we make money with Open Source > *Guest speaker: The Apache way (of course!) > *Guest speaker: Why I trust open source software in my production > environment > *Guest speaker: Why NOT open source (?) > *Popular open source licenses - discussion around each > *??? > > I've only been on this list since ApacheCon this year, so I'm not sure > what areas (if any) I would have commit access to in the community > project, but I am more than willing to provide the materials I create as > part of the class for those similarly interested in putting on such a > program. > > P.S. > I'm in the process of mining > http://community.apache.org/speakers/slides.html for additional ideas, > too... > > -- > Daniel Ruggeri > > -- Niclas Hedhman, Software Developer http://zest.apache.org - New Energy for Java
Re: Standards for mail archive statistics gathering?
If you want to unsubscribe, please find instructions at http://apache.org/foundation/mailinglists.html And the name of this list is dev@community.apache.org Cheers Niclas On Thu, May 7, 2015 at 7:48 AM, Betty James wrote: > Oh my gosh. How do I get off this thread. don't know how I got on, but I > am just a totally ignorant individual using Open Office and trying to > donate (which doesn't sound necessary anymore)so unless you are in good > shape and in your 70's try to figure out how I can get off the list! > > Betty B. James > > On Tue, May 5, 2015 at 7:33 AM, Boris Baldassari < > castalia.laborat...@gmail.com> wrote: > > > Hi Folks, > > > > Sorry for the late answer on this thread. Don't know what has been done > > since then, but I've some experience to share on this, so here are my > 2c.. > > > > * Parsing dates and time zones: > > If you are to use Perl, the Date::Parse module handles dates and time > > zones pretty well. As for Python I don't know -- there probably is a > module > > for that too.. > > I used Date::Parse to parse ASF mboxes (notably for Ant and JMeter, the > > data sets have been published here [0]), and it worked great. I do have a > > Perl script to do that, which I can provide -- but I have no access I'm > > aware of in the dev scm, and not sure if Perl is the most common language > > here.. so please let me know. > > > > * Parsing mboxes for software repository data mining: > > There is a suite of tools exactly targeted at this kind of duty on > github: > > Metrics Grimoire [1], developed (and used) by Bitergia [2]. I don't know > > how they manage time zones, but the toolsuite is widely used around (see > > [3] or [4] as examples) so I believe they are quite robust. It includes > > tools for data retrieval as well as visualisation. > > > > * As for the feedback/thoughts about the architecture and formats: > > I love the REST-API idea proposed by Rob. That's really easy to access > and > > retrieve through scripts on-demand. CSV and JSON are my favourite > formats, > > because they are, again, easy to parse and widely used -- every language > > and library has some facility to read them natively. > > > > > > Cheers, > > > > > > [0] http://castalia.solutions/datasets/ > > [1] https://metricsgrimoire.github.io/ > > [2] http://bitergia.com > > [3] Eclipse Dashboard: http://dashboard.eclipse.org/ > > [4] OpenStack Dashboard: http://activity.openstack.org/dash/browser/ > > > > > > > > -- > > Boris Baldassari > > Castalia Solutions -- Elegant Software Engineering > > Web: http://castalia.solutions > > Phone: +33 6 48 03 82 89 > > > > > > Le 28/04/2015 16:11, Rich Bowen a écrit : > > > >> > >> > >> On 04/27/2015 09:36 AM, Shane Curcuru wrote: > >> > >>> I'm interested in working on some visualizations of mailing list > >>> activity over time, in particular some simple analyses, like thread > >>> length/participants and the like. Given that the raw data can all be > >>> precomputed from mbox archives, is there any semi-standard way to > >>> distill and save metadata about mboxes? > >>> > >>> If we had a generic static database of past mail metadata and > statistics > >>> (i.e. not details of contents, but perhaps overall # of lines of text > or > >>> something), it would be interesting to see what kinds of visualizations > >>> that different people would come up with. > >>> > >>> Anyone have pointers to either a data format or the best parsing > library > >>> for this? I'm trying to think ahead, and work on the parsing, storing > >>> statistics, and visualizations as separate pieces so it's easier for > >>> different people to collaborate on something. > >>> > >> > >> Roberto posted something to the list a month or so ago about the efforts > >> that he's been working on for this kind of thing. You might ping him. > >> > >> --Rich > >> > >> > >> > > > -- Niclas Hedhman, Software Developer http://zest.apache.org - New Energy for Java
Re: Standards for mail archive statistics gathering?
Oh my gosh. How do I get off this thread. don't know how I got on, but I am just a totally ignorant individual using Open Office and trying to donate (which doesn't sound necessary anymore)so unless you are in good shape and in your 70's try to figure out how I can get off the list! Betty B. James On Tue, May 5, 2015 at 7:33 AM, Boris Baldassari < castalia.laborat...@gmail.com> wrote: > Hi Folks, > > Sorry for the late answer on this thread. Don't know what has been done > since then, but I've some experience to share on this, so here are my 2c.. > > * Parsing dates and time zones: > If you are to use Perl, the Date::Parse module handles dates and time > zones pretty well. As for Python I don't know -- there probably is a module > for that too.. > I used Date::Parse to parse ASF mboxes (notably for Ant and JMeter, the > data sets have been published here [0]), and it worked great. I do have a > Perl script to do that, which I can provide -- but I have no access I'm > aware of in the dev scm, and not sure if Perl is the most common language > here.. so please let me know. > > * Parsing mboxes for software repository data mining: > There is a suite of tools exactly targeted at this kind of duty on github: > Metrics Grimoire [1], developed (and used) by Bitergia [2]. I don't know > how they manage time zones, but the toolsuite is widely used around (see > [3] or [4] as examples) so I believe they are quite robust. It includes > tools for data retrieval as well as visualisation. > > * As for the feedback/thoughts about the architecture and formats: > I love the REST-API idea proposed by Rob. That's really easy to access and > retrieve through scripts on-demand. CSV and JSON are my favourite formats, > because they are, again, easy to parse and widely used -- every language > and library has some facility to read them natively. > > > Cheers, > > > [0] http://castalia.solutions/datasets/ > [1] https://metricsgrimoire.github.io/ > [2] http://bitergia.com > [3] Eclipse Dashboard: http://dashboard.eclipse.org/ > [4] OpenStack Dashboard: http://activity.openstack.org/dash/browser/ > > > > -- > Boris Baldassari > Castalia Solutions -- Elegant Software Engineering > Web: http://castalia.solutions > Phone: +33 6 48 03 82 89 > > > Le 28/04/2015 16:11, Rich Bowen a écrit : > >> >> >> On 04/27/2015 09:36 AM, Shane Curcuru wrote: >> >>> I'm interested in working on some visualizations of mailing list >>> activity over time, in particular some simple analyses, like thread >>> length/participants and the like. Given that the raw data can all be >>> precomputed from mbox archives, is there any semi-standard way to >>> distill and save metadata about mboxes? >>> >>> If we had a generic static database of past mail metadata and statistics >>> (i.e. not details of contents, but perhaps overall # of lines of text or >>> something), it would be interesting to see what kinds of visualizations >>> that different people would come up with. >>> >>> Anyone have pointers to either a data format or the best parsing library >>> for this? I'm trying to think ahead, and work on the parsing, storing >>> statistics, and visualizations as separate pieces so it's easier for >>> different people to collaborate on something. >>> >> >> Roberto posted something to the list a month or so ago about the efforts >> that he's been working on for this kind of thing. You might ping him. >> >> --Rich >> >> >> >
Re: What might an open source class look like?
This sounds fantastic. I’d be happy to collaborate and get this implemented at USC.. ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Daniel Ruggeri Reply-To: "dev@community.apache.org" Date: Wednesday, May 6, 2015 at 1:42 PM To: "dev@community.apache.org" Subject: What might an open source class look like? >Hi, all; > We had an interesting chat during the barcamp at ACNA2015 discussing >ideas for spreading the word about open source. A few folks mentioned >that it would be a good idea to partner with local universities to do >talks/programs/etc. This sounded like an interesting idea so I >squirreled it away in the back of my mind to be revisited after I >settled back into the not-apachecon-routine... Interestingly enough, the >day I got back from ApacheCon, a former professor (and mentor of mine) >had asked if I would be willing to send the head of the IS program a >letter of recommendation to accompany his nomination for an award. I >mentioned the idea of doing something with the university regarding open >source and introducing students to the idea in the P.S. of the email... >Well, one conversation led to another and now I find myself teaching a >credited class about open source in the fall. > > I think this is really neat and exciting but a challenge at the same >time. Since the idea was planted in my head w/ the ASF, I thought it >would be a good idea to float the question here to ask, "What would go >in a college class about open source?" I think I can work through a >syllabus, but I'd love to hear suggestions from those who have been >involved in the ASF longer than my 4-ish years. > >Here are some of the ideas I have in mind for things to cover: >*What IS open source? The history/birth of the movement. >*Source control with Subversion/GIT/? >*Bug tracking >*Mailing lists/IRC/communication tools >*Participating in an open source community >*Lab(s) where we create a repository and commit/work through examples of >using the tools >*Guest speaker: How we make money with Open Source >*Guest speaker: The Apache way (of course!) >*Guest speaker: Why I trust open source software in my production >environment >*Guest speaker: Why NOT open source (?) >*Popular open source licenses - discussion around each >*??? > >I've only been on this list since ApacheCon this year, so I'm not sure >what areas (if any) I would have commit access to in the community >project, but I am more than willing to provide the materials I create as >part of the class for those similarly interested in putting on such a >program. > >P.S. >I'm in the process of mining >http://community.apache.org/speakers/slides.html for additional ideas, >too... > >-- >Daniel Ruggeri >
What might an open source class look like?
Hi, all; We had an interesting chat during the barcamp at ACNA2015 discussing ideas for spreading the word about open source. A few folks mentioned that it would be a good idea to partner with local universities to do talks/programs/etc. This sounded like an interesting idea so I squirreled it away in the back of my mind to be revisited after I settled back into the not-apachecon-routine... Interestingly enough, the day I got back from ApacheCon, a former professor (and mentor of mine) had asked if I would be willing to send the head of the IS program a letter of recommendation to accompany his nomination for an award. I mentioned the idea of doing something with the university regarding open source and introducing students to the idea in the P.S. of the email... Well, one conversation led to another and now I find myself teaching a credited class about open source in the fall. I think this is really neat and exciting but a challenge at the same time. Since the idea was planted in my head w/ the ASF, I thought it would be a good idea to float the question here to ask, "What would go in a college class about open source?" I think I can work through a syllabus, but I'd love to hear suggestions from those who have been involved in the ASF longer than my 4-ish years. Here are some of the ideas I have in mind for things to cover: *What IS open source? The history/birth of the movement. *Source control with Subversion/GIT/? *Bug tracking *Mailing lists/IRC/communication tools *Participating in an open source community *Lab(s) where we create a repository and commit/work through examples of using the tools *Guest speaker: How we make money with Open Source *Guest speaker: The Apache way (of course!) *Guest speaker: Why I trust open source software in my production environment *Guest speaker: Why NOT open source (?) *Popular open source licenses - discussion around each *??? I've only been on this list since ApacheCon this year, so I'm not sure what areas (if any) I would have commit access to in the community project, but I am more than willing to provide the materials I create as part of the class for those similarly interested in putting on such a program. P.S. I'm in the process of mining http://community.apache.org/speakers/slides.html for additional ideas, too... -- Daniel Ruggeri
Re: Standards for mail archive statistics gathering?
> For visualization, for sure, json is the current natural format when data is > consumed from the browser. > I don't have great experience on this, and what I'm missing with json > currently is a common practice on documenting a structure: are there common > practices? In podling streams [0], we make extensive use of json schema [1] from which we generate POJOs with a maven plugin jsonschema2pojo [2] which makes manipulating the objects in Java/Scala pleasant. I expect other languages have similar jsonschema-based ORM paradigms as well. This pattern supports inheritance both within and across projects - for example see how [3] extends [4] which extends [5]. These schemas are relatively self documenting, but generating documentation or other artifacts is straight-forward as they are themselves json documents. > Because for simple json structure, documentation is not really necessary, but > once the structure goes complex, documentation is really a key requirement for > people to use or extend. And I already see this shortcoming with the 11 json > files from projects-new.a.o = https://projects-new.apache.org/json/foundation/ Having used these json documents a few weeks ago to build an apache community visualization [6] IMO the current crop of project-new jsons are intermediate artifacts rather than a sufficiently cross-purpose data model, a role currently held by DOAP mbox and misc others all with some inherent shortcomings most notably lack of navigability between silos. I'd like to nominate activity streams [7] with community-specific extensions (such as those roughly prototyped here: [8] ) as a potential core data model for this effort going forward and I'm happy to help apply some of the useful tools and connectors within podling streams toward that end. Converting external structured sources into normalized documents and indexing those activities to power data-centric APIs and visualizations are wheelhouse use cases for this project, as they say. [0] http://streams.incubator.apache.org/ [1] http://json-schema.org/documentation.html [2] http://www.jsonschema2pojo.org/ [3] https://github.com/steveblackmon/streams-apache/blob/master/activities/src/main/jsonschema/objectTypes/committee.json [4] https://github.com/apache/incubator-streams/blob/master/streams-pojo/src/main/jsonschema/objectTypes/group.json [5] https://github.com/apache/incubator-streams/blob/master/streams-pojo/src/main/jsonschema/object.json [6] http://72.182.111.65:3000/workspace/3 [7] http://activitystrea.ms/ [8] https://github.com/steveblackmon/streams-apache/blob/master/activities/src/main/jsonschema Steve Blackmon sblack...@apache.org On Wed, May 6, 2015 at 2:05 AM, Hervé BOUTEMY wrote: > Le mardi 5 mai 2015 21:26:36 Shane Curcuru a écrit : >> On 5/5/15 7:33 AM, Boris Baldassari wrote: >> > Hi Folks, >> > >> > Sorry for the late answer on this thread. Don't know what has been done >> > since then, but I've some experience to share on this, so here are my 2c.. >> >> No, more input is always appreciated! Hervé is doing some >> centralization of the projects-new.a.o data capture, which is related >> but slightly separate. > +1 > this can give a common place to put code once experiments show that we should > add a new data source > >> But this is going to be a long-term project > +1 > >> with >> plenty of different people helping I bet. > I hope so... > >> >> ... >> >> > * Parsing mboxes for software repository data mining: >> > There is a suite of tools exactly targeted at this kind of duty on >> > github: Metrics Grimoire [1], developed (and used) by Bitergia [2]. I >> > don't know how they manage time zones, but the toolsuite is widely used >> > around (see [3] or [4] as examples) so I believe they are quite robust. >> > It includes tools for data retrieval as well as visualisation. >> >> Drat. Metrics Grimoire looks pretty nifty - essentially a set of >> frameworks for extracting metadata from a bunch of sources - but it's >> GPL, so personally I have no interest in working on it. If someone else >> uses it to generate datasets that's great. >> >> > * As for the feedback/thoughts about the architecture and formats: >> > I love the REST-API idea proposed by Rob. That's really easy to access >> > and retrieve through scripts on-demand. CSV and JSON are my favourite >> > formats, because they are, again, easy to parse and widely used -- every >> > language and library has some facility to read them natively. >> >> Yup - again, like project visualization, to make any of this simple for >> newcomers to try stuff, we need to separate data gathering / model / >> visualization. Since most of these are spare time projects, having easy >> chunks makes it simpler for different people to try their hand at it. > For visualization, for sure, json is the current natural format when data is > consumed from the browser. > I don't have great experience on this, and what I'm missing with json > currently is a common practice on docume
Re: Captcha on Apache mirror
On Tue, May 5, 2015 at 8:49 PM, Niclas Hedhman wrote: > I am still getting it... :-/ I don't get the Captcha either, but Cloudflare may be presenting that to folks in specific geo-regions or they think your IP has triggered anti-DDOS protections. -- justin
Re: Standards for mail archive statistics gathering?
Hi all, Le 06/05/2015 03:26, Shane Curcuru a écrit : Drat. Metrics Grimoire looks pretty nifty - essentially a set of frameworks for extracting metadata from a bunch of sources - but it's GPL, so personally I have no interest in working on it. If someone else uses it to generate datasets that's great. Argh. I had forgotten about the licencing incompatibility, my mistake. Well, as you point out I guess the product can still be used without modifications, if needed. I'm following this ml for a few months now, but I'm not sure how far this project is planned to go. Are the mailing lists the only artefact to be analysed, or do you intend to provide also configuration management or issue tacking data? Yup - again, like project visualization, to make any of this simple for newcomers to try stuff, we need to separate data gathering / model / visualization. Since most of these are spare time projects, having easy chunks makes it simpler for different people to try their hand at it. +1 for the architectural separation of concerns, definitely. For analysability and maintainability, easy access and usage, and as a consequence for dissemination. -- Boris Baldassari Castalia Solutions -- Elegant Software Engineering Web: http://castalia.solutions Tel: +33 6 48 03 82 89
