Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Heather Booker]

Hi Jesus,

I have a version of the task running, I'd love if you could take a
look and let me know if there are any changes you'd like to see.

It's at https://github.com/heatherbooker/xen-outreachy

It gets the mailboxes, analyzes them using Perceval and an
implementation of the well known jwz's threading algorithm
(https://www.jwz.org/doc/threading.html) then indexes them
in Elasticsearch.
Each document in ES is a message, with its id being the
Message-ID and type being a modified Subject line from the
first message in a thread.

I hope this is what was intended for the task!

PS - Should I continue copying these messages to the
whole xen-devel mailing list, or is sending them to you
sufficient?

Thanks!

Heather

On Mon, Apr 17, 2017 at 2:04 AM, Jesus M. Gonzalez-Barahona <
j...@bitergia.com> wrote:

> On Sun, 2017-04-16 at 21:26 -0700, Heather Booker wrote:
> > Hi Jesus!
> >
> > I appreciate the info on the unicode error. I might have missed it,
> > but I also asked about the general microtask specifications. Here
> > was my original inquiry:
> > > And to clarify, my understanding is that the final result of
> > this task
> > > is an index of Xen data, with two types: commits and messages.
> > > Each commit document should contain its original information
> > > from git, plus the name of the branch it was developed in. And
> > > should only the mbox messages which appear to be associated
> > > with a specific commit exist in the final index? Is there some
> > > key information in messages that is supposed to indicate the
> > > association of a given commit with a git branch? I would be
> > > grateful if you could specify the end goal a little more. :D
> >
> > Yeah, so overall I'm not sure I understand the relationship of
> > branches to the mailing list messages. Is this to be a simple
> > string parsing task wherein I should scan the message body
> > for the word "branch"? (I am guessing not ;P)
>
> I'm sorry, I understood that text was about the project, not about the
> microtask. The microtask is about either:
>
> * Producing an ES index with messages labeled by thread (by applying a
> threading algorithm to messages retrieved from archives), or
>
> * Producing an ES index with commits labeled by branch (by following
> refes, and parents information in the output produced by Perceval).
>
> In the complete project, both will be used to produce the final indexes
> that power the code review dashboard.
>
> > I will be happy to get back on developing once I better grasp
> > the goal! :)
>
> More clear now?
>
> If you want, let's schedule some IRC slot for clarifying whatever is
> not clear.
>
> Jesus.
>
> > Thanks!
> >
> > Heather
> >
> > On Sun, Apr 16, 2017 at 4:23 PM, Jesus M. Gonzalez-Barahona  > rgia.com> wrote:
> > > On Thu, 2017-04-13 at 00:47 -0700, Heather Booker wrote:
> > > > Hi,
> > > >
> > > > I submitted an application for this code review dashboard and
> > > > would love to keep working on the microtask once I get some
> > > > more info. :)
> > >
> > > Great! I answered your message, could you progress with the task?
> > >
> > > > I also came up with a general idea of how the project might be
> > > > split up - any feedback on this would be welcome! I wrote:
> > > >
> > > > "As said by Jesus, the big picture of this project will be
> > > porting
> > > > everything behind the current code review dashboard to use
> > > > Grimoire Lab tools, from the current state of using
> > > > MetricsGrimoire and custom scripts. I expect this would involve
> > > > Perceval for analyzing data, and Grimoire Elk may be useful in
> > > > further stages, or may be too general - this is something I would
> > > > wish to explore.
> > > > This project will also involve a migration from SQL to
> > > Elasticsearch
> > > > - because I believe the relevant data is mostly / all available
> > > in
> > > > places online, I am unsure whether this would need to be a direct
> > > > migration. However, looking at the current SQL setup would be
> > > > beneficial to understanding the desired format of the
> > > Elasticsearch
> > > > indexes.
> > > > I would love to dive into this project and have 3 main parts -
> > > > getting
> > > > data into ES, turning it into dashboard displays, and then fine
> > > > tuning
> > > > and perhaps augmenting the dashboard to improve its usefulness.
> > > > Getting data into ES may seem simple but I believe that once it
> > > > needs to be used for the dashboard, many realizations will pop up
> > > > - thus I’d like to leave maybe 2-3 weeks for that first step, 6-7
> > > > weeks
> > > > for the visualizations (which will include querying the data),
> > > and
> > > > the
> > > > final 3 weeks for touch ups and improvements."
> > >
> > > The plan could be sound, but would need some tweaks, once your
> > > skills
> > > in Python are clear, which could be the main blocker for the first
> > > stages.
> > >
> > > > Does this sound like an accurate summary and reasonable
> > > timeline?
> 

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Sun, 2017-04-16 at 21:26 -0700, Heather Booker wrote:
> Hi Jesus!
> 
> I appreciate the info on the unicode error. I might have missed it,
> but I also asked about the general microtask specifications. Here
> was my original inquiry:
> > And to clarify, my understanding is that the final result of
> this task
> > is an index of Xen data, with two types: commits and messages.
> > Each commit document should contain its original information
> > from git, plus the name of the branch it was developed in. And
> > should only the mbox messages which appear to be associated
> > with a specific commit exist in the final index? Is there some
> > key information in messages that is supposed to indicate the
> > association of a given commit with a git branch? I would be
> > grateful if you could specify the end goal a little more. :D
> 
> Yeah, so overall I'm not sure I understand the relationship of
> branches to the mailing list messages. Is this to be a simple
> string parsing task wherein I should scan the message body
> for the word "branch"? (I am guessing not ;P)

I'm sorry, I understood that text was about the project, not about the
microtask. The microtask is about either:

* Producing an ES index with messages labeled by thread (by applying a
threading algorithm to messages retrieved from archives), or

* Producing an ES index with commits labeled by branch (by following
refes, and parents information in the output produced by Perceval).

In the complete project, both will be used to produce the final indexes
that power the code review dashboard.

> I will be happy to get back on developing once I better grasp
> the goal! :)

More clear now?

If you want, let's schedule some IRC slot for clarifying whatever is
not clear.

Jesus.

> Thanks!
> 
> Heather
> 
> On Sun, Apr 16, 2017 at 4:23 PM, Jesus M. Gonzalez-Barahona  rgia.com> wrote:
> > On Thu, 2017-04-13 at 00:47 -0700, Heather Booker wrote:
> > > Hi,
> > >
> > > I submitted an application for this code review dashboard and
> > > would love to keep working on the microtask once I get some
> > > more info. :)
> > 
> > Great! I answered your message, could you progress with the task?
> > 
> > > I also came up with a general idea of how the project might be
> > > split up - any feedback on this would be welcome! I wrote:
> > >
> > > "As said by Jesus, the big picture of this project will be
> > porting
> > > everything behind the current code review dashboard to use
> > > Grimoire Lab tools, from the current state of using
> > > MetricsGrimoire and custom scripts. I expect this would involve
> > > Perceval for analyzing data, and Grimoire Elk may be useful in
> > > further stages, or may be too general - this is something I would
> > > wish to explore.
> > > This project will also involve a migration from SQL to
> > Elasticsearch
> > > - because I believe the relevant data is mostly / all available
> > in
> > > places online, I am unsure whether this would need to be a direct
> > > migration. However, looking at the current SQL setup would be
> > > beneficial to understanding the desired format of the
> > Elasticsearch
> > > indexes.
> > > I would love to dive into this project and have 3 main parts -
> > > getting
> > > data into ES, turning it into dashboard displays, and then fine
> > > tuning
> > > and perhaps augmenting the dashboard to improve its usefulness.
> > > Getting data into ES may seem simple but I believe that once it
> > > needs to be used for the dashboard, many realizations will pop up
> > > - thus I’d like to leave maybe 2-3 weeks for that first step, 6-7
> > > weeks
> > > for the visualizations (which will include querying the data),
> > and
> > > the
> > > final 3 weeks for touch ups and improvements."
> > 
> > The plan could be sound, but would need some tweaks, once your
> > skills
> > in Python are clear, which could be the main blocker for the first
> > stages.
> > 
> > > Does this sound like an accurate summary and reasonable
> > timeline? 
> > > And I am guessing that from Jesus's involvement with the threads
> > > that Jesus would be the mentor, is that correct? :)
> > 
> > Yes, I would be ;-)
> > 
> >         Jesus.
> > 
> > > Thanks!
> > >
> > > Heather
> > >
> > >
> > > On Sun, Apr 9, 2017 at 9:50 PM, Heather Booker  > gmai
> > > l.com> wrote:
> > > > Hi Jesus,
> > > >
> > > > While using the Elasticsearch python library
> > > > (https://elasticsearch-py.readthedocs.io/en/master/) to add
> > mbox
> > > > messages to an index, I would get a UnicodeEncodeError:
> > > > "'utf-8' codec can't encode character '\udca0' in position 767:
> > > > surrogates not allowed".
> > > >
> > > > Investigating in Grimoire elk https://github.com/grim
> > > > oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0
> > > > 8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that 
> > > > perhaps that tool instead uses Latin-1 encoding, but I found
> > that
> > > > to then produce a serialization error (their custom error
> > message:

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Heather Booker]

Hi Jesus!

I appreciate the info on the unicode error. I might have missed it,
but I also asked about the general microtask specifications. Here
was my original inquiry:
> And to clarify, my understanding is that the final result of this task
> is an index of Xen data, with two types: commits and messages.
> Each commit document should contain its original information
> from git, plus the name of the branch it was developed in. And
> should only the mbox messages which appear to be associated
> with a specific commit exist in the final index? Is there some
> key information in messages that is supposed to indicate the
> association of a given commit with a git branch? I would be
> grateful if you could specify the end goal a little more. :D

Yeah, so overall I'm not sure I understand the relationship of
branches to the mailing list messages. Is this to be a simple
string parsing task wherein I should scan the message body
for the word "branch"? (I am guessing not ;P)

I will be happy to get back on developing once I better grasp
the goal! :)

Thanks!

Heather

On Sun, Apr 16, 2017 at 4:23 PM, Jesus M. Gonzalez-Barahona <
j...@bitergia.com> wrote:

> On Thu, 2017-04-13 at 00:47 -0700, Heather Booker wrote:
> > Hi,
> >
> > I submitted an application for this code review dashboard and
> > would love to keep working on the microtask once I get some
> > more info. :)
>
> Great! I answered your message, could you progress with the task?
>
> > I also came up with a general idea of how the project might be
> > split up - any feedback on this would be welcome! I wrote:
> >
> > "As said by Jesus, the big picture of this project will be porting
> > everything behind the current code review dashboard to use
> > Grimoire Lab tools, from the current state of using
> > MetricsGrimoire and custom scripts. I expect this would involve
> > Perceval for analyzing data, and Grimoire Elk may be useful in
> > further stages, or may be too general - this is something I would
> > wish to explore.
> > This project will also involve a migration from SQL to Elasticsearch
> > - because I believe the relevant data is mostly / all available in
> > places online, I am unsure whether this would need to be a direct
> > migration. However, looking at the current SQL setup would be
> > beneficial to understanding the desired format of the Elasticsearch
> > indexes.
> > I would love to dive into this project and have 3 main parts -
> > getting
> > data into ES, turning it into dashboard displays, and then fine
> > tuning
> > and perhaps augmenting the dashboard to improve its usefulness.
> > Getting data into ES may seem simple but I believe that once it
> > needs to be used for the dashboard, many realizations will pop up
> > - thus I’d like to leave maybe 2-3 weeks for that first step, 6-7
> > weeks
> > for the visualizations (which will include querying the data), and
> > the
> > final 3 weeks for touch ups and improvements."
>
> The plan could be sound, but would need some tweaks, once your skills
> in Python are clear, which could be the main blocker for the first
> stages.
>
> > Does this sound like an accurate summary and reasonable timeline?
> > And I am guessing that from Jesus's involvement with the threads
> > that Jesus would be the mentor, is that correct? :)
>
> Yes, I would be ;-)
>
> Jesus.
>
> > Thanks!
> >
> > Heather
> >
> >
> > On Sun, Apr 9, 2017 at 9:50 PM, Heather Booker  > l.com> wrote:
> > > Hi Jesus,
> > >
> > > While using the Elasticsearch python library
> > > (https://elasticsearch-py.readthedocs.io/en/master/) to add mbox
> > > messages to an index, I would get a UnicodeEncodeError:
> > > "'utf-8' codec can't encode character '\udca0' in position 767:
> > > surrogates not allowed".
> > >
> > > Investigating in Grimoire elk https://github.com/grim
> > > oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0
> > > 8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that
> > > perhaps that tool instead uses Latin-1 encoding, but I found that
> > > to then produce a serialization error (their custom error message:
> > > "Unable to serialize %r (type: %s)"). I suppose this is because
> > > now it's bytes; of course, converting back to string after encoding
> > > just cycles back to the first error.
> > >
> > > As somewhat of a Python newbie I don't really know how to tackle
> > > this! My thought atm is to splice the offending character out
> > > of the message.
> > >
> > > And to clarify, my understanding is that the final result of this
> > > task
> > > is an index of Xen data, with two types: commits and messages.
> > > Each commit document should contain its original information
> > > from git, plus the name of the branch it was developed in. And
> > > should only the mbox messages which appear to be associated
> > > with a specific commit exist in the final index? Is there some
> > > key information in messages that is supposed to indicate the
> > > association of a given commit with a git branch? I w

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Thu, 2017-04-13 at 00:47 -0700, Heather Booker wrote:
> Hi,
> 
> I submitted an application for this code review dashboard and
> would love to keep working on the microtask once I get some
> more info. :)

Great! I answered your message, could you progress with the task?

> I also came up with a general idea of how the project might be
> split up - any feedback on this would be welcome! I wrote:
> 
> "As said by Jesus, the big picture of this project will be porting
> everything behind the current code review dashboard to use
> Grimoire Lab tools, from the current state of using
> MetricsGrimoire and custom scripts. I expect this would involve
> Perceval for analyzing data, and Grimoire Elk may be useful in
> further stages, or may be too general - this is something I would
> wish to explore.
> This project will also involve a migration from SQL to Elasticsearch
> - because I believe the relevant data is mostly / all available in
> places online, I am unsure whether this would need to be a direct
> migration. However, looking at the current SQL setup would be
> beneficial to understanding the desired format of the Elasticsearch
> indexes.
> I would love to dive into this project and have 3 main parts -
> getting
> data into ES, turning it into dashboard displays, and then fine
> tuning
> and perhaps augmenting the dashboard to improve its usefulness.
> Getting data into ES may seem simple but I believe that once it
> needs to be used for the dashboard, many realizations will pop up
> - thus I’d like to leave maybe 2-3 weeks for that first step, 6-7
> weeks
> for the visualizations (which will include querying the data), and
> the
> final 3 weeks for touch ups and improvements."

The plan could be sound, but would need some tweaks, once your skills
in Python are clear, which could be the main blocker for the first
stages.

> Does this sound like an accurate summary and reasonable timeline? 
> And I am guessing that from Jesus's involvement with the threads
> that Jesus would be the mentor, is that correct? :)

Yes, I would be ;-)

Jesus.

> Thanks!
> 
> Heather
> 
> 
> On Sun, Apr 9, 2017 at 9:50 PM, Heather Booker  l.com> wrote:
> > Hi Jesus,
> > 
> > While using the Elasticsearch python library
> > (https://elasticsearch-py.readthedocs.io/en/master/) to add mbox
> > messages to an index, I would get a UnicodeEncodeError:
> > "'utf-8' codec can't encode character '\udca0' in position 767:
> > surrogates not allowed".
> > 
> > Investigating in Grimoire elk https://github.com/grim
> > oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0
> > 8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that 
> > perhaps that tool instead uses Latin-1 encoding, but I found that
> > to then produce a serialization error (their custom error message:
> > "Unable to serialize %r (type: %s)"). I suppose this is because
> > now it's bytes; of course, converting back to string after encoding
> > just cycles back to the first error.
> > 
> > As somewhat of a Python newbie I don't really know how to tackle
> > this! My thought atm is to splice the offending character out
> > of the message. 
> > 
> > And to clarify, my understanding is that the final result of this
> > task
> > is an index of Xen data, with two types: commits and messages.
> > Each commit document should contain its original information
> > from git, plus the name of the branch it was developed in. And
> > should only the mbox messages which appear to be associated
> > with a specific commit exist in the final index? Is there some
> > key information in messages that is supposed to indicate the
> > association of a given commit with a git branch? I would be
> > grateful if you could specify the end goal a little more. :D
> > 
> > Thanks so much!
> > 
> > Heather
> > 
> > 
> > 
> > On Sat, Apr 8, 2017 at 10:02 AM, Jesus M. Gonzalez-Barahona  > tergia.com> wrote:
> > > On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
> > > > Hi Jesus, 
> > > >
> > > > Thanks for your reply!
> > > >
> > > > So about the task, instructions say after analyzing mboxes with
> > > > Perceval to
> > > > "store the resulting raw index in ElasticSearch" - what does
> > > raw
> > > > index mean?
> > > 
> > > In this context, I mean "storing the JSON documents produced by
> > > Perceval in an ElasticSearch index, as such". ElasticSearch
> > > stores JSON
> > > documents, so it is just uploading the output of Perceval to it.
> > > 
> > > > In terms of figuring out the elasticsearch structure, do I want
> > > an
> > > > index
> > > > (xen-devel mbox) with a type (message) and each object from the
> > > > perceval
> > > > output to be one document? Or should it be more fine-grained?
> > > 
> > > Exactly.
> > > 
> > > Saludos,
> > > 
> > >         Jesus.
> > > 
> > > > Cheers,
> > > >
> > > > Heather
> > > >
> > > > On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona  > > @biter
> > > > gia.com> wrote:
> > > > > On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wr

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Sun, 2017-04-09 at 21:50 -0700, Heather Booker wrote:
> Hi Jesus,
> 
> While using the Elasticsearch python library
> (https://elasticsearch-py.readthedocs.io/en/master/) to add mbox
> messages to an index, I would get a UnicodeEncodeError:
> "'utf-8' codec can't encode character '\udca0' in position 767:
> surrogates not allowed".
> 

What happens here is that Perceval has some assumptions about character
encoding, when reading messages (to convert them to Unicode strings).
If they are not fulfilled, it converts the character as "surrogate".
When trying to produce utf8 from those, that cannnot be done, since the
space for "surrogate" Unicode is thought to convert back to the
original encoding. But JSON expects the encoding to be utf8, so no luck
here.

The trick is to provide a serializer which either skips those messages,
or produces a "escaped" encoding for them.

See http://lucumr.pocoo.org/2013/7/2/the-updated-guide-to-unicode/ for
a detailed explanation.

Please, let me know if you can work from here...

Jesus.

> Investigating in Grimoire elk https://github.com/grim
> oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0
> 8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that 
> perhaps that tool instead uses Latin-1 encoding, but I found that
> to then produce a serialization error (their custom error message:
> "Unable to serialize %r (type: %s)"). I suppose this is because
> now it's bytes; of course, converting back to string after encoding
> just cycles back to the first error.
> 
> As somewhat of a Python newbie I don't really know how to tackle
> this! My thought atm is to splice the offending character out
> of the message. 
> 
> And to clarify, my understanding is that the final result of this
> task
> is an index of Xen data, with two types: commits and messages.
> Each commit document should contain its original information
> from git, plus the name of the branch it was developed in. And
> should only the mbox messages which appear to be associated
> with a specific commit exist in the final index? Is there some
> key information in messages that is supposed to indicate the
> association of a given commit with a git branch? I would be
> grateful if you could specify the end goal a little more. :D
> 
> Thanks so much!
> 
> Heather
> 
> 
> 
> On Sat, Apr 8, 2017 at 10:02 AM, Jesus M. Gonzalez-Barahona  rgia.com> wrote:
> > On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
> > > Hi Jesus, 
> > >
> > > Thanks for your reply!
> > >
> > > So about the task, instructions say after analyzing mboxes with
> > > Perceval to
> > > "store the resulting raw index in ElasticSearch" - what does raw
> > > index mean?
> > 
> > In this context, I mean "storing the JSON documents produced by
> > Perceval in an ElasticSearch index, as such". ElasticSearch stores
> > JSON
> > documents, so it is just uploading the output of Perceval to it.
> > 
> > > In terms of figuring out the elasticsearch structure, do I want
> > an
> > > index
> > > (xen-devel mbox) with a type (message) and each object from the
> > > perceval
> > > output to be one document? Or should it be more fine-grained?
> > 
> > Exactly.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > > Cheers,
> > >
> > > Heather
> > >
> > > On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona  > iter
> > > gia.com> wrote:
> > > > On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> > > > > Hi!
> > > > >
> > > > > I'd love to work on the Code Review Dashboard project for
> > this
> > > > round
> > > > > of Outreachy.
> > > >
> > > > Great!!
> > > >
> > > > > Are the steps outlined
> > > > > here http://markmail.org/message/7adkmords3imkswd still the
> > first
> > > > > contribution you'd like to see?
> > > >
> > > > Yes.
> > > >
> > > > > So is this a project that has been worked on in previous
> > rounds
> > > > of
> > > > > GSOC/Outreachy also?
> > > > > If so is there a place to find links to the previous
> > participants
> > > > > blogs? :)
> > > >
> > > > No. We had one participation at some point, but couldn't even
> > start
> > > > for
> > > > personal reasons. There are some people considering working on
> > this
> > > > for
> > > > this next round of Outreachy, however. You'll see their
> > messages in
> > > > this mailing list.
> > > >
> > > > > Should questions about how the specifications/completion of
> > the
> > > > > microtask be addressed to
> > > > > IRC or this list? If IRC, which channel - #xen-opw or
> > #metrics-
> > > > > grimoire? On that note, I'm 
> > > > > curious why #metrics-grimoire is the listed channel on the
> > > > project
> > > > > page - are main contributors
> > > > > involved in both projects? Or is it just because the Xen
> > > > dashboard
> > > > > doesn't have a channel?
> > > >
> > > > The code review is for the Xen project, but it is done with (I
> > > > mean,
> > > > the ssoftware used for it is) GrimoireLab, which for historical
> > > > reasons
> > > > uses the #metrics-grimoire c

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Heather Booker]

Hi,

I submitted an application for this code review dashboard and
would love to keep working on the microtask once I get some
more info. :)

I also came up with a general idea of how the project might be
split up - any feedback on this would be welcome! I wrote:

"As said by Jesus, the big picture of this project will be porting
everything behind the current code review dashboard to use
Grimoire Lab tools, from the current state of using
MetricsGrimoire and custom scripts. I expect this would involve
Perceval for analyzing data, and Grimoire Elk may be useful in
further stages, or may be too general - this is something I would
wish to explore.
This project will also involve a migration from SQL to Elasticsearch
- because I believe the relevant data is mostly / all available in
places online, I am unsure whether this would need to be a direct
migration. However, looking at the current SQL setup would be
beneficial to understanding the desired format of the Elasticsearch
indexes.
I would love to dive into this project and have 3 main parts - getting
data into ES, turning it into dashboard displays, and then fine tuning
and perhaps augmenting the dashboard to improve its usefulness.
Getting data into ES may seem simple but I believe that once it
needs to be used for the dashboard, many realizations will pop up
- thus I’d like to leave maybe 2-3 weeks for that first step, 6-7 weeks
for the visualizations (which will include querying the data), and the
final 3 weeks for touch ups and improvements."

Does this sound like an accurate summary and reasonable timeline?
And I am guessing that from Jesus's involvement with the threads
that Jesus would be the mentor, is that correct? :)

Thanks!

Heather


On Sun, Apr 9, 2017 at 9:50 PM, Heather Booker 
wrote:

> Hi Jesus,
>
> While using the Elasticsearch python library
> (https://elasticsearch-py.readthedocs.io/en/master/) to add mbox
> messages to an index, I would get a UnicodeEncodeError:
> "'utf-8' codec can't encode character '\udca0' in position 767:
> surrogates not allowed".
>
> Investigating in Grimoire elk https://github.com/grim
> oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0
> 8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that
> perhaps that tool instead uses Latin-1 encoding, but I found that
> to then produce a serialization error (their custom error message:
> "Unable to serialize %r (type: %s)"). I suppose this is because
> now it's bytes; of course, converting back to string after encoding
> just cycles back to the first error.
>
> As somewhat of a Python newbie I don't really know how to tackle
> this! My thought atm is to splice the offending character out
> of the message.
>
> And to clarify, my understanding is that the final result of this task
> is an index of Xen data, with two types: commits and messages.
> Each commit document should contain its original information
> from git, plus the name of the branch it was developed in. And
> should only the mbox messages which appear to be associated
> with a specific commit exist in the final index? Is there some
> key information in messages that is supposed to indicate the
> association of a given commit with a git branch? I would be
> grateful if you could specify the end goal a little more. :D
>
> Thanks so much!
>
> Heather
>
>
>
> On Sat, Apr 8, 2017 at 10:02 AM, Jesus M. Gonzalez-Barahona <
> j...@bitergia.com> wrote:
>
>> On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
>> > Hi Jesus,
>> >
>> > Thanks for your reply!
>> >
>> > So about the task, instructions say after analyzing mboxes with
>> > Perceval to
>> > "store the resulting raw index in ElasticSearch" - what does raw
>> > index mean?
>>
>> In this context, I mean "storing the JSON documents produced by
>> Perceval in an ElasticSearch index, as such". ElasticSearch stores JSON
>> documents, so it is just uploading the output of Perceval to it.
>>
>> > In terms of figuring out the elasticsearch structure, do I want an
>> > index
>> > (xen-devel mbox) with a type (message) and each object from the
>> > perceval
>> > output to be one document? Or should it be more fine-grained?
>>
>> Exactly.
>>
>> Saludos,
>>
>> Jesus.
>>
>> > Cheers,
>> >
>> > Heather
>> >
>> > On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona > > gia.com> wrote:
>> > > On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
>> > > > Hi!
>> > > >
>> > > > I'd love to work on the Code Review Dashboard project for this
>> > > round
>> > > > of Outreachy.
>> > >
>> > > Great!!
>> > >
>> > > > Are the steps outlined
>> > > > here http://markmail.org/message/7adkmords3imkswd still the first
>> > > > contribution you'd like to see?
>> > >
>> > > Yes.
>> > >
>> > > > So is this a project that has been worked on in previous rounds
>> > > of
>> > > > GSOC/Outreachy also?
>> > > > If so is there a place to find links to the previous participants
>> > > > blogs? :)
>> > >
>> > > No. We had one participation at some point, but couldn't 

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Heather Booker]

Hi Jesus,

While using the Elasticsearch python library
(https://elasticsearch-py.readthedocs.io/en/master/) to add mbox
messages to an index, I would get a UnicodeEncodeError:
"'utf-8' codec can't encode character '\udca0' in position 767:
surrogates not allowed".

Investigating in Grimoire elk https://github.com/grim
oirelab/GrimoireELK/blob/96b00bc682485976104a6825ca63ae0
8639deacc/grimoire_elk/elk/mbox.py#L200 seems to show that
perhaps that tool instead uses Latin-1 encoding, but I found that
to then produce a serialization error (their custom error message:
"Unable to serialize %r (type: %s)"). I suppose this is because
now it's bytes; of course, converting back to string after encoding
just cycles back to the first error.

As somewhat of a Python newbie I don't really know how to tackle
this! My thought atm is to splice the offending character out
of the message.

And to clarify, my understanding is that the final result of this task
is an index of Xen data, with two types: commits and messages.
Each commit document should contain its original information
from git, plus the name of the branch it was developed in. And
should only the mbox messages which appear to be associated
with a specific commit exist in the final index? Is there some
key information in messages that is supposed to indicate the
association of a given commit with a git branch? I would be
grateful if you could specify the end goal a little more. :D

Thanks so much!

Heather



On Sat, Apr 8, 2017 at 10:02 AM, Jesus M. Gonzalez-Barahona <
j...@bitergia.com> wrote:

> On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
> > Hi Jesus,
> >
> > Thanks for your reply!
> >
> > So about the task, instructions say after analyzing mboxes with
> > Perceval to
> > "store the resulting raw index in ElasticSearch" - what does raw
> > index mean?
>
> In this context, I mean "storing the JSON documents produced by
> Perceval in an ElasticSearch index, as such". ElasticSearch stores JSON
> documents, so it is just uploading the output of Perceval to it.
>
> > In terms of figuring out the elasticsearch structure, do I want an
> > index
> > (xen-devel mbox) with a type (message) and each object from the
> > perceval
> > output to be one document? Or should it be more fine-grained?
>
> Exactly.
>
> Saludos,
>
> Jesus.
>
> > Cheers,
> >
> > Heather
> >
> > On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona  > gia.com> wrote:
> > > On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> > > > Hi!
> > > >
> > > > I'd love to work on the Code Review Dashboard project for this
> > > round
> > > > of Outreachy.
> > >
> > > Great!!
> > >
> > > > Are the steps outlined
> > > > here http://markmail.org/message/7adkmords3imkswd still the first
> > > > contribution you'd like to see?
> > >
> > > Yes.
> > >
> > > > So is this a project that has been worked on in previous rounds
> > > of
> > > > GSOC/Outreachy also?
> > > > If so is there a place to find links to the previous participants
> > > > blogs? :)
> > >
> > > No. We had one participation at some point, but couldn't even start
> > > for
> > > personal reasons. There are some people considering working on this
> > > for
> > > this next round of Outreachy, however. You'll see their messages in
> > > this mailing list.
> > >
> > > > Should questions about how the specifications/completion of the
> > > > microtask be addressed to
> > > > IRC or this list? If IRC, which channel - #xen-opw or #metrics-
> > > > grimoire? On that note, I'm
> > > > curious why #metrics-grimoire is the listed channel on the
> > > project
> > > > page - are main contributors
> > > > involved in both projects? Or is it just because the Xen
> > > dashboard
> > > > doesn't have a channel?
> > >
> > > The code review is for the Xen project, but it is done with (I
> > > mean,
> > > the ssoftware used for it is) GrimoireLab, which for historical
> > > reasons
> > > uses the #metrics-grimoire channel. That's why it is likely that
> > > you
> > > find somebody from the project there.
> > >
> > > If you have questions, and find me around in IRC, please ping me.
> > > If
> > > I'm not available, please send an email message.
> > >
> > > Saludos,
> > >
> > > Jesus.
> > >
> > > > Thanks!
> > > >
> > > > Heather
> > > > ___
> > > > Xen-devel mailing list
> > > > Xen-devel@lists.xen.org
> > > > https://lists.xen.org/xen-devel
> > > --
> > > Bitergia: http://bitergia.com
> > > /me at Twitter: https://twitter.com/jgbarah
> > >
> > >
> >
> > ___
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > https://lists.xen.org/xen-devel
> --
> Bitergia: http://bitergia.com
> /me at Twitter: https://twitter.com/jgbarah
>
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Fri, 2017-04-07 at 15:49 -0700, Heather Booker wrote:
> Hi Jesus, 
> 
> Thanks for your reply!
> 
> So about the task, instructions say after analyzing mboxes with
> Perceval to
> "store the resulting raw index in ElasticSearch" - what does raw
> index mean?

In this context, I mean "storing the JSON documents produced by
Perceval in an ElasticSearch index, as such". ElasticSearch stores JSON
documents, so it is just uploading the output of Perceval to it.

> In terms of figuring out the elasticsearch structure, do I want an
> index
> (xen-devel mbox) with a type (message) and each object from the
> perceval
> output to be one document? Or should it be more fine-grained?

Exactly.

Saludos,

Jesus.

> Cheers,
> 
> Heather
> 
> On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona  gia.com> wrote:
> > On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> > > Hi!
> > >
> > > I'd love to work on the Code Review Dashboard project for this
> > round
> > > of Outreachy.
> > 
> > Great!!
> > 
> > > Are the steps outlined
> > > here http://markmail.org/message/7adkmords3imkswd still the first
> > > contribution you'd like to see?
> > 
> > Yes.
> > 
> > > So is this a project that has been worked on in previous rounds
> > of
> > > GSOC/Outreachy also?
> > > If so is there a place to find links to the previous participants
> > > blogs? :)
> > 
> > No. We had one participation at some point, but couldn't even start
> > for
> > personal reasons. There are some people considering working on this
> > for
> > this next round of Outreachy, however. You'll see their messages in
> > this mailing list.
> > 
> > > Should questions about how the specifications/completion of the
> > > microtask be addressed to
> > > IRC or this list? If IRC, which channel - #xen-opw or #metrics-
> > > grimoire? On that note, I'm 
> > > curious why #metrics-grimoire is the listed channel on the
> > project
> > > page - are main contributors
> > > involved in both projects? Or is it just because the Xen
> > dashboard
> > > doesn't have a channel?
> > 
> > The code review is for the Xen project, but it is done with (I
> > mean,
> > the ssoftware used for it is) GrimoireLab, which for historical
> > reasons
> > uses the #metrics-grimoire channel. That's why it is likely that
> > you
> > find somebody from the project there.
> > 
> > If you have questions, and find me around in IRC, please ping me.
> > If
> > I'm not available, please send an email message.
> > 
> > Saludos,
> > 
> >         Jesus.
> > 
> > > Thanks!
> > >
> > > Heather
> > > ___
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > https://lists.xen.org/xen-devel
> > --
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
> > 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Heather Booker]

Hi Jesus,

Thanks for your reply!

So about the task, instructions say after analyzing mboxes with Perceval to
"store the resulting raw index in ElasticSearch" - what does raw index mean?
In terms of figuring out the elasticsearch structure, do I want an index
(xen-devel mbox) with a type (message) and each object from the perceval
output to be one document? Or should it be more fine-grained?

Cheers,

Heather

On Thu, Apr 6, 2017 at 7:05 AM, Jesus M. Gonzalez-Barahona  wrote:

> On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> > Hi!
> >
> > I'd love to work on the Code Review Dashboard project for this round
> > of Outreachy.
>
> Great!!
>
> > Are the steps outlined
> > here http://markmail.org/message/7adkmords3imkswd still the first
> > contribution you'd like to see?
>
> Yes.
>
> > So is this a project that has been worked on in previous rounds of
> > GSOC/Outreachy also?
> > If so is there a place to find links to the previous participants
> > blogs? :)
>
> No. We had one participation at some point, but couldn't even start for
> personal reasons. There are some people considering working on this for
> this next round of Outreachy, however. You'll see their messages in
> this mailing list.
>
> > Should questions about how the specifications/completion of the
> > microtask be addressed to
> > IRC or this list? If IRC, which channel - #xen-opw or #metrics-
> > grimoire? On that note, I'm
> > curious why #metrics-grimoire is the listed channel on the project
> > page - are main contributors
> > involved in both projects? Or is it just because the Xen dashboard
> > doesn't have a channel?
>
> The code review is for the Xen project, but it is done with (I mean,
> the ssoftware used for it is) GrimoireLab, which for historical reasons
> uses the #metrics-grimoire channel. That's why it is likely that you
> find somebody from the project there.
>
> If you have questions, and find me around in IRC, please ping me. If
> I'm not available, please send an email message.
>
> Saludos,
>
> Jesus.
>
> > Thanks!
> >
> > Heather
> > ___
> > Xen-devel mailing list
> > Xen-devel@lists.xen.org
> > https://lists.xen.org/xen-devel
> --
> Bitergia: http://bitergia.com
> /me at Twitter: https://twitter.com/jgbarah
>
>
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Wed, 2017-04-05 at 16:43 -0700, Heather Booker wrote:
> Hi!
> 
> I'd love to work on the Code Review Dashboard project for this round
> of Outreachy.

Great!!

> Are the steps outlined
> here http://markmail.org/message/7adkmords3imkswd still the first
> contribution you'd like to see?

Yes.

> So is this a project that has been worked on in previous rounds of
> GSOC/Outreachy also?
> If so is there a place to find links to the previous participants
> blogs? :)

No. We had one participation at some point, but couldn't even start for
personal reasons. There are some people considering working on this for
this next round of Outreachy, however. You'll see their messages in
this mailing list.

> Should questions about how the specifications/completion of the
> microtask be addressed to
> IRC or this list? If IRC, which channel - #xen-opw or #metrics-
> grimoire? On that note, I'm 
> curious why #metrics-grimoire is the listed channel on the project
> page - are main contributors
> involved in both projects? Or is it just because the Xen dashboard
> doesn't have a channel?

The code review is for the Xen project, but it is done with (I mean,
the ssoftware used for it is) GrimoireLab, which for historical reasons
uses the #metrics-grimoire channel. That's why it is likely that you
find somebody from the project there.

If you have questions, and find me around in IRC, please ping me. If
I'm not available, please send an email message.

Saludos,

Jesus.

> Thanks!
> 
> Heather
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] Outreachy project - Xen Code Review Dashboard

[Heather Booker]

Hi!

I'd love to work on the Code Review Dashboard project for this round of
Outreachy.

Are the steps outlined here http://markmail.org/message/7adkmords3imkswd
still the first
contribution you'd like to see?

So is this a project that has been worked on in previous rounds of
GSOC/Outreachy also?
If so is there a place to find links to the previous participants blogs? :)

Should questions about how the specifications/completion of the microtask
be addressed to
IRC or this list? If IRC, which channel - #xen-opw or #metrics-grimoire? On
that note, I'm
curious why #metrics-grimoire is the listed channel on the project page -
are main contributors
involved in both projects? Or is it just because the Xen dashboard doesn't
have a channel?

Thanks!

Heather
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Sun, 2017-04-02 at 13:09 -0700, vaishnavi.ur...@gmail.com wrote:
> Hi Jesus,
> I understand that the Perceval scripts are written in Python. I am
> familiar with Python and comfortable working in it. So does the micro
> task remain the same? To run Perceval on xen-devel list and save
> output to elastic search ?

Yes, you can start that way, a the link suggested by Lars suggests.

> Also, when are you generally online on IRC? 

At different times, feel free to ping me (jgbarah at #metrics-grimoire
on Freenode) if you see me. But maybe it is better to set some slot.
I'm in CEST timezone. What timezone are you?

Saludos,

Jesus.

> Thanks a lot!
> 
> Vaishnavi
> 
> > On Apr 2, 2017, at 11:20 AM, Jesus M. Gonzalez-Barahona  > ia.com> wrote:
> > 
> > > On Thu, 2017-03-30 at 18:48 -0700, Vaishnavi Ramesh Jayaraman
> > > wrote:
> > > Hi,
> > > I am Vaishnavi, interested in contributing to the Xen Project as
> > > part
> > > of the Outreachy Program. I am particularly interested in working
> > > on
> > > the Xen Code Review Dashboard.
> > > 
> > > I have worked on the ElasticSearch - Logstash- Kibana (ELK) stack
> > > previously and am comfortable with Javascript.
> > > 
> > > It would be great if you could give me pointers on how to get
> > > started!
> > > 
> > > Also, I am unable to join the mailing list for this project -
> > > metrics
> > > -grimo...@lists.libresoft.es 
> > 
> > Hi, Vaishnavi,
> > 
> > First of all, thanks for your interest.
> > 
> > And now, a warning notice: this project will require mainly Python
> > and
> > noSQL (ElasticSearch, in particular) knowledge. I see you're
> > familiar
> > with ELK, what about Python?
> > 
> > [Lars, as I just commented in another message, I now notice this is
> > wrong in the project description at
> > https://wiki.xenproject.org/wiki/Outreach_Program_Projects
> > sorry about that. Could we change it?]
> > 
> > If you're still interested, I guess it would be good to have a
> > quick
> > IRC chat, and discuss about next steps.
> > 
> > Saludos,
> > 
> >    Jesus.
> > 
> > > ___
> > > Xen-devel mailing list
> > > Xen-devel@lists.xen.org
> > > https://lists.xen.org/xen-devel
> > 
> > -- 
> > Bitergia: http://bitergia.com
> > /me at Twitter: https://twitter.com/jgbarah
> > 
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[vaishnavi . ur777]

Hi Jesus,
I understand that the Perceval scripts are written in Python. I am familiar 
with Python and comfortable working in it. So does the micro task remain the 
same? To run Perceval on xen-devel list and save output to elastic search ?

Also, when are you generally online on IRC? 

Thanks a lot!

Vaishnavi

> On Apr 2, 2017, at 11:20 AM, Jesus M. Gonzalez-Barahona  
> wrote:
> 
>> On Thu, 2017-03-30 at 18:48 -0700, Vaishnavi Ramesh Jayaraman wrote:
>> Hi,
>> I am Vaishnavi, interested in contributing to the Xen Project as part
>> of the Outreachy Program. I am particularly interested in working on
>> the Xen Code Review Dashboard.
>> 
>> I have worked on the ElasticSearch - Logstash- Kibana (ELK) stack
>> previously and am comfortable with Javascript.
>> 
>> It would be great if you could give me pointers on how to get
>> started!
>> 
>> Also, I am unable to join the mailing list for this project - metrics
>> -grimo...@lists.libresoft.es 
> 
> Hi, Vaishnavi,
> 
> First of all, thanks for your interest.
> 
> And now, a warning notice: this project will require mainly Python and
> noSQL (ElasticSearch, in particular) knowledge. I see you're familiar
> with ELK, what about Python?
> 
> [Lars, as I just commented in another message, I now notice this is
> wrong in the project description at
> https://wiki.xenproject.org/wiki/Outreach_Program_Projects
> sorry about that. Could we change it?]
> 
> If you're still interested, I guess it would be good to have a quick
> IRC chat, and discuss about next steps.
> 
> Saludos,
> 
>Jesus.
> 
>> ___
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> https://lists.xen.org/xen-devel
> -- 
> Bitergia: http://bitergia.com
> /me at Twitter: https://twitter.com/jgbarah
> 

___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Jesus M. Gonzalez-Barahona]

On Thu, 2017-03-30 at 18:48 -0700, Vaishnavi Ramesh Jayaraman wrote:
> Hi,
> I am Vaishnavi, interested in contributing to the Xen Project as part
> of the Outreachy Program. I am particularly interested in working on
> the Xen Code Review Dashboard.
> 
> I have worked on the ElasticSearch - Logstash- Kibana (ELK) stack
> previously and am comfortable with Javascript.
> 
> It would be great if you could give me pointers on how to get
> started!
> 
> Also, I am unable to join the mailing list for this project - metrics
> -grimo...@lists.libresoft.es 

Hi, Vaishnavi,

First of all, thanks for your interest.

And now, a warning notice: this project will require mainly Python and
noSQL (ElasticSearch, in particular) knowledge. I see you're familiar
with ELK, what about Python?

[Lars, as I just commented in another message, I now notice this is
wrong in the project description at
https://wiki.xenproject.org/wiki/Outreach_Program_Projects
sorry about that. Could we change it?]

If you're still interested, I guess it would be good to have a quick
IRC chat, and discuss about next steps.

Saludos,

Jesus.

> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
-- 
Bitergia: http://bitergia.com
/me at Twitter: https://twitter.com/jgbarah


___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Outreachy project - Xen Code Review Dashboard

[Lars Kurth]

Hi,

> On 31 Mar 2017, at 02:48, Vaishnavi Ramesh Jayaraman 
>  wrote:
> 
> Hi,
> I am Vaishnavi, interested in contributing to the Xen Project as part of the 
> Outreachy Program. I am particularly interested in working on the Xen Code 
> Review Dashboard.
> 
> I have worked on the ElasticSearch - Logstash- Kibana (ELK) stack previously 
> and am comfortable with Javascript.
> 
> It would be great if you could give me pointers on how to get started!

You may want to look at http://markmail.org/message/7adkmords3imkswd 


Regards
Lars___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

[Xen-devel] Outreachy project - Xen Code Review Dashboard

[Vaishnavi Ramesh Jayaraman]

Hi,
I am Vaishnavi, interested in contributing to the Xen Project as part of
the Outreachy Program. I am particularly interested in working on the Xen
Code Review Dashboard.

I have worked on the ElasticSearch - Logstash- Kibana (ELK) stack
previously and am comfortable with Javascript.

It would be great if you could give me pointers on how to get started!

Also, I am unable to join the mailing list for this project -
metrics-grimo...@lists.libresoft.es


Could you please add me? Thanks!

Looking forward to contributing!

Vaishnavi
___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel