Re: Podling Report Reminder - November 2016

[Matthew Hayes]

I added the report for DataFu:

http://wiki.apache.org/incubator/November2016

-Matt

On Mon, Oct 24, 2016 at 5:00 PM,  wrote:

> Dear podling,
>
> This email was sent by an automated system on behalf of the Apache
> Incubator PMC. It is an initial reminder to give you plenty of time to
> prepare your quarterly board report.
>
> The board meeting is scheduled for Wed, 16 November 2016, 10:30 am PDT.
> The report for your podling will form a part of the Incubator PMC
> report. The Incubator PMC requires your report to be submitted 2 weeks
> before the board meeting, to allow sufficient time for review and
> submission (Wed, November 02).
>
> Please submit your report with sufficient time to allow the Incubator
> PMC, and subsequently board members to review and digest. Again, the
> very latest you should submit your report is 2 weeks prior to the board
> meeting.
>
> Thanks,
>
> The Apache Incubator PMC
>
> Submitting your Report
>
> --
>
> Your report should contain the following:
>
> *   Your project name
> *   A brief description of your project, which assumes no knowledge of
> the project or necessarily of its field
> *   A list of the three most important issues to address in the move
> towards graduation.
> *   Any issues that the Incubator PMC or ASF Board might wish/need to be
> aware of
> *   How has the community developed since the last report
> *   How has the project developed since the last report.
>
> This should be appended to the Incubator Wiki page at:
>
> http://wiki.apache.org/incubator/November2016
>
> Note: This is manually populated. You may need to wait a little before
> this page is created from a template.
>
> Mentors
> ---
>
> Mentors should review reports for their project(s) and sign them off on
> the Incubator wiki page. Signing off reports shows that you are
> following the project - projects that are not signed may raise alarms
> for the Incubator PMC.
>
> Incubator PMC
>

[jira] [Commented] (DATAFU-87) Edit distance

[Eyal Allweil (JIRA)]

[ 
https://issues.apache.org/jira/browse/DATAFU-87?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15606106#comment-15606106
 ] 

Eyal Allweil commented on DATAFU-87:


Hi Joydeep,

I want to begin by apologizing for the time it's taken us to get to your 
contribution. Did you ever continue with it? Have you compared your 
implementation with [the one in Apache Commons 
Text|https://github.com/apache/commons-text/blob/master/src/main/java/org/apache/commons/text/similarity/LevenshteinDistance.java]
 or [Commons 
Lang|https://github.com/apache/commons-lang/blob/master/src/main/java/org/apache/commons/lang3/StringUtils.java#L7731]?
 (I think they follow the same algorithm, from _Algorithms on Strings, Trees 
and Sequences_ by Dan Gusfield and Chas Emerick)

> Edit distance
> -
>
> Key: DATAFU-87
> URL: https://issues.apache.org/jira/browse/DATAFU-87
> Project: DataFu
>  Issue Type: New Feature
>Affects Versions: 1.3.0
>Reporter: Joydeep Banerjee
> Attachments: DATAFU-87.patch
>
>
> [This is work-in-progress]
> Given 2 strings, provide a measure of dis-similarity (Levenshtein distance) 
> between them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (DATAFU-98) New UDF for Histogram / Frequency counting

[Eyal Allweil (JIRA)]

[ 
https://issues.apache.org/jira/browse/DATAFU-98?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15605952#comment-15605952
 ] 

Eyal Allweil commented on DATAFU-98:


Hi Russell.

First of all, I want to apologize for the time it's taken us to get to your 
contribution. I think it could be quite useful. Having said that, I wonder if 
the current version - without counters - gives us enough of an advantage over 
vanilla Pig. I think the following code (modified from your unit test) gives us 
nearly the same functionality as the UDF in the patch:

{noformat}
data_in = LOAD 'input' as (val:int);
-- data_in: "0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10", "11", "20"

intermediate_data = FOREACH data_in GENERATE val, (val / 5 * 5) AS binStart;

data_out = FOREACH (GROUP intermediate_data BY binStart) GENERATE group AS 
binStart, COUNT(intermediate_data) AS binCount;
-- data_out: (0,5),(5,5),(10,2),(20,1)

{noformat}

Unlike your UDF, missing bins are not included. But while including missing 
bins can be useful, I do wonder if a single skewed value can cause problems, 
especially with small bin sizes and long values. (as a performance-related 
aside, I would try to have FrequencyCounter.toBag() called only in the Final 
implementations, instead of the first two stages of the algebraic 
implementation, to minimize the data copied).

So it seems to me the current UDF has the advantage of having the missing bins, 
and it's obviously more readable and convenient than rewriting the Pig code I 
wrote above. Did you (or you, [~andrew.musselman]) run any performance tests? 
Maybe the Algebraic implementation runs faster than the vanilla Pig code by 
virtue of the combiner use.

Last (but not least!) the version you mentioned with counters sounds like it 
could be really great.


> New UDF for Histogram / Frequency counting
> --
>
> Key: DATAFU-98
> URL: https://issues.apache.org/jira/browse/DATAFU-98
> Project: DataFu
>  Issue Type: New Feature
>Reporter: Russell Melick
> Attachments: DATAFU-98.patch
>
>
> I was thinking of creating a new UDF to compute histograms / frequency counts 
> of input bags.  It seems like it would make sense to support ints, longs, 
> float, and doubles.  
> I tried looking around to see if this was already implemented, but 
> ValueHistogram and AggregateWordHistogram were about the only things I found. 
>  They seem to exist as an example job, and only work for Strings.
> https://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/lib/aggregate/ValueHistogram.html
> https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/examples/AggregateWordHistogram.html
> Should the user specify the bin size or the number of bins?  Specifying bin 
> size probably makes the implementation simpler since you can bin things 
> without having seen all of the data.
> I think it would make sense to implement a version of this that didn't need 
> any reducers.  It could use counters to keep track of the counts per bin 
> without sending any data to a reducer.  You would be able to call this 
> without a preceding GROUP BY as well.
> Here's my proposal for the two udfs.  This assumes the input data is two 
> columns, memberId and numConnections.
> {code}
> DEFINE BinnedFrequency datafu.pig.stats.BinnedFrequency('min=0;binSize=50')
> connections = LOAD 'connections' AS memberId, numConnections;
> connectionHistogram = FOREACH (GROUP connections ALL) GENERATE 
> BinnedFrequency(connections.numConnections);
> {code}
> The output here would be a bag with the frequency counts
> {code}
> {('0-49', 5), ('50-99', 0), ('100-149', 10)}
> {code}
> {code}
> DEFINE BinnedFrequencyCounter 
> datafu.pig.stats.BinnedFrequencyCounter('min=0;binSize=50;name=numConnectionsHistogram')
> connections = LOAD 'connections' AS memberId, numConnections;
> connections = FOREACH connections GENERATE 
> BinnedFrequencyCounter(numConnections);
> {code}
> The output here would just be a counter for each bin, all sharing the same 
> group of numConnectionsHistogram.  It would look something like
> numConnectionsHistogram.'0-49' = 5
> numConnectionsHistogram.'50-99' = 0
> numConnectionsHistogram.'100-149' = 10



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)