Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.
The following page has been changed by CorinneC: http://wiki.apache.org/pig/PigTutorial ------------------------------------------------------------------------------ The tutorial pig script (tutorial.pig or tutorial-local.pig) does the following: + - * Registers the tutorial JAR file so that the user-defined functions (UDFs) can be called in the script. + * Register the tutorial JAR file so that the included UDFs can be called in the script. + {{{ + REGISTER ./tutorial.jar; + }}} + + - * Uses the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to load the excite log file (excite.log or excite-small.log) into the ârawâ bag as an array of records with the fields '''user''', '''time''', and '''query'''. + * Use the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to load the excite log file (excite.log or excite-small.log) into the ârawâ bag as an array of records with the fields '''user''', '''time''', and '''query'''. Input: (user,time,query) + {{{ + raw = LOAD 'excite.log' USING PigStorage('\t') AS (user, time, query); + }}} + - * Calls the NonURLDetector UDF to remove records if the query field is empty or a URL. + * Call the NonURLDetector UDF to remove records if the query field is empty or a URL. + {{{ + clean1 = FILTER raw BY org.apache.pig.tutorial.NonURLDetector(query); + }}} + - * Calls the !ToLower UDF to change the query field to lowercase. + * Call the !ToLower UDF to change the query field to lowercase. + {{{ + clean2 = FOREACH clean1 GENERATE user, time, org.apache.pig.tutorial.ToLower(query) as query; + }}} + - * Calls the !NonPornDetector UDF to remove records if the query field contains porn terms. + * Call the !NonPornDetector UDF to remove records if the query field contains porn terms. - * Calls the !ExtractHour UDF to extract the hour from the time field. + {{{ + clean3 = FILTER clean2 BY org.apache.pig.tutorial.NonPornDetector(query); + }}} + + * Because the log file only contains queries for a single day, we are only interested in the hour. The excite query log timestamp format is YYMMDDHHMMSS. Call the !ExtractHour UDF to extract the hour (HH) from the time field. + {{{ + houred = FOREACH clean3 GENERATE user, org.apache.pig.tutorial.ExtractHour(time) as hour, query; + }}} + - * Calls the N!GramGenerator UDF to compose the n-grams of the query. + * Call the N!GramGenerator UDF to compose the n-grams of the query. + {{{ + ngramed1 = FOREACH houred GENERATE user, hour, flatten(org.apache.pig.tutorial.NGramGenerator(query)) as ngram; + }}} + - * Uses the [http://wiki.apache.org/pig/PigLatin#DISTINCT:_Eliminating_duplicates_in_data DISTINCT] command to get the unique n-grams for all records. + * Use the [http://wiki.apache.org/pig/PigLatin#DISTINCT:_Eliminating_duplicates_in_data DISTINCT] command to get the unique n-grams for all records. + {{{ + ngramed2 = DISTINCT ngramed1; + }}} + - * Uses the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together GROUP] command to group records by n-gram and hour. + * Use the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together GROUP] command to group records by n-gram and hour. + {{{ + hour_frequency1 = GROUP ngramed2 BY (ngram, hour); + }}} + - * Uses the [http://wiki.apache.org/pig/PigBuiltins COUNT] function to get the count (occurrences) of each n-gram. + * Use the [http://wiki.apache.org/pig/PigBuiltins COUNT] function to get the count (occurrences) of each n-gram. + {{{ + hour_frequency2 = FOREACH hour_frequency1 GENERATE flatten($0), COUNT($1) as count; + }}} + - * Uses the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together GROUP] command to group records by n-gram only. + * Use the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together GROUP] command to group records by n-gram only. Each group now corresponds to a distinct n-gram and has the count for each hour. - * Calls the !ScoreGenerator UDF to calculate a "popularity" score for the n-gram. + {{{ + uniq_frequency1 = GROUP hour_frequency2 BY group::ngram; + }}} + + * For each group, identify the hour in which this n-gram is used with a particularly high frequency. Call the !ScoreGenerator UDF to calculate a "popularity" score for the n-gram. + {{{ + uniq_frequency2 = FOREACH uniq_frequency1 GENERATE flatten($0), flatten(org.apache.pig.tutorial.ScoreGenerator($1)); + }}} + - * Uses the [http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data FOREACH-GENERATE] command to assign names to the fields. + * Use the [http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data FOREACH-GENERATE] command to assign names to the fields. + {{{ + uniq_frequency3 = FOREACH uniq_frequency2 GENERATE $1 as hour, $0 as ngram, $2 as score, $3 as count, $4 as mean; + }}} + - * Uses the [http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_ FILTER] command to move all records with a score less than or equal to 2.0. + * Use the [http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_ FILTER] command to move all records with a score less than or equal to 2.0. + {{{ + filtered_uniq_frequency = FILTER uniq_frequency3 BY score > 2.0; + }}} + - * Uses the [http://wiki.apache.org/pig/PigLatin#ORDER:_Sorting_data_according_to_some_field ORDER] command to sort the remaining records by hour and score. + * Use the [http://wiki.apache.org/pig/PigLatin#ORDER:_Sorting_data_according_to_some_field ORDER] command to sort the remaining records by hour and score. + {{{ + ordered_uniq_frequency = ORDER filtered_uniq_frequency BY (hour, score); + }}} + - * Uses the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to store the results. The output file contains a list of n-grams with the following fields: '''hour''', '''ngram''', '''score''', '''count''', '''mean''' + * Use the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to store the results. The output file contains a list of n-grams with the following fields: '''hour''', '''ngram''', '''score''', '''count''', '''mean''' Output: (hour, n-gram, score, count, average_counts_among_all_hours) - + {{{ + STORE ordered_uniq_frequency INTO '/tmp/tutorial-results' USING PigStorage(); + }}} [[Anchor(Tutorial_Join_Pig_Script)]] == Tutorial-Join Pig Script ==
