[ https://issues.apache.org/jira/browse/DATAFU-150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950069#comment-16950069 ]
Eyal Allweil edited comment on DATAFU-150 at 10/12/19 3:34 PM: --------------------------------------------------------------- [~russell.jurney], I was trying to understand what this does and found the [OneHotEncoder in Spark ML|https://org.apache.spark.ml.feature.OneHotEncoder]. Can you explain the difference between the two? (I'm afraid my data science skills aren't what they should be) was (Author: eyal): [~russell.jurney], I was trying to understand what this does and found the [OneHotEncoder in Spark ML|org.apache.spark.ml.feature.OneHotEncoder]. Can you explain the difference between the two? (I'm afraid my data science skills aren't what they should be) > Add MultiLabelOneHotEncoder > --------------------------- > > Key: DATAFU-150 > URL: https://issues.apache.org/jira/browse/DATAFU-150 > Project: DataFu > Issue Type: Improvement > Reporter: Russell Jurney > Assignee: Russell Jurney > Priority: Major > > I have created the following code in Python to one-hot encode multilabel data > and would like to add it to DataFu: > {code:java} > questions_tags = filtered_lists.map( > lambda x: Row( > _Body=x[0], > _Tags=x[1] > ) > ).toDF() > questions_tags.show() > # Create indexes for each multilabel tag > enumerated_labels = [ > z for z in enumerate( > sorted( > remaining_tags_df.rdd > .groupBy(lambda x: 1) > .flatMap(lambda x: [y.tag for y in x[1]]) > .collect() > ) > ) > ] > tag_index = {x: i for i, x in enumerated_labels} > index_tag = {i: x for i, x in enumerated_labels}# Explicitly free RAM > def one_hot_encode(tag_list, enumerated_labels): > """PySpark can't one-hot-encode multilabel data, so we do it ourselves.""" > one_hot_row = [] > for i, label in enumerated_labels: > if index_tag[i] in tag_list: > one_hot_row.append(1) > else: > one_hot_row.append(0) > assert(len(one_hot_row) == len(enumerated_labels)) > return one_hot_row > # Write the one-hot-encoded questions to S3 as a parquet file > one_hot_questions = questions_tags.rdd.map( > lambda x: Row( > _Body=x._Body, > _Tags=one_hot_encode(x._Tags, enumerated_labels) > ) > ) > # Create a DataFrame > schema = T.StructType([ > T.StructField("_Body", T.ArrayType( > T.StringType() > )), > T.StructField("_Tags", T.ArrayType( > T.IntegerType() > )) > ]) > one_hot_df = spark.createDataFrame( > one_hot_questions, > schema > ) > one_hot_df.show() > {code} > Which shows: > {code} > +--------------------+--------------------+ > | _Body| _Tags| > +--------------------+--------------------+ > |[Convert, Decimal...|[0, 0, 0, 0, 0, 0...| > |[Percentage, widt...|[0, 0, 0, 0, 0, 0...| > |[How, I, calculat...|[0, 1, 0, 0, 0, 0...| > |[Calculate, relat...|[0, 0, 0, 0, 0, 0...| > |[Determine, user,...|[0, 0, 0, 0, 0, 0...| > |[Difference, Math...|[0, 1, 0, 0, 0, 0...| > |[Filling, DataSet...|[0, 0, 1, 0, 0, 0...| > |[Binary, Data, My...|[0, 0, 0, 0, 0, 0...| > |[What, fastest, w...|[0, 0, 0, 0, 0, 0...| > |[Throw, error, My...|[0, 0, 0, 0, 0, 0...| > |[How, use, C, soc...|[0, 0, 0, 0, 0, 0...| > |[Unloading, ByteA...|[0, 0, 0, 0, 0, 0...| > |[Check, changes, ...|[0, 0, 0, 0, 0, 0...| > |[Reliable, timer,...|[0, 1, 0, 0, 0, 0...| > |[Best, way, allow...|[0, 0, 0, 0, 0, 0...| > |[Multiple, submit...|[0, 0, 0, 0, 0, 0...| > |[How, I, get, dis...|[0, 0, 1, 0, 0, 0...| > |[Paging, collecti...|[0, 0, 1, 0, 0, 0...| > |[How, I, add, exi...|[0, 0, 0, 0, 0, 0...| > |[Getting, Subclip...|[0, 0, 0, 0, 0, 0...| > +--------------------+--------------------+ > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)