from:"nzw0301"

[GitHub] incubator-hivemall pull request #140: Change the organization of nzw

2018-04-01 Thread nzw0301

GitHub user nzw0301 opened a pull request:

https://github.com/apache/incubator-hivemall/pull/140

Change the organization of nzw

## What changes were proposed in this pull request?

Update Kento Nozawa's organization.

## What type of PR is it?

[Documentation]

## What is the Jira issue?

This fix is a minor fix for release, so I have not yet created a jira 
ticket. Let me know if I should create it.

## How was this patch tested?

By compiling the source code, confirming this change does not break the 
structure of pom.xml.

## Checklist

(Please remove this section if not needed; check `x` for YES, blank for NO)

- [x] Did you apply source code formatter, i.e., `mvn formatter:format`, 
for your commit?
- [ ] Did you run system tests on Hive (or Spark)?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nzw0301/incubator-hivemall change-nzw-org

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hivemall/pull/140.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #140


commit a7c59bb78546c93abb8df1ad91867525f54c2b0c
Author: Kento NOZAWA <k_nzw@...>
Date:   2018-04-02T03:21:05Z

Change the organization of nzw




---

[GitHub] incubator-hivemall issue #116: [WIP][HIVEMALL-118] word2vec

2017-09-28 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/116
  
@myui I resolved conflicts.


---

[GitHub] incubator-hivemall pull request #116: [WIP][HIVEMALL-118] word2vec

2017-09-28 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/116#discussion_r141553131
  
--- Diff: core/src/main/java/hivemall/embedding/Word2VecUDTF.java ---
@@ -0,0 +1,364 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package hivemall.embedding;
+
+import hivemall.UDTFWithOptions;
+import hivemall.utils.collections.IMapIterator;
+import hivemall.utils.collections.maps.Int2FloatOpenHashTable;
+import hivemall.utils.collections.maps.OpenHashTable;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.lang.Primitives;
+
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.hadoop.hive.ql.exec.Description;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import 
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ListObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorUtils;
+import org.apache.hadoop.io.FloatWritable;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.io.Text;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import java.util.List;
+import java.util.Arrays;
+import java.util.ArrayList;
+
+@Description(
+name = "train_word2vec",
+value = "_FUNC_(array<array> negative_table, 
array doc [, const string options]) - Returns a prediction model")
+public class Word2VecUDTF extends UDTFWithOptions {
+protected transient AbstractWord2VecModel model;
+@Nonnegative
+private float startingLR;
+@Nonnegative
+private long numTrainWords;
+private OpenHashTable<String, Integer> word2index;
+
+@Nonnegative
+private int dim;
+@Nonnegative
+private int win;
+@Nonnegative
+private int neg;
+@Nonnegative
+private int iter;
+private boolean skipgram;
+private boolean isStringInput;
+
+private Int2FloatOpenHashTable S;
+private int[] aliasWordIds;
+
+private ListObjectInspector negativeTableOI;
+private ListObjectInspector negativeTableElementListOI;
+private PrimitiveObjectInspector negativeTableElementOI;
+
+private ListObjectInspector docOI;
+private PrimitiveObjectInspector wordOI;
+
+@Override
+public StructObjectInspector initialize(ObjectInspector[] argOIs) 
throws UDFArgumentException {
+final int numArgs = argOIs.length;
+
+if (numArgs != 3) {
+throw new UDFArgumentException(getClass().getSimpleName()
++ " takes 3 arguments:  [, constant string options]: "
++ Arrays.toString(argOIs));
+}
+
+processOptions(argOIs);
+
+this.negativeTableOI = HiveUtils.asListOI(argOIs[0]);
+this.negativeTableElementListOI = 
HiveUtils.asListOI(negativeTableOI.getListElementObjectInspector());
+this.docOI = HiveUtils.asListOI(argOIs[1]);
+
+this.isStringInput = HiveUtils.isStringListOI(argOIs[1]);
+
+if (isStringInput) {
+this.negativeTableElementOI = 
HiveUtils.asStringOI(negativeTableElementListOI.getListElementObjectInspector());
+this.wordOI = 
HiveUtils.asStringOI(docOI.getListElementObjectInspector());
+} else {
+

[GitHub] incubator-hivemall pull request #116: [WIP][HIVEMALL-118] word2vec

2017-09-28 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/116#discussion_r141551510
  
--- Diff: core/src/main/java/hivemall/embedding/AbstractWord2VecModel.java 
---
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package hivemall.embedding;
+
+import hivemall.math.random.PRNG;
+import hivemall.math.random.RandomNumberGeneratorFactory;
+import hivemall.utils.collections.maps.Int2FloatOpenHashTable;
+
+import javax.annotation.Nonnegative;
+import javax.annotation.Nonnull;
+import java.util.List;
+
+public abstract class AbstractWord2VecModel {
+// cached sigmoid function parameters
+protected static final int MAX_SIGMOID = 6;
+protected static final int SIGMOID_TABLE_SIZE = 1000;
+protected float[] sigmoidTable;
+
+
+@Nonnegative
+protected int dim;
+protected int win;
+protected int neg;
+protected int iter;
+
+// learning rate parameters
+@Nonnegative
+protected float lr;
+@Nonnegative
+private float startingLR;
+@Nonnegative
+private long numTrainWords;
+@Nonnegative
+protected long wordCount;
+@Nonnegative
+private long lastWordCount;
+
+protected PRNG rnd;
+
+protected Int2FloatOpenHashTable contextWeights;
+protected Int2FloatOpenHashTable inputWeights;
+protected Int2FloatOpenHashTable S;
+protected int[] aliasWordId;
+
+protected AbstractWord2VecModel(final int dim, final int win, final 
int neg, final int iter,
+final float startingLR, final long numTrainWords, final 
Int2FloatOpenHashTable S,
+final int[] aliasWordId) {
+this.win = win;
+this.neg = neg;
+this.iter = iter;
+this.dim = dim;
+this.startingLR = this.lr = startingLR;
+this.numTrainWords = numTrainWords;
+
+// alias sampler for negative sampling
+this.S = S;
+this.aliasWordId = aliasWordId;
+
+this.wordCount = 0L;
+this.lastWordCount = 0L;
+this.rnd = RandomNumberGeneratorFactory.createPRNG(1001);
+
+this.sigmoidTable = initSigmoidTable();
+
+// TODO how to estimate size
+this.inputWeights = new Int2FloatOpenHashTable(10578 * dim);
--- End diff --

There is no reason.


---

[GitHub] incubator-hivemall issue #117: [WIP][HIVEMALL-17-2] Revise SLIM implementati...

2017-09-27 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/117
  
@myui Thank you for your review and refactoring! 


---

[GitHub] incubator-hivemall issue #111: [HIVEMALL-17] Support SLIM

2017-09-13 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/111
  
@myui done.


---

[GitHub] incubator-hivemall issue #111: [HIVEMALL-17] Support SLIM

2017-09-13 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/111
  
@myui ok, please give me a time.


---

[GitHub] incubator-hivemall pull request #111: [HIVEMALL-17] Support SLIM

2017-09-12 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/111#discussion_r138523391
  
--- Diff: core/src/main/java/hivemall/recommend/SlimUDTF.java ---
@@ -0,0 +1,636 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package hivemall.recommend;
+
+
+import hivemall.UDTFWithOptions;
+import hivemall.annotations.VisibleForTesting;
+import hivemall.common.ConversionState;
+import hivemall.math.matrix.sparse.DoKMatrix;
+import hivemall.math.vector.VectorProcedure;
+import hivemall.utils.collections.maps.Int2FloatOpenHashTable;
+import hivemall.utils.hadoop.HiveUtils;
+import hivemall.utils.io.FileUtils;
+import hivemall.utils.io.NioStatefullSegment;
+import hivemall.utils.lang.NumberUtils;
+import hivemall.utils.lang.Primitives;
+import hivemall.utils.lang.SizeOf;
+import hivemall.utils.lang.mutable.MutableDouble;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+import org.apache.commons.logging.Log;
+import org.apache.commons.logging.LogFactory;
+import org.apache.hadoop.hive.ql.exec.Description;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.serde2.objectinspector.*;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.*;
+import org.apache.hadoop.io.DoubleWritable;
+import org.apache.hadoop.io.IntWritable;
+import org.apache.hadoop.mapred.Counters;
+import org.apache.hadoop.mapred.Reporter;
+
+import javax.annotation.Nonnull;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.util.*;
+
+
+@Description(
+name = "train_slim",
+value = "_FUNC_( int i, map<int, double> r_i, map<int, map<int, 
double>> topKRatesOfI, int j, map<int, double> r_j [, constant string options]) 
" +
+"- Returns row index, column index and non-zero weight 
value of prediction model")
+public class SlimUDTF extends UDTFWithOptions {
+private static final Log logger = LogFactory.getLog(SlimUDTF.class);
+
+private double l1;
+private double l2;
+private int numIterations;
+private int previousItemId;
+
+private transient DoKMatrix weightMatrix; // item-item weight matrix
+private transient DoKMatrix dataMatrix; // item-user matrix to get the 
number of nnz values in column
+
+private PrimitiveObjectInspector itemIOI;
+private PrimitiveObjectInspector itemJOI;
+private MapObjectInspector riOI;
+private MapObjectInspector rjOI;
+
+private MapObjectInspector knnItemsOI;
+private PrimitiveObjectInspector knnItemsKeyOI;
+private MapObjectInspector knnItemsValueOI;
+private PrimitiveObjectInspector knnItemsValueKeyOI;
+private PrimitiveObjectInspector knnItemsValueValueOI;
+
+private PrimitiveObjectInspector riKeyOI;
+private PrimitiveObjectInspector riValueOI;
+
+private PrimitiveObjectInspector rjKeyOI;
+private PrimitiveObjectInspector rjValueOI;
+
+// used to store KNN data into temporary file for iterative training
+private NioStatefullSegment fileIO;
+private ByteBuffer inputBuf;
+
+private ConversionState cvState;
+private long observedTrainingExamples;
+
+public SlimUDTF() {}
+
+@Override
+public StructObjectInspector initialize(ObjectInspector[] argOIs) 
throws UDFArgumentException {
+final int numArgs = argOIs.length;
+
+if (numArgs == 1 && HiveUtils.isStringOI(argOIs[0])) {
--- End diff --

I add this a few line to show the slim explanation without other arguments. 
But it may not be conventional way to show `help` for hivemall.


---

[GitHub] incubator-hivemall issue #111: [HIVEMALL-17] Support SLIM

2017-09-12 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/111
  
@myui @takuti If you have a time, could you review this PR?



---

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

2017-09-02 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/111#discussion_r136694290
  
--- Diff: 
core/src/main/java/hivemall/utils/collections/maps/Int2DoubleOpenHashTable.java 
---
@@ -0,0 +1,417 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package hivemall.utils.collections.maps;
+
+import hivemall.utils.math.Primes;
+
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.util.Arrays;
+
+/**
+ * An open-addressing hash table with double hashing
+ *
+ * @see http://en.wikipedia.org/wiki/Double_hashing
+ */
+public class Int2DoubleOpenHashTable implements Externalizable {
+protected static final byte FREE = 0;
+protected static final byte FULL = 1;
+protected static final byte REMOVED = 2;
+
+private static final float DEFAULT_LOAD_FACTOR = 0.7f;
+private static final float DEFAULT_GROW_FACTOR = 2.0f;
+
+protected final transient float _loadFactor;
+protected final transient float _growFactor;
+
+protected int _used = 0;
+protected int _threshold;
+protected double defaultReturnValue = -1.d;
+
+protected int[] _keys;
+protected double[] _values;
+protected byte[] _states;
+
+protected Int2DoubleOpenHashTable(int size, float loadFactor, float 
growFactor,
+boolean forcePrime) {
+if (size < 1) {
+throw new IllegalArgumentException();
+}
+this._loadFactor = loadFactor;
+this._growFactor = growFactor;
+int actualSize = forcePrime ? Primes.findLeastPrimeNumber(size) : 
size;
+this._keys = new int[actualSize];
+this._values = new double[actualSize];
+this._states = new byte[actualSize];
+this._threshold = (int) (actualSize * _loadFactor);
+}
+
+public Int2DoubleOpenHashTable(int size, float loadFactor, float 
growFactor) {
+this(size, loadFactor, growFactor, true);
+}
+
+public Int2DoubleOpenHashTable(int size) {
+this(size, DEFAULT_LOAD_FACTOR, DEFAULT_GROW_FACTOR, true);
+}
+
+/**
+ * Only for {@link Externalizable}
+ */
+public Int2DoubleOpenHashTable() {// required for serialization
+this._loadFactor = DEFAULT_LOAD_FACTOR;
+this._growFactor = DEFAULT_GROW_FACTOR;
+}
+
+public void defaultReturnValue(double v) {
+this.defaultReturnValue = v;
+}
+
+public boolean containsKey(int key) {
+return findKey(key) >= 0;
+}
+
+/**
+ * @return -1.d if not found
+ */
+public double get(int key) {
+int i = findKey(key);
+if (i < 0) {
+return defaultReturnValue;
+}
+return _values[i];
+}
+
+public double put(int key, double value) {
+int hash = keyHash(key);
+int keyLength = _keys.length;
+int keyIdx = hash % keyLength;
+
+boolean expanded = preAddEntry(keyIdx);
+if (expanded) {
+keyLength = _keys.length;
+keyIdx = hash % keyLength;
+}
+
+int[] keys = _keys;
+double[] values = _values;
+byte[] states = _states;
+
+if (states[keyIdx] == FULL) {// double hashing
+if (keys[keyIdx] == key) {
+double old = values[keyIdx];
+values[keyIdx] = value;
+return old;
+}
+// try second hash
+int decr = 1 + (hash % (keyLength - 2));
+for (;;) {
+keyIdx -= decr;
+if (keyIdx < 0) {
+

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

2017-09-01 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/111#discussion_r136633220
  
--- Diff: 
core/src/main/java/hivemall/utils/collections/maps/Int2DoubleOpenHashTable.java 
---
@@ -0,0 +1,417 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package hivemall.utils.collections.maps;
+
+import hivemall.utils.math.Primes;
+
+import java.io.Externalizable;
+import java.io.IOException;
+import java.io.ObjectInput;
+import java.io.ObjectOutput;
+import java.util.Arrays;
+
+/**
+ * An open-addressing hash table with double hashing
+ *
+ * @see http://en.wikipedia.org/wiki/Double_hashing
+ */
+public class Int2DoubleOpenHashTable implements Externalizable {
+protected static final byte FREE = 0;
+protected static final byte FULL = 1;
+protected static final byte REMOVED = 2;
+
+private static final float DEFAULT_LOAD_FACTOR = 0.7f;
+private static final float DEFAULT_GROW_FACTOR = 2.0f;
+
+protected final transient float _loadFactor;
+protected final transient float _growFactor;
+
+protected int _used = 0;
+protected int _threshold;
+protected double defaultReturnValue = -1.d;
+
+protected int[] _keys;
+protected double[] _values;
+protected byte[] _states;
+
+protected Int2DoubleOpenHashTable(int size, float loadFactor, float 
growFactor,
+boolean forcePrime) {
+if (size < 1) {
+throw new IllegalArgumentException();
+}
+this._loadFactor = loadFactor;
+this._growFactor = growFactor;
+int actualSize = forcePrime ? Primes.findLeastPrimeNumber(size) : 
size;
+this._keys = new int[actualSize];
+this._values = new double[actualSize];
+this._states = new byte[actualSize];
+this._threshold = (int) (actualSize * _loadFactor);
+}
+
+public Int2DoubleOpenHashTable(int size, float loadFactor, float 
growFactor) {
+this(size, loadFactor, growFactor, true);
+}
+
+public Int2DoubleOpenHashTable(int size) {
+this(size, DEFAULT_LOAD_FACTOR, DEFAULT_GROW_FACTOR, true);
+}
+
+/**
+ * Only for {@link Externalizable}
+ */
+public Int2DoubleOpenHashTable() {// required for serialization
+this._loadFactor = DEFAULT_LOAD_FACTOR;
+this._growFactor = DEFAULT_GROW_FACTOR;
+}
+
+public void defaultReturnValue(double v) {
+this.defaultReturnValue = v;
+}
+
+public boolean containsKey(int key) {
+return findKey(key) >= 0;
+}
+
+/**
+ * @return -1.d if not found
+ */
+public double get(int key) {
+int i = findKey(key);
+if (i < 0) {
+return defaultReturnValue;
+}
+return _values[i];
+}
+
+public double put(int key, double value) {
+int hash = keyHash(key);
+int keyLength = _keys.length;
+int keyIdx = hash % keyLength;
+
+boolean expanded = preAddEntry(keyIdx);
+if (expanded) {
+keyLength = _keys.length;
+keyIdx = hash % keyLength;
+}
+
+int[] keys = _keys;
+double[] values = _values;
+byte[] states = _states;
+
+if (states[keyIdx] == FULL) {// double hashing
+if (keys[keyIdx] == key) {
+double old = values[keyIdx];
+values[keyIdx] = value;
+return old;
+}
+// try second hash
+int decr = 1 + (hash % (keyLength - 2));
+for (;;) {
+keyIdx -= decr;
+if (keyIdx < 0) {
+

[GitHub] incubator-hivemall issue #111: [WIP][HIVEMALL-17] Support SLIM

2017-09-01 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/111
  
After a few checks, I'll remove [WIP] tag in this PR.

- whether hive input KNN data can be read from tem file
- whether loss function correctly decreases


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

2017-08-31 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/111#discussion_r136500350
  
--- Diff: docs/gitbook/recommend/movielens_cf.md ---
@@ -61,7 +61,8 @@ partial_result as ( -- launch DIMSUM in a MapReduce 
fashion
 movie_features f
   left outer join movie_magnitude m
 ),
-similarity as ( -- reduce (i.e., sum up) mappers' partial results
+similarity as (
+-- reduce (i.e., sum up) mappers' partial results 
--- End diff --

If the number of `'` is even in comment, `'` can be used in the same line:

```
> select (1 --'sample'
> );
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

2017-08-30 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/111#discussion_r136113122
  
--- Diff: core/src/main/java/hivemall/recommend/SlimUDTF.java ---
@@ -217,13 +347,48 @@ private void train(int i, Map Ri, Map 
topKRatesOfI, int j, Map
 double eui = rui - predict(u, i, topKRatesOfI, j);
 gradSum += ruj * eui;
 rateSum += ruj * ruj;
-errs += eui * eui;
+
+if (this.numIterations > 1){
+this.A.unsafeSet((int) u, j, ruj); // need optimize
+}
 }
 
 gradSum /= N;
 rateSum /= N;
-errs /= N;
 
+this.W.unsafeSet(i, j, getUpdateTerm(gradSum, rateSum));
+}
+
+
+//private void train(int i, Map Ri, Map topKRatesOfI, int 
j, Map Rj) {
+//int N = Rj.size();
+//double gradSum = 0.d;
+//double rateSum = 0.d;
+//double errs = 0.d;
+//for (Map.Entry userRate : Rj.entrySet()) {
+//Object u = userRate.getKey();
+//double ruj = 
PrimitiveObjectInspectorUtils.getDouble(userRate.getValue(),
+//this.itemJRateValueOI);
+//double rui = 0.d;
+//if (Ri.containsKey(u)) {
+//rui = PrimitiveObjectInspectorUtils.getDouble(Ri.get(u), 
this.itemIRateValueOI);
+//}
+//
+//double eui = rui - predict(u, i, topKRatesOfI, j);
+//gradSum += ruj * eui;
+//rateSum += ruj * ruj;
+//errs += eui * eui;
+//}
+//
+//gradSum /= N;
+//rateSum /= N;
+//errs /= N;
+//
+//this.loss += errs;
+//this.W.unsafeSet(i, j, getUpdateTerm(gradSum, rateSum));
+//}
+
+private double getUpdateTerm(double gradSum, double rateSum){
--- End diff --

Sorry, I missed an error related this definition.
This function refers class variables `l1` and `l2`, so I will change it to 
`getUpdateTerm(final double gradSum, final double rateSum, final double l1, 
final double l2)`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

2017-08-30 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/111#discussion_r136111887
  
--- Diff: core/src/main/java/hivemall/math/matrix/sparse/DoKMatrix.java ---
@@ -309,6 +309,20 @@ public void eachNonZeroInColumn(@Nonnegative final int 
col,
 }
 }
 
+public void eachNonZeroCell(@Nonnull final VectorProcedure procedure) {
+if (nnz == 0) {
+return;
+}
+final IMapIterator itor = elements.entries();
+while (itor.next() != -1) {
+long k = itor.getKey();
+int row = Primitives.getHigh(k);
+int col = Primitives.getLow(k);
+double value = itor.getValue();
+procedure.apply(row, col, value);
--- End diff --

Oop, Thanks! 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

2017-08-29 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/107
  
The issue above is avoided by creating table:

```
create table data as (
  select 1 as truth, 1 as predicted
);
```

```
// ok
select
  fmeasure(array(truth), array(predicted))
from data
;

// ok
select
  f1score(array(truth), array(predicted))
from data
;
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

2017-08-28 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/107
  
But I found another issue for f1score and fmeasure.

Both function cannot work on `EMR v5.8.0`

```sql
hive> select f1score(array(1), array(1));
FAILED: IllegalArgumentException Size requested for unknown type: 
org.apache.hadoop.hive.ql.exec.UDAFEvaluator
```

```sql
hive> select fmeasure(array(1), array(1));
FAILED: IllegalArgumentException Size requested for unknown type: 
java.lang.String
```

However, they can work on `EMR v5.0.0`.
I don't know why the failures occur on newer EMR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

2017-08-28 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/107
  
I tested @takuti's query. The buggy code (previous code) returns `1.0`.
On the other hand, the fixed code return correct value`0.5`.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

2017-08-28 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/107
  
I will check whether the return value is the same tomorrow.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

2017-08-28 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/107#discussion_r135471771
  
--- Diff: docs/gitbook/eval/binary_classification_measures.md ---
@@ -0,0 +1,261 @@
+
+
+
+
+# Binary problems
+
+Binary classification problem is the task to predict the label of each 
data given two categorized dataset.
+
+Hivemall provides some tutorials to deal with binary classification 
problems as follows:
+
+- [Online advertisement click prediction](../binaryclass/general.html)
+- [News classification](../binaryclass/news20_dataset.html)
+
+This page focuses on the evaluation of the results from such binary 
classification problems.
+If your classifier outputs probability rather than 0/1 label, evaluation 
based on [Area Under the ROC Curve](./auc.md) would be more appropriate.
+
+
+# Example
+
+For the metrics explanation, this page introduces toy example data and two 
metrics.
+
+## Data
+
+The following table shows the sample of binary classification's prediction.
+In this case, `1` means positive label and `0` means negative label.
+Left column includes supervised label data,
+and center column includes predicted label by a binary classifier.
+
+| truth label| predicted label | |
+|:---:|:---:|:---:|
+| 1 | 0 |False Negative|
+| 0 | 1 |False Positive|
+| 0 | 0 |True Negative|
+| 1 | 1 |True Positive|
+| 0 | 1 |False Positive|
+| 0 | 0 |True Negative|
+
+## Preliminary metrics
+
+Some evaluation metrics are calculated based on 4 values:
+
+- True Positive (TP): truth label is positive and predicted label is also 
positive
+- True Negative (TN): truth label is negative and predicted label is also 
negative
+- False Positive (FP): truth label is negative but predicted label is 
positive
+- False Negative (FN): truth label is positive but predicted label is 
negative
+
+`TR` and `TN` represent correct classification, and `FP` and `FN` 
illustrate incorrect ones.
+
+In this example, we can obtain those values:
+
+- TP: 1
+- TN: 2
+- FP: 2
+- FN: 1
+
+if you want to know about those metrics, Wikipedia provides [more detail 
information](https://en.wikipedia.org/wiki/Sensitivity_and_specificity).
+
+### Recall
+
+Recall indicates the true positive rate in truth positive labels.
+The value is computed by the following equation:
+
+$$
+\mathrm{recall} = \frac{\mathrm{\#TP}}{\mathrm{\#TP} + \mathrm{\#FN}}
+$$
+
+In the previous example, $$\mathrm{precision} = \frac{1}{2}$$.
+
+### Precision
+
+Precision indicates the true positive rate in positive predictive labels.
+The value is computed by the following equation:
+
+$$
+\mathrm{precision} = \frac{\mathrm{\#TP}}{\mathrm{\#TP} + \mathrm{\#FP}}
+$$
+
+In the previous example, $$\mathrm{precision} = \frac{1}{3}$$.
+
+# Metrics
+
+## F1-score
+
+F1-score is the harmonic mean of recall and precision.
+F1-score is computed by the following equation:
+
+$$
+\mathrm{F}_1 = 2 \frac{\mathrm{precision} * 
\mathrm{recall}}{\mathrm{precision} + \mathrm{recall}}
+$$
+
+Hivemall's `fmeasure` function provides the option which can switch 
`micro`(default) or `binary` by passing `average` argument.
+
+
+>  Caution
+> Hivemall also provides `f1score` function, but it is old function to 
obtain F1-score. The value of `f1score` is based on set operation. So, we 
recommend to use `fmeasure` function to get F1-score based on this article.
+
+You can learn more about this from the following external resource:
+
+- [scikit-learn's 
F1-score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html)
+
+
+### Micro average
+
+If `micro` is passed to `average`, 
+recall and precision are modified to consider True Negative.
+So, micro f1score are calculated by those modified recall and precision.
+
+$$
+\mathrm{recall} = \frac{\mathrm{\#TP} + \mathrm{\#TN}}{\mathrm{\#TP} + 
\mathrm{\#FN} + \mathrm{\#TN}}
+$$
+
+$$
+\mathrm{precision} = \frac{\mathrm{\#TP} + \mathrm{\#TN}}{\mathrm{\#TP} + 
\mathrm{\#FP} + \mathrm{\#TN}}
+$$
+
+If `average` argument is omitted, `fmeasure` use default value: `'-average 
micro'`.
+
+The following query shows the example to obtain F1-score.
+Each row value has the same type (`int` or `boolean`).
+If row value's type is `int`, `1` is considered as the positive label, and 
`-1` or `0` is considered as the negative label.
+
+
+```sql
+WITH data as (
+  select 1 as truth, 0 as predicted
+union all
+  select 0 as truth, 1 as predicted
+union all
+  select 0 as tr

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

2017-08-28 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/107#discussion_r135464601
  
--- Diff: core/src/test/java/hivemall/evaluation/FMeasureUDAFTest.java ---
@@ -0,0 +1,393 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package hivemall.evaluation;
+
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
+import 
org.apache.hadoop.hive.ql.udf.generic.SimpleGenericUDAFParameterInfo;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import 
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils;
+import 
org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.List;
+
+
+public class FMeasureUDAFTest {
+FMeasureUDAF fmeasure;
+GenericUDAFEvaluator evaluator;
+ObjectInspector[] inputOIs;
+FMeasureUDAF.FMeasureAggregationBuffer agg;
+
+@Before
+public void setUp() throws Exception {
+fmeasure = new FMeasureUDAF();
+inputOIs = new ObjectInspector[] {
+
ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.writableLongObjectInspector),
+
ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.writableLongObjectInspector),
+ObjectInspectorUtils.getConstantObjectInspector(
+
PrimitiveObjectInspectorFactory.javaStringObjectInspector, "-beta 1.")};
+
+evaluator = fmeasure.getEvaluator(new 
SimpleGenericUDAFParameterInfo(inputOIs, false, false));
+
+agg = (FMeasureUDAF.FMeasureAggregationBuffer) 
evaluator.getNewAggregationBuffer();
+}
+
+private void setUpWithArguments(double beta, String average) throws 
Exception {
+fmeasure = new FMeasureUDAF();
+inputOIs = new ObjectInspector[] {
+
ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.writableLongObjectInspector),
+
ObjectInspectorFactory.getStandardListObjectInspector(PrimitiveObjectInspectorFactory.writableLongObjectInspector),
+ObjectInspectorUtils.getConstantObjectInspector(
+
PrimitiveObjectInspectorFactory.javaStringObjectInspector, "-beta " + beta
++ " -average " + average)};
+
+evaluator = fmeasure.getEvaluator(new 
SimpleGenericUDAFParameterInfo(inputOIs, false, false));
+agg = (FMeasureUDAF.FMeasureAggregationBuffer) 
evaluator.getNewAggregationBuffer();
+}
+
+private void binarySetUp(Object actual, Object predicted, double beta, 
String average)
+throws Exception {
+fmeasure = new FMeasureUDAF();
+inputOIs = new ObjectInspector[3];
+
+String actualClassName = actual.getClass().getName();
+if (actualClassName.equals("java.lang.Integer")) {
+inputOIs[0] = 
PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(PrimitiveObjectInspector.PrimitiveCategory.INT);
+} else if (actualClassName.equals("java.lang.Boolean")) {
+inputOIs[0] = 
PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(PrimitiveObjectInspector.PrimitiveCategory.BOOLEAN);
+} else if ((actualClassName.equals("java.lang.String"))) {
+inputOIs[0] = 
PrimitiveObjectInspectorFactory.getPrimitiveJavaObjectInspector(PrimitiveObjectInspector.PrimitiveCategory.STRING);
+

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

2017-08-28 Thread nzw0301

GitHub user nzw0301 opened a pull request:

https://github.com/apache/incubator-hivemall/pull/111

[WIP][HIVEMALL-17] Support SLIM

## What changes were proposed in this pull request?

Add new UDTF: `train_slim`

## What type of PR is it?

Improvement

## What is the Jira issue?

https://issues.apache.org/jira/browse/HIVEMALL-17

## How was this patch tested?

- Add `recommend/SlimUDTFTest`
- manual tests on EMR

## How to use this feature?

- Please see markdown file: `movielens_slim.md` in this PR.

## Checklist

- [x] Did you apply source code formatter, i.e., `mvn formatter:format`, 
for your commit?


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nzw0301/incubator-hivemall HIVEMALL-17

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-hivemall/pull/111.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #111


commit a9b89656ef4b54cb938e59e3bb546a2aa33a9fd3
Author: Kento NOZAWA <k_...@klis.tsukuba.ac.jp>
Date:   2017-08-18T03:30:14Z

Init Slim UDTF

commit 5728499405400d68faec2be58cd11e788ce8c607
Author: Kento NOZAWA <k_...@klis.tsukuba.ac.jp>
Date:   2017-08-25T07:15:18Z

Move comment

commit 7ac96304f06b41c0a72c1be885cd66d75acfd966
Author: Kento NOZAWA <k_...@klis.tsukuba.ac.jp>
Date:   2017-08-25T09:58:27Z

Init slim document

commit 23672cf5ddeceb06cad10e9a0496d860d786d601
Author: Kento NOZAWA <k_...@klis.tsukuba.ac.jp>
Date:   2017-08-28T07:22:39Z

Update slim

- Update docs
- Remove degub print in test
- Rename function for unification: `slim_train` to `train_slim`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

2017-08-27 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/107
  
@takuti @myui Thank you for your kind comments. 
I completed update based on reviews.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

2017-08-27 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/107#discussion_r135444092
  
--- Diff: docs/gitbook/eval/multilabel_classification_measures.md ---
@@ -0,0 +1,144 @@
+
+
+
+
+# Multi-label classification
+
+
+Multi-label classification problem is the task to predict the labels given 
categorized dataset.
+Each sample $$i$$ has $$l_i$$ labels, where $$L$$ is the number of unique 
labels in the dataset, and $$0 \leq  l_i \leq |L| $$.
--- End diff --

Yes, I fixed it. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

2017-08-27 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/107#discussion_r135432462
  
--- Diff: core/src/main/java/hivemall/evaluation/FMeasureUDAF.java ---
@@ -18,118 +18,387 @@
  */
 package hivemall.evaluation;
 
-import hivemall.utils.hadoop.WritableUtils;
+import hivemall.UDAFEvaluatorWithOptions;
+import hivemall.utils.hadoop.HiveUtils;
 
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 
+import hivemall.utils.lang.Primitives;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+
 import org.apache.hadoop.hive.ql.exec.Description;
-import org.apache.hadoop.hive.ql.exec.UDAF;
-import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
 import org.apache.hadoop.hive.serde2.io.DoubleWritable;
-import org.apache.hadoop.io.IntWritable;
-
-@SuppressWarnings("deprecation")
-@Description(name = "f1score",
-value = "_FUNC_(array[int], array[int]) - Return a F-measure/F1 
score")
-public final class FMeasureUDAF extends UDAF {
-
-public static class Evaluator implements UDAFEvaluator {
-
-public static class PartialResult {
-long tp;
-/** tp + fn */
-long totalAcutal;
-/** tp + fp */
-long totalPredicted;
-
-PartialResult() {
-this.tp = 0L;
-this.totalPredicted = 0L;
-this.totalAcutal = 0L;
-}
+import org.apache.hadoop.hive.serde2.objectinspector.*;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.IntObjectInspector;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.io.LongWritable;
 
-void updateScore(final List actual, final 
List predicted) {
-final int numActual = actual.size();
-final int numPredicted = predicted.size();
-int countTp = 0;
-for (int i = 0; i < numPredicted; i++) {
-IntWritable p = predicted.get(i);
-if (actual.contains(p)) {
-countTp++;
-}
+import javax.annotation.Nonnull;
+
+@Description(
+name = "fmeasure",
+value = "_FUNC_(array | int | boolean, array | int | boolean, 
String) - Return a F-measure (f1score is the special with beta=1.)")
+public final class FMeasureUDAF extends AbstractGenericUDAFResolver {
+@Override
+public GenericUDAFEvaluator getEvaluator(@Nonnull TypeInfo[] typeInfo) 
throws SemanticException {
+if (typeInfo.length != 2 && typeInfo.length != 3) {
+throw new UDFArgumentTypeException(typeInfo.length - 1,
+"_FUNC_ takes two or three arguments");
+}
+
+boolean isArg1ListOrIntOrBoolean = 
HiveUtils.isListTypeInfo(typeInfo[0])
+|| HiveUtils.isIntegerTypeInfo(typeInfo[0])
+|| HiveUtils.isBooleanTypeInfo(typeInfo[0]);
+if (!isArg1ListOrIntOrBoolean) {
+throw new UDFArgumentTypeException(0,
+"The first argument `array/int/boolean actual` is invalid 
form: " + typeInfo[0]);
+}
+
+boolean isArg2ListOrIntOrBoolean = 
HiveUtils.isListTypeInfo(typeInfo[1])
+|| HiveUtils.isIntegerTypeInfo(typeInfo[1])
+|| HiveUtils.isBooleanTypeInfo(typeInfo[1]);
+if (!isArg2ListOrIntOrBoolean) {
+throw new UDFArgumentTypeException(1,
+"The first argument `array/int/boolean actual` is invalid 
form: " + typeInfo[1]);
+}
+
+if (typeInfo[0] != typeInfo[1]) {
+throw new UDFArgumentTypeException(1, "The first argument's 
`actual` type is "
++ typeInfo[0] + ", but the second argument 
`predicated`'s type is not match: "
++ typeInfo[1]);
+

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

2017-08-25 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/107#discussion_r135227946
  
--- Diff: core/src/main/java/hivemall/evaluation/FMeasureUDAF.java ---
@@ -18,118 +18,387 @@
  */
 package hivemall.evaluation;
 
-import hivemall.utils.hadoop.WritableUtils;
+import hivemall.UDAFEvaluatorWithOptions;
+import hivemall.utils.hadoop.HiveUtils;
 
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 
+import hivemall.utils.lang.Primitives;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.Options;
+
 import org.apache.hadoop.hive.ql.exec.Description;
-import org.apache.hadoop.hive.ql.exec.UDAF;
-import org.apache.hadoop.hive.ql.exec.UDAFEvaluator;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException;
+import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.udf.generic.AbstractGenericUDAFResolver;
+import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator;
 import org.apache.hadoop.hive.serde2.io.DoubleWritable;
-import org.apache.hadoop.io.IntWritable;
-
-@SuppressWarnings("deprecation")
-@Description(name = "f1score",
-value = "_FUNC_(array[int], array[int]) - Return a F-measure/F1 
score")
-public final class FMeasureUDAF extends UDAF {
-
-public static class Evaluator implements UDAFEvaluator {
-
-public static class PartialResult {
-long tp;
-/** tp + fn */
-long totalAcutal;
-/** tp + fp */
-long totalPredicted;
-
-PartialResult() {
-this.tp = 0L;
-this.totalPredicted = 0L;
-this.totalAcutal = 0L;
-}
+import org.apache.hadoop.hive.serde2.objectinspector.*;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.BooleanObjectInspector;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+import 
org.apache.hadoop.hive.serde2.objectinspector.primitive.IntObjectInspector;
+import org.apache.hadoop.hive.serde2.typeinfo.TypeInfo;
+import org.apache.hadoop.io.LongWritable;
 
-void updateScore(final List actual, final 
List predicted) {
-final int numActual = actual.size();
-final int numPredicted = predicted.size();
-int countTp = 0;
-for (int i = 0; i < numPredicted; i++) {
-IntWritable p = predicted.get(i);
-if (actual.contains(p)) {
-countTp++;
-}
+import javax.annotation.Nonnull;
+
+@Description(
+name = "fmeasure",
+value = "_FUNC_(array | int | boolean, array | int | boolean, 
String) - Return a F-measure (f1score is the special with beta=1.)")
+public final class FMeasureUDAF extends AbstractGenericUDAFResolver {
+@Override
+public GenericUDAFEvaluator getEvaluator(@Nonnull TypeInfo[] typeInfo) 
throws SemanticException {
+if (typeInfo.length != 2 && typeInfo.length != 3) {
+throw new UDFArgumentTypeException(typeInfo.length - 1,
+"_FUNC_ takes two or three arguments");
+}
+
+boolean isArg1ListOrIntOrBoolean = 
HiveUtils.isListTypeInfo(typeInfo[0])
+|| HiveUtils.isIntegerTypeInfo(typeInfo[0])
+|| HiveUtils.isBooleanTypeInfo(typeInfo[0]);
+if (!isArg1ListOrIntOrBoolean) {
+throw new UDFArgumentTypeException(0,
+"The first argument `array/int/boolean actual` is invalid 
form: " + typeInfo[0]);
+}
+
+boolean isArg2ListOrIntOrBoolean = 
HiveUtils.isListTypeInfo(typeInfo[1])
+|| HiveUtils.isIntegerTypeInfo(typeInfo[1])
+|| HiveUtils.isBooleanTypeInfo(typeInfo[1]);
+if (!isArg2ListOrIntOrBoolean) {
+throw new UDFArgumentTypeException(1,
+"The first argument `array/int/boolean actual` is invalid 
form: " + typeInfo[1]);
+}
+
+if (typeInfo[0] != typeInfo[1]) {
+throw new UDFArgumentTypeException(1, "The first argument's 
`actual` type is "
++ typeInfo[0] + ", but the second argument 
`predicated`'s type is not match: "
++ typeInfo[1]);
+

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

2017-08-21 Thread nzw0301

Github user nzw0301 commented on the issue:

https://github.com/apache/incubator-hivemall/pull/107
  
@takuti Thank you for your useful review.
I will fix this PR based on your comment.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #107: [WIP][HIVEMALL-132] Generalize f1score...

2017-08-06 Thread nzw0301

Github user nzw0301 commented on a diff in the pull request:

https://github.com/apache/incubator-hivemall/pull/107#discussion_r131559560
  
--- Diff: docs/gitbook/eval/multilabel_classification_measures.md ---
@@ -0,0 +1,148 @@
+
+
+
+
+# Multi-label classification
+
+
+Multi-label classification problem is predicting the labels given 
categorized dataset.
+Each sample $$i$$ has $$l_i$$ labels ($$0 \leq  l_i \leq |L| $$  )
+, where $$L$$ is the number of unique labels in the geven dataset.
+
+This page focuses on evaluation of the results from such Multi-label 
classification problems.
+
+# Examples
+
+For the metrics explanation, this page introduces toy example data and two 
metrics.
+
+## Data
+
+The following table shows the sample of Multi-label classification's 
prediction.
+Animal names represent the tags of blog post.
+Left column includes supervised labels,
+Right column includes are predicted labels by a Multi-label classifier.
+
+| truth labels| predicted labels |
+|:---:|:---:|
+|cat, dog | cat, bird |
+| cat, bird | cat, dog |
+| | cat |
+| bird | bird |
+| bird, cat | bird, cat |
+| cat, dog, bird | cat, dog |
+| dog | dog, bird|
+
+
+# Evaluation metrics for multi-label classification
+
+Hivemall provises micro F1-score and micro F-measure.
+
+Given $$N$$ blog posts, we uses 
+
+Define $$L$$ is the set of the tag of blog posts, and 
+$$l_i$$ is a tag set of $$i$$th document.
+In the same manner,
+$$p_i$$ is a predicted tag set of $$i$$th document.
+
+
+
+## Micro F1-score
+
+
+F1-score is the harmonic mean of recall and precision.
+
+The value is computed by the following equation:
+
+$$
+\mathrm{F}_1 = 2 \frac
+{\sum_i |l_i \cap p_i |}
+{ 2* \sum_i |l_i \cap p_i | + \sum_i |l_i - p_i | + \sum_i |p_i - l_i | }
+$$
+
+The Following query shows the example to obtain F1-score.
+
+```sql
+WITH data as (
+  select array("cat", "dog") as actual, array("cat", "bird") as 
predicted
+union all
+  select array("cat", "bird")as actual, array("cat", "dog")  as 
predicted
+union all
+  select array() as actual, array("cat") as 
predicted
+union all
+  select array("bird")   as actual, array("bird")as 
predicted
+union all
+  select array("bird", "cat")as actual, array("bird", "cat") as 
predicted
+union all
+  select array("cat", "dog", "bird") as actual, array("cat", "dog")  as 
predicted
+union all
+  select array("dog")as actual, array("dog", "bird") as 
predicted
+)
+select
+  f1score(actual, predicted)
--- End diff --

Thank you for your review.
OK, I will update arguments.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-hivemall pull request #140: Change the organization of nzw

[GitHub] incubator-hivemall issue #116: [WIP][HIVEMALL-118] word2vec

[GitHub] incubator-hivemall pull request #116: [WIP][HIVEMALL-118] word2vec

[GitHub] incubator-hivemall pull request #116: [WIP][HIVEMALL-118] word2vec

[GitHub] incubator-hivemall issue #117: [WIP][HIVEMALL-17-2] Revise SLIM implementati...

[GitHub] incubator-hivemall issue #111: [HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall issue #111: [HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall pull request #111: [HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall issue #111: [HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall issue #111: [WIP][HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

[GitHub] incubator-hivemall pull request #111: [WIP][HIVEMALL-17] Support SLIM

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

[GitHub] incubator-hivemall pull request #107: [HIVEMALL-132] Generalize f1score UDAF...

[GitHub] incubator-hivemall issue #107: [HIVEMALL-132] Generalize f1score UDAF to sup...

[GitHub] incubator-hivemall pull request #107: [WIP][HIVEMALL-132] Generalize f1score...

28 matches

Site Navigation

Mail list logo

Footer information