spark git commit: [SPARK-16117][MLLIB] hide LibSVMFileFormat and move its doc to LibSVMDataSource

2016-06-21 Thread rxin 
Repository: spark
Updated Branches:
  refs/heads/master dbfdae4e4 -> f4e8c31ad


[SPARK-16117][MLLIB] hide LibSVMFileFormat and move its doc to LibSVMDataSource

## What changes were proposed in this pull request?

LibSVMFileFormat implements data source for LIBSVM format. However, users do 
not really need to call its APIs to use it. So we should hide it in the public 
API docs. The main issue is that we still need to put the documentation and 
example code somewhere. The proposal it to have a dummy class to hold the 
documentation, as a workaround to https://issues.scala-lang.org/browse/SI-8124.

## How was this patch tested?

Manually checked the generated API doc and tested loading LIBSVM data.

Author: Xiangrui Meng 

Closes #13819 from mengxr/SPARK-16117.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f4e8c31a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f4e8c31a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f4e8c31a

Branch: refs/heads/master
Commit: f4e8c31adf45af05751e0d77aefb5cacc58375ee
Parents: dbfdae4
Author: Xiangrui Meng 
Authored: Tue Jun 21 15:46:14 2016 -0700
Committer: Reynold Xin 
Committed: Tue Jun 21 15:46:14 2016 -0700

--
 .../ml/source/libsvm/LibSVMDataSource.scala | 56 
 .../spark/ml/source/libsvm/LibSVMRelation.scala | 41 ++
 2 files changed, 59 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/f4e8c31a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
--
diff --git 
a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala 
b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
new file mode 100644
index 000..73d8130
--- /dev/null
+++ 
b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.source.libsvm
+
+import org.apache.spark.ml.linalg.Vector
+import org.apache.spark.sql.{DataFrame, DataFrameReader}
+
+/**
+ * `libsvm` package implements Spark SQL data source API for loading LIBSVM 
data as [[DataFrame]].
+ * The loaded [[DataFrame]] has two columns: `label` containing labels stored 
as doubles and
+ * `features` containing feature vectors stored as [[Vector]]s.
+ *
+ * To use LIBSVM data source, you need to set "libsvm" as the format in 
[[DataFrameReader]] and
+ * optionally specify options, for example:
+ * {{{
+ *   // Scala
+ *   val df = spark.read.format("libsvm")
+ * .option("numFeatures", "780")
+ * .load("data/mllib/sample_libsvm_data.txt")
+ *
+ *   // Java
+ *   Dataset df = spark.read().format("libsvm")
+ * .option("numFeatures, "780")
+ * .load("data/mllib/sample_libsvm_data.txt");
+ * }}}
+ *
+ * LIBSVM data source supports the following options:
+ *  - "numFeatures": number of features.
+ *If unspecified or nonpositive, the number of features will be determined 
automatically at the
+ *cost of one additional pass.
+ *This is also useful when the dataset is already split into multiple 
files and you want to load
+ *them separately, because some features may not present in certain files, 
which leads to
+ *inconsistent feature dimensions.
+ *  - "vectorType": feature vector type, "sparse" (default) or "dense".
+ *
+ * Note that this class is public for documentation purpose. Please don't use 
this class directly.
+ * Rather, use the data source API as illustrated above.
+ *
+ * @see [[https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/ LIBSVM 
datasets]]
+ */
+class LibSVMDataSource private() {}

http://git-wip-us.apache.org/repos/asf/spark/blob/f4e8c31a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMRelation.scala
--
diff --git

spark git commit: [SPARK-16117][MLLIB] hide LibSVMFileFormat and move its doc to LibSVMDataSource

2016-06-21 Thread rxin 
Repository: spark
Updated Branches:
  refs/heads/branch-2.0 aeda9a153 -> 8d5a62d5f


[SPARK-16117][MLLIB] hide LibSVMFileFormat and move its doc to LibSVMDataSource

## What changes were proposed in this pull request?

LibSVMFileFormat implements data source for LIBSVM format. However, users do 
not really need to call its APIs to use it. So we should hide it in the public 
API docs. The main issue is that we still need to put the documentation and 
example code somewhere. The proposal it to have a dummy class to hold the 
documentation, as a workaround to https://issues.scala-lang.org/browse/SI-8124.

## How was this patch tested?

Manually checked the generated API doc and tested loading LIBSVM data.

Author: Xiangrui Meng 

Closes #13819 from mengxr/SPARK-16117.

(cherry picked from commit f4e8c31adf45af05751e0d77aefb5cacc58375ee)
Signed-off-by: Reynold Xin 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8d5a62d5
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8d5a62d5
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8d5a62d5

Branch: refs/heads/branch-2.0
Commit: 8d5a62d5f8ee9a2f4643075efdbee9822e85d141
Parents: aeda9a1
Author: Xiangrui Meng 
Authored: Tue Jun 21 15:46:14 2016 -0700
Committer: Reynold Xin 
Committed: Tue Jun 21 15:46:19 2016 -0700

--
 .../ml/source/libsvm/LibSVMDataSource.scala | 56 
 .../spark/ml/source/libsvm/LibSVMRelation.scala | 41 ++
 2 files changed, 59 insertions(+), 38 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/8d5a62d5/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
--
diff --git 
a/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala 
b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
new file mode 100644
index 000..73d8130
--- /dev/null
+++ 
b/mllib/src/main/scala/org/apache/spark/ml/source/libsvm/LibSVMDataSource.scala
@@ -0,0 +1,56 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.source.libsvm
+
+import org.apache.spark.ml.linalg.Vector
+import org.apache.spark.sql.{DataFrame, DataFrameReader}
+
+/**
+ * `libsvm` package implements Spark SQL data source API for loading LIBSVM 
data as [[DataFrame]].
+ * The loaded [[DataFrame]] has two columns: `label` containing labels stored 
as doubles and
+ * `features` containing feature vectors stored as [[Vector]]s.
+ *
+ * To use LIBSVM data source, you need to set "libsvm" as the format in 
[[DataFrameReader]] and
+ * optionally specify options, for example:
+ * {{{
+ *   // Scala
+ *   val df = spark.read.format("libsvm")
+ * .option("numFeatures", "780")
+ * .load("data/mllib/sample_libsvm_data.txt")
+ *
+ *   // Java
+ *   Dataset df = spark.read().format("libsvm")
+ * .option("numFeatures, "780")
+ * .load("data/mllib/sample_libsvm_data.txt");
+ * }}}
+ *
+ * LIBSVM data source supports the following options:
+ *  - "numFeatures": number of features.
+ *If unspecified or nonpositive, the number of features will be determined 
automatically at the
+ *cost of one additional pass.
+ *This is also useful when the dataset is already split into multiple 
files and you want to load
+ *them separately, because some features may not present in certain files, 
which leads to
+ *inconsistent feature dimensions.
+ *  - "vectorType": feature vector type, "sparse" (default) or "dense".
+ *
+ * Note that this class is public for documentation purpose. Please don't use 
this class directly.
+ * Rather, use the data source API as illustrated above.
+ *
+ * @see [[https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/ LIBSVM 
datasets]]
+ */
+class LibSVMDataSource private() {}