[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140647785 [Test build #42526 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42526/consoleFull) for PR 8740 at commit [`392baf0`](https://github.com/apache/spark/commit/392baf044d4f29bbdf2e40d76fd5e53baa8a862c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140646066 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140646082 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user vinodkc commented on the pull request: https://github.com/apache/spark/pull/8682#issuecomment-140645936 Sure I'll work on SPARK-10631 Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...
Github user tnachen commented on the pull request: https://github.com/apache/spark/pull/8349#issuecomment-140645786 This is only recently merged so this is not yet released, so Mesosphere DCOS won't able to support Python yet. And if you wan tot provide s3 you just need to give it a s3://./x.py prefix --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...
Github user viesti commented on the pull request: https://github.com/apache/spark/pull/8349#issuecomment-140644766 So gave mesosphere a go (neat that there is a cloudformation template for that :)), but didn't find a way to tell how to transfer my local program onto the cluster since in the submission request that get's sent, the python file points to a local path i.e.: ```"appResource" : "file:/Users/xxx/programming/yyy/spark/test.py"``` How should I tell that the last argument to `spark-submit` could be a file say in S3? Or should I do submission from a working directory that mimics mesos slave working directory? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/8682#issuecomment-140642543 LGTM. Merged into master. @vinodkc `values` needs API doc, alone with some other public methods. I created a JIRA for it: https://issues.apache.org/jira/browse/SPARK-10631. Could you help add them? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8682 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/8740#discussion_r39597206 --- Diff: mllib/src/test/java/org/apache/spark/ml/feature/JavaPackage.java --- @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.ml.feature; --- End diff -- Makes sense, for now do you think we should keep the test or 86 it until we get the syncing solution in place. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8740#discussion_r39597002
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/package-info.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+/**
+ * Feature transformers
+ *
+ * The `ml.feature` package provides common feature transformers that help
convert raw data or
+ * features into more suitable forms for model fitting.
+ * Most feature transformers are implemented as {@link
org.apache.spark.ml.Transformer}s, which
+ * transforms one {@link org.apache.spark.sql.DataFrame} into another,
e.g.,
+ * {@link org.apache.spark.feature.HashingTF}.
+ * Some feature transformers are implemented as {@link
org.apache.spark.ml.Estimator}}s, because the
+ * transformation requires some aggregated information of the dataset,
e.g., document
+ * frequencies in {@link org.apache.spark.ml.feature.IDF}.
+ * For those feature transformers, calling {@link
org.apache.spark.ml.Estimator#fit} is required to
+ * obtain the model first, e.g., {@link
org.apache.spark.ml.feature.IDFModel}, in order to apply
+ * transformation.
+ * The transformation is usually done by appending new columns to the input
+ * {@link org.apache.spark.sql.DataFrame}, so all input columns are
carried over.
+ *
+ * We try to make each transformer minimal, so it becomes flexible to
assemble feature
+ * transformation pipelines.
+ * {@link org.apache.spark.ml.Pipeline} can be used to chain feature
transformers, and
+ * {@link org.apache.spark.ml.feature.VectorAssembler} can be used to
combine multiple feature
+ * transformations, for example:
+ *
+ *
+ *
+ * import java.util.Arrays;
+ * import java.util.List;
+ *
+ * import org.apache.spark.api.java.JavaRDD;
+ * // Import factory methods provided by DataTypes.
+ * import org.apache.spark.sql.types.DataTypes;
+ * // Import StructType and StructField
+ * import org.apache.spark.sql.types.StructType;
+ * import org.apache.spark.sql.types.StructField;
+ * import org.apache.spark.sql.DataFrame;
+ * import org.apache.spark.sql.RowFactory;
+ * import org.apache.spark.sql.Row;
+ *
+ * import org.apache.spark.ml.feature.*;
+ * import org.apache.spark.ml.Pipeline;
+ * import org.apache.spark.ml.PipelineStage;
+ * import org.apache.spark.ml.PipelineModel;
+ *
+ * // a DataFrame with three columns: id (integer), text (string), and
rating (double).
+ * List fields = Arrays.asList(
+ * DataTypes.createStructField("id", DataTypes.IntegerType, false),
+ * DataTypes.createStructField("text", DataTypes.StringType, false),
+ * DataTypes.createStructField("rating", DataTypes.DoubleType,
false));
+ * StructType schema = DataTypes.createStructType(fields);
+ * JavaRDD rowRDD = jsc.parallelize(
--- End diff --
Not in this PR, we could add `createDataFrame(List, StructType)` to
`SQLContext`: https://issues.apache.org/jira/browse/SPARK-10630
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/8740#discussion_r39597009 --- Diff: mllib/src/test/java/org/apache/spark/ml/feature/JavaPackage.java --- @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.ml.feature; --- End diff -- Thanks for testing the code! I think it might be an overkill to include a unit test for package doc. The problem would be keeping the content in-sync in the future. https://issues.apache.org/jira/browse/SPARK-10383 is for this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8740#discussion_r39596999
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/package-info.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+/**
+ * Feature transformers
+ *
+ * The `ml.feature` package provides common feature transformers that help
convert raw data or
+ * features into more suitable forms for model fitting.
+ * Most feature transformers are implemented as {@link
org.apache.spark.ml.Transformer}s, which
+ * transforms one {@link org.apache.spark.sql.DataFrame} into another,
e.g.,
+ * {@link org.apache.spark.feature.HashingTF}.
+ * Some feature transformers are implemented as {@link
org.apache.spark.ml.Estimator}}s, because the
+ * transformation requires some aggregated information of the dataset,
e.g., document
+ * frequencies in {@link org.apache.spark.ml.feature.IDF}.
+ * For those feature transformers, calling {@link
org.apache.spark.ml.Estimator#fit} is required to
+ * obtain the model first, e.g., {@link
org.apache.spark.ml.feature.IDFModel}, in order to apply
+ * transformation.
+ * The transformation is usually done by appending new columns to the input
+ * {@link org.apache.spark.sql.DataFrame}, so all input columns are
carried over.
+ *
+ * We try to make each transformer minimal, so it becomes flexible to
assemble feature
+ * transformation pipelines.
+ * {@link org.apache.spark.ml.Pipeline} can be used to chain feature
transformers, and
+ * {@link org.apache.spark.ml.feature.VectorAssembler} can be used to
combine multiple feature
+ * transformations, for example:
+ *
+ *
+ *
+ * import java.util.Arrays;
+ * import java.util.List;
+ *
+ * import org.apache.spark.api.java.JavaRDD;
+ * // Import factory methods provided by DataTypes.
+ * import org.apache.spark.sql.types.DataTypes;
+ * // Import StructType and StructField
+ * import org.apache.spark.sql.types.StructType;
+ * import org.apache.spark.sql.types.StructField;
+ * import org.apache.spark.sql.DataFrame;
+ * import org.apache.spark.sql.RowFactory;
+ * import org.apache.spark.sql.Row;
+ *
+ * import org.apache.spark.ml.feature.*;
+ * import org.apache.spark.ml.Pipeline;
+ * import org.apache.spark.ml.PipelineStage;
+ * import org.apache.spark.ml.PipelineModel;
+ *
+ * // a DataFrame with three columns: id (integer), text (string), and
rating (double).
+ * List fields = Arrays.asList(
--- End diff --
We can avoid importing `List` to construct `schema` directly. With `import
static ...DataTypes.*;`, the code could be simpler:
~~~java
StructType schema = createStructType(Arrays.asList(
createStructField("id", IntegerType, false),
...
));
~~~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8740#discussion_r39596988
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/package-info.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+/**
+ * Feature transformers
+ *
+ * The `ml.feature` package provides common feature transformers that help
convert raw data or
+ * features into more suitable forms for model fitting.
+ * Most feature transformers are implemented as {@link
org.apache.spark.ml.Transformer}s, which
+ * transforms one {@link org.apache.spark.sql.DataFrame} into another,
e.g.,
+ * {@link org.apache.spark.feature.HashingTF}.
+ * Some feature transformers are implemented as {@link
org.apache.spark.ml.Estimator}}s, because the
+ * transformation requires some aggregated information of the dataset,
e.g., document
+ * frequencies in {@link org.apache.spark.ml.feature.IDF}.
+ * For those feature transformers, calling {@link
org.apache.spark.ml.Estimator#fit} is required to
+ * obtain the model first, e.g., {@link
org.apache.spark.ml.feature.IDFModel}, in order to apply
+ * transformation.
+ * The transformation is usually done by appending new columns to the input
+ * {@link org.apache.spark.sql.DataFrame}, so all input columns are
carried over.
+ *
+ * We try to make each transformer minimal, so it becomes flexible to
assemble feature
+ * transformation pipelines.
+ * {@link org.apache.spark.ml.Pipeline} can be used to chain feature
transformers, and
+ * {@link org.apache.spark.ml.feature.VectorAssembler} can be used to
combine multiple feature
+ * transformations, for example:
+ *
+ *
+ *
+ * import java.util.Arrays;
+ * import java.util.List;
+ *
+ * import org.apache.spark.api.java.JavaRDD;
+ * // Import factory methods provided by DataTypes.
--- End diff --
the comment is not necessary
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8740#discussion_r39596994
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/package-info.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+/**
+ * Feature transformers
+ *
+ * The `ml.feature` package provides common feature transformers that help
convert raw data or
+ * features into more suitable forms for model fitting.
+ * Most feature transformers are implemented as {@link
org.apache.spark.ml.Transformer}s, which
+ * transforms one {@link org.apache.spark.sql.DataFrame} into another,
e.g.,
+ * {@link org.apache.spark.feature.HashingTF}.
+ * Some feature transformers are implemented as {@link
org.apache.spark.ml.Estimator}}s, because the
+ * transformation requires some aggregated information of the dataset,
e.g., document
+ * frequencies in {@link org.apache.spark.ml.feature.IDF}.
+ * For those feature transformers, calling {@link
org.apache.spark.ml.Estimator#fit} is required to
+ * obtain the model first, e.g., {@link
org.apache.spark.ml.feature.IDFModel}, in order to apply
+ * transformation.
+ * The transformation is usually done by appending new columns to the input
+ * {@link org.apache.spark.sql.DataFrame}, so all input columns are
carried over.
+ *
+ * We try to make each transformer minimal, so it becomes flexible to
assemble feature
+ * transformation pipelines.
+ * {@link org.apache.spark.ml.Pipeline} can be used to chain feature
transformers, and
+ * {@link org.apache.spark.ml.feature.VectorAssembler} can be used to
combine multiple feature
+ * transformations, for example:
+ *
+ *
+ *
+ * import java.util.Arrays;
+ * import java.util.List;
+ *
+ * import org.apache.spark.api.java.JavaRDD;
+ * // Import factory methods provided by DataTypes.
+ * import org.apache.spark.sql.types.DataTypes;
+ * // Import StructType and StructField
--- End diff --
ditto
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user mengxr commented on a diff in the pull request:
https://github.com/apache/spark/pull/8740#discussion_r39596991
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/feature/package-info.java ---
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+/**
+ * Feature transformers
+ *
+ * The `ml.feature` package provides common feature transformers that help
convert raw data or
+ * features into more suitable forms for model fitting.
+ * Most feature transformers are implemented as {@link
org.apache.spark.ml.Transformer}s, which
+ * transforms one {@link org.apache.spark.sql.DataFrame} into another,
e.g.,
+ * {@link org.apache.spark.feature.HashingTF}.
+ * Some feature transformers are implemented as {@link
org.apache.spark.ml.Estimator}}s, because the
+ * transformation requires some aggregated information of the dataset,
e.g., document
+ * frequencies in {@link org.apache.spark.ml.feature.IDF}.
+ * For those feature transformers, calling {@link
org.apache.spark.ml.Estimator#fit} is required to
+ * obtain the model first, e.g., {@link
org.apache.spark.ml.feature.IDFModel}, in order to apply
+ * transformation.
+ * The transformation is usually done by appending new columns to the input
+ * {@link org.apache.spark.sql.DataFrame}, so all input columns are
carried over.
+ *
+ * We try to make each transformer minimal, so it becomes flexible to
assemble feature
+ * transformation pipelines.
+ * {@link org.apache.spark.ml.Pipeline} can be used to chain feature
transformers, and
+ * {@link org.apache.spark.ml.feature.VectorAssembler} can be used to
combine multiple feature
+ * transformations, for example:
+ *
+ *
+ *
+ * import java.util.Arrays;
+ * import java.util.List;
+ *
+ * import org.apache.spark.api.java.JavaRDD;
+ * // Import factory methods provided by DataTypes.
+ * import org.apache.spark.sql.types.DataTypes;
--- End diff --
`import static ...DataTypes.*;`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9182][SQL] Cast filters are not passed ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8718#issuecomment-140631024 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9182][SQL] Cast filters are not passed ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8718#issuecomment-140631028 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42523/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9182][SQL] Cast filters are not passed ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8718#issuecomment-140630773 [Test build #42523 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42523/console) for PR 8718 at commit [`0ecdb53`](https://github.com/apache/spark/commit/0ecdb5355409397830efeaeaa357119538447166). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TaskCommitDenied(` * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9669][MESOS] Support PySpark on Mesos c...
Github user viesti commented on the pull request: https://github.com/apache/spark/pull/8349#issuecomment-140625049 @tnachen Thanks for the kind advice! :) I'll have a try, although we found out that AWS EMR seems to support running Spark also and got an initial hello world PySpark script running on it. Figuring out now what fits our (simple) needs best. It never hurts to ask the community I guess :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8682#issuecomment-140624143 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42525/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8682#issuecomment-140624142 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8682#issuecomment-140624097 [Test build #42525 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42525/console) for PR 8682 at commit [`b77e420`](https://github.com/apache/spark/commit/b77e420ed0d737e99c763cb7150949a0f0ec9e11). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-3718] FsHistoryProvider should consider...
Github user abraithwaite commented on the pull request: https://github.com/apache/spark/pull/2573#issuecomment-140622568 Hello! I was reading the explanation and I'm not quite sure I understand the reasoning still. I spent a bit too long trying to figure out how to configure the executors to log to the correct hdfs directory. How exactly does a spark application connect _directly_ to a spark history server? It's my understanding (correct me if I'm wrong) that the application logs to a directory and the history server reads that directory. So even if you had two history servers, they'd presumably both only have one log directory configuration parameter, no? Clearly, the docs should at least be cleared up on the monitoring page. https://spark.apache.org/docs/latest/monitoring.html has no mention of spark.eventLog.dir (although it does mention spark.eventLog.enabled). It seems intuitive that these would be the same property. /cc @andrewor14 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10529][SQL]When creating multiple HiveC...
Github user GavinGavinNo1 commented on the pull request: https://github.com/apache/spark/pull/8713#issuecomment-140621464 @marmbrus Sorry to disturb again. Could you please give me a reply? It's my first try. Maybe I need some advice. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9963] [ML] RandomForest cleanup: replac...
Github user lkhamsurenl commented on the pull request: https://github.com/apache/spark/pull/8609#issuecomment-140621230 Sorry for the late response! I see what you're saying. The difficulty with that approach I am having is in inside def binSeqOp(agg: Array[DTStatsAggregator], baggedPoint: BaggedPoint[TreePoint]): Array[DTStatsAggregator], where the function is called: treeToNodeToIndexInfo: Map[Int, Map[Int, NodeIndexInfo]] uses nodeIndex to get the NodeInfoIndex, which requires us to know the nodeIndex. If we convert the LearningNode to Node then implement the predictImpl() there is no nodeIndex I believe. Correct me if I'm wrong --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8682#issuecomment-140621207 [Test build #42525 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42525/consoleFull) for PR 8682 at commit [`b77e420`](https://github.com/apache/spark/commit/b77e420ed0d737e99c763cb7150949a0f0ec9e11). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6028][Core]A new RPC implemetation base...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6457#issuecomment-140620519 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42517/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6028][Core]A new RPC implemetation base...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/6457#issuecomment-140620517 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6028][Core]A new RPC implemetation base...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/6457#issuecomment-140620297 [Test build #42517 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42517/console) for PR 6457 at commit [`e8ecab8`](https://github.com/apache/spark/commit/e8ecab8c20e496b961b4ce51dac1e33d840dc2d4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8682#issuecomment-140619959 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10516][ MLlib]Added values property in ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8682#issuecomment-140619998 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10101] [SQL] Added stringDataType optio...
Github user pallavipr commented on the pull request: https://github.com/apache/spark/pull/8374#issuecomment-140619445 Looks good Rama. We are almost done with DB2 changes - will send for review soon. One question, did you introduce a stringDataType property in connection url? And StringType will be mapped to the value provided for stringDataType? Thanks, Pallavi --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9585] add config to enable inputFormat ...
Github user XuTingjun commented on the pull request: https://github.com/apache/spark/pull/7918#issuecomment-140619178 @JoshRosen, Can you have a look on this? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140615999 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42522/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140615996 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140615919 [Test build #42524 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42524/console) for PR 7987 at commit [`92c8287`](https://github.com/apache/spark/commit/92c828710cdd4ad4580dc06ea1b9ba51e2b5ed8f). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Interaction(override val uid: String) extends Transformer` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140615923 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42524/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140615922 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140615714 [Test build #42524 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42524/consoleFull) for PR 7987 at commit [`92c8287`](https://github.com/apache/spark/commit/92c828710cdd4ad4580dc06ea1b9ba51e2b5ed8f). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9182][SQL] Cast filters are not passed ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8718#issuecomment-140615136 [Test build #42523 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42523/consoleFull) for PR 8718 at commit [`0ecdb53`](https://github.com/apache/spark/commit/0ecdb5355409397830efeaeaa357119538447166). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9182][SQL] Cast filters are not passed ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8718#issuecomment-140614868 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140614869 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9182][SQL] Cast filters are not passed ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8718#issuecomment-140614859 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140614860 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user ericl commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140614349 @mengxr I did the refactoring as suggested --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140613869 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9698] [ML] Add RInteraction transformer...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/7987#issuecomment-140613858 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9987][SQL]Implement the local Aggregate...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8769#issuecomment-140613727 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9987][SQL]Implement the local Aggregate...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8769#issuecomment-140613728 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42518/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9987][SQL]Implement the local Aggregate...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8769#issuecomment-140613667 [Test build #42518 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42518/console) for PR 8769 at commit [`1e5ed75`](https://github.com/apache/spark/commit/1e5ed753767ec15305b1710c6aa727bc548f560a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TaskCommitDenied(` * ` final val probabilityCol: Param[String] = new Param[String](this, "probabilityCol", "Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities")` * `case class AggregateNode(` * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/8777#discussion_r39589775 --- Diff: python/pyspark/sql/functions.py --- @@ -189,6 +190,14 @@ def approxCountDistinct(col, rsd=None): return Column(jc) +@since(1.6) +def broadcast(df): +"""Marks a DataFrame as small enough for use in broadcast joins.""" + +sc = SparkContext._active_spark_context +return DataFrame(sc._jvm.functions.broadcast(df._jdf),sc._jsc) --- End diff -- add space after comma --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user rxin commented on a diff in the pull request: https://github.com/apache/spark/pull/8777#discussion_r39589771 --- Diff: python/pyspark/sql/functions.py --- @@ -189,6 +190,14 @@ def approxCountDistinct(col, rsd=None): return Column(jc) +@since(1.6) +def broadcast(df): +"""Marks a DataFrame as small enough for use in broadcast joins.""" --- End diff -- can you add a test for this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-6624][SQL]Convert filters into CNF for ...
Github user yjshen commented on the pull request: https://github.com/apache/spark/pull/8200#issuecomment-140610959 @marmbrus converting a filter into CNF may lead to an expanded filter, which I think is not necessarily a general optimisation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/8776#discussion_r39589393 --- Diff: docs/sql-programming-guide.md --- @@ -2261,7 +2261,7 @@ Several caching related features are not supported yet: ## Compatibility with Apache Hive Spark SQL is designed to be compatible with the Hive Metastore, SerDes and UDFs. Currently Spark -SQL is based on Hive 0.12.0 and 0.13.1. +SQL is based on Hive 0.12.0 and 1.2.1. --- End diff -- How about we say that Hive SerDes and UDFs are based on Hive 1.2.1, and Spark SQL can be connected to different versions of Hive Metastore (from 0.12.0 to 1.2.1. Also see http://spark.apache.org/docs/latest/sql-programming-guide.html#interacting-with-different-versions-of-hive-metastore). @marmbrus What do you think? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10595] [ML] [MLLIB] [DOCS] Various ML g...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8752 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9078] [SQL] Allow jdbc dialects to over...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/8676 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10595] [ML] [MLLIB] [DOCS] Various ML g...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/8752#issuecomment-140608613 Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9078] [SQL] Allow jdbc dialects to over...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/8676#issuecomment-140608496 It has been merged to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9078] [SQL] Allow jdbc dialects to over...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8676#issuecomment-140607740 @rxin I reverted the patch that caused those. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9078] [SQL] Allow jdbc dialects to over...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/8676#issuecomment-140607734 I've merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9078] [SQL] Allow jdbc dialects to over...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/8676#issuecomment-140607639 @vanzin do you know what's going on with the tests? [error] Execution of test test.org.apache.spark.sql.JavaApplySchemaSuite failed: java.lang.ClassNotFoundException: org.apache.spark.deploy.yarn.ExtendedYarnTest --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10593] [SQL] fix resolve output of Gene...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8755#issuecomment-140607363 [Test build #1764 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1764/console) for PR 8755 at commit [`887474e`](https://github.com/apache/spark/commit/887474e6908ea5f31108065d8c16f6ce5e88782d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8777#issuecomment-140607334 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10577] [PySpark] DataFrame hint for bro...
GitHub user Jianfeng-chs opened a pull request: https://github.com/apache/spark/pull/8777 [SPARK-10577] [PySpark] DataFrame hint for broadcast join https://issues.apache.org/jira/browse/SPARK-10577 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Jianfeng-chs/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8777.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8777 commit ded210be4bb7c01f6df1ede6269cfb2b1db325d2 Author: Jian Feng Date: 2015-09-16T02:18:55Z [SPARK-10577] [PySpark] DataFrame hint for broadcast join https://issues.apache.org/jira/browse/SPARK-10577 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140606717 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42520/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140606716 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140606637 [Test build #42520 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42520/console) for PR 8740 at commit [`f844d55`](https://github.com/apache/spark/commit/f844d55dfc307d9b7ec9a6a7a064928f252827c8). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TaskCommitDenied(` * ` final val probabilityCol: Param[String] = new Param[String](this, "probabilityCol", "Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities")` * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
reviews@spark.apache.org
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8544#issuecomment-140606432 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
reviews@spark.apache.org
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8544#issuecomment-140606392 [Test build #42515 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42515/console) for PR 8544 at commit [`edbbf6f`](https://github.com/apache/spark/commit/edbbf6fae97f67c5d9a309019514745cf35a2cbe). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TaskCommitDenied(` * ` final val probabilityCol: Param[String] = new Param[String](this, "probabilityCol", "Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities")` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
reviews@spark.apache.org
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8544#issuecomment-140606433 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42515/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10300] [BUILD] [TESTS] Add support for ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8775#issuecomment-140606242 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10300] [BUILD] [TESTS] Add support for ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8775#issuecomment-140606243 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42514/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10300] [BUILD] [TESTS] Add support for ...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8775#issuecomment-140606177 [Test build #42514 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42514/console) for PR 8775 at commit [`f3bb7b4`](https://github.com/apache/spark/commit/f3bb7b46288dbcfe3cc8554b084f38da7c20d3b4). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140606066 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42519/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140606065 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10515] When killing executor, there is ...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8668#issuecomment-140606005 To clarify: the YARN backend also needs to know whether the executor being killed needs to be replaced or not. Right now, when the executor is not to be replaced, that's communicated to the YARN backend using two RPCs: one to kill the executor, one to update the number of requested executors. So for your current patch to work on YARN, you'd have to propagate that information (whether the executor needs to be replaced) in the `KillExecutors` message sent to the YARN backend, and make the AM updating its bookkeeping accordingly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140605943 [Test build #42519 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42519/console) for PR 8740 at commit [`0e1a49e`](https://github.com/apache/spark/commit/0e1a49ec80ea3c4e75dbd5bf17eba996fa4ffadd). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TaskCommitDenied(` * ` final val probabilityCol: Param[String] = new Param[String](this, "probabilityCol", "Column name for predicted class conditional probabilities. Note: Not all models output well-calibrated probability estimates! These probabilities should be treated as confidences, not precise probabilities")` * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8776#issuecomment-140602316 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8776#issuecomment-140602320 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42521/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8776#issuecomment-140601943 [Test build #42521 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42521/console) for PR 8776 at commit [`4662a25`](https://github.com/apache/spark/commit/4662a2502aed9d7c566fbc134a94daff752455b9). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `case class TaskCommitDenied(` * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10515] When killing executor, there is ...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8668#issuecomment-140601612 Ok, I think I see what the problem is. But your fix is not correct. The problem is here: doRequestTotalExecutors( numExistingExecutors + numPendingExecutors - executorsPendingToRemove.size) By subtracting `executorsPendingToRemove.size` when that list contains an executor that is pending replacement, that replacement will be lost. The fix is to keep track of how many replacement executors the code is waiting for, and account for that in the above equation, not to remove that code altogether. > So there is no need to change the number of executors when killing executors. That's not true, in YARN, at least. See SPARK-6325. So you can't make your current change unless you also change how the YARN backend does accounting for the running executors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
Github user yhuai commented on a diff in the pull request: https://github.com/apache/spark/pull/8776#discussion_r39587260 --- Diff: docs/sql-programming-guide.md --- @@ -1954,7 +1954,7 @@ without the need to write any code. ## Running the Thrift JDBC/ODBC server The Thrift JDBC/ODBC server implemented here corresponds to the [`HiveServer2`](https://cwiki.apache.org/confluence/display/Hive/Setting+Up+HiveServer2) -in Hive 0.13. You can test the JDBC server with the beeline script that comes with either Spark or Hive 0.13. +in Hive 1.2.1 You can test the JDBC server with the beeline script that comes with either Spark or Hive 1.2.1. --- End diff -- @liancheng We should say Hive 1.2.1 at here, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8776#issuecomment-140597352 [Test build #42521 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42521/consoleFull) for PR 8776 at commit [`4662a25`](https://github.com/apache/spark/commit/4662a2502aed9d7c566fbc134a94daff752455b9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10531] [CORE] AppId is set as AppName i...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/8688#issuecomment-140597528 @zjffdu could you take a look at whether `SparkUI.setAppName` is really needed at all? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8776#issuecomment-140597145 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8776#issuecomment-140597131 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][SQL][DOC] Documentation about th...
GitHub user sarutak opened a pull request: https://github.com/apache/spark/pull/8776 [SPARK-10584][SQL][DOC] Documentation about the compatible Hive version is wrong. In Spark 1.5.0, Spark SQL is compatible with Hive 0.12.0 through 1.2.1 but the documentation is wrong. /CC @yhuai You can merge this pull request into a Git repository by running: $ git pull https://github.com/sarutak/spark SPARK-10584-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/8776.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8776 commit 4662a2502aed9d7c566fbc134a94daff752455b9 Author: Kousuke Saruta Date: 2015-09-16T01:48:12Z Fix the description of the Hive version in the document --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140596833 [Test build #42520 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42520/consoleFull) for PR 8740 at commit [`f844d55`](https://github.com/apache/spark/commit/f844d55dfc307d9b7ec9a6a7a064928f252827c8). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140595598 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140595613 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140595362 [Test build #42519 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42519/consoleFull) for PR 8740 at commit [`0e1a49e`](https://github.com/apache/spark/commit/0e1a49ec80ea3c4e75dbd5bf17eba996fa4ffadd). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140594433 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10077][DOCS][ML] Add package info for j...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8740#issuecomment-140594442 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10613] [SPARK-10624] [SQL] Reduce Local...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8764#issuecomment-140594239 [Test build #1762 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1762/console) for PR 8764 at commit [`3bd5ac7`](https://github.com/apache/spark/commit/3bd5ac7af53085a25b4894d84b0d4168ce6fd44d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9987][SQL]Implement the local Aggregate...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8769#issuecomment-140594122 [Test build #42518 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42518/consoleFull) for PR 8769 at commit [`1e5ed75`](https://github.com/apache/spark/commit/1e5ed753767ec15305b1710c6aa727bc548f560a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10584][DOC][SQL] Documentation about sp...
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/8739#issuecomment-140594156 O.K. I'll do it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10613] [SPARK-10624] [SQL] Reduce Local...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8764#issuecomment-140594002 [Test build #1763 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1763/console) for PR 8764 at commit [`3bd5ac7`](https://github.com/apache/spark/commit/3bd5ac7af53085a25b4894d84b0d4168ce6fd44d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9987][SQL]Implement the local Aggregate...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8769#issuecomment-140593857 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-9987][SQL]Implement the local Aggregate...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/8769#issuecomment-140593845 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-10613] [SPARK-10624] [SQL] Reduce Local...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/8764#issuecomment-140593623 [Test build #1761 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1761/console) for PR 8764 at commit [`3bd5ac7`](https://github.com/apache/spark/commit/3bd5ac7af53085a25b4894d84b0d4168ce6fd44d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `abstract class LocalNode(conf: SQLConf) extends QueryPlan[LocalNode] with Logging ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
