[jira] (FLINK-5633) ClassCastException: X cannot be cast to X when re-submitting a job.
Title: Message Title Giuliano Caliari closed an issue as Workaround Flink / FLINK-5633 ClassCastException: X cannot be cast to X when re-submitting a job. Change By: Giuliano Caliari Resolution: Workaround Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5633) ClassCastException: X cannot be cast to X when re-submitting a job.
Title: Message Title Giuliano Caliari commented on FLINK-5633 Re: ClassCastException: X cannot be cast to X when re-submitting a job. Hey Stephan Ewen, I've confirmed that Yuri's solution also works for us. Closing it. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5687) Missing null check for return value from listFiles() in JarListHandler
Title: Message Title Sachin Goel commented on FLINK-5687 Re: Missing null check for return value from listFiles() in JarListHandler I think the checks in WebRuntimeMonitor while creating the upload directory take care of this. We check that the directory does exist and is writable. Is there any other way a null value might be returned? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5689) Kafka Connector Documentation gives Compile time errors (Java)
Title: Message Title
Vikram Rawat created an issue
Flink / FLINK-5689
Kafka Connector Documentation gives Compile time errors (Java)
Issue Type:
Improvement
Affects Versions:
1.1.4
Assignee:
Unassigned
Components:
Kafka Connector
Created:
31/Jan/17 06:31
Environment:
Java API for using Kafka Connector
Labels:
kafka
Priority:
Major
Reporter:
Vikram Rawat
https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/streaming/connectors/kafka.html
Kafka Consumer Java section: Java code snippet given on the page, is as below: DataStream stream = env .addSource(new FlinkKafkaConsumer08<>("topic", new SimpleStringSchema(), properties)) .print();
[jira] (FLINK-2168) Add HBaseTableSource
Title: Message Title ASF GitHub Bot commented on FLINK-2168 Re: Add HBaseTableSource Github user ramkrish86 commented on the issue: https://github.com/apache/flink/pull/3149 @tonycox I have addressed all your latest comments including making HBaseTableSource a ProjectableTableSource. @wuchong , @fhueske Are you guys fine with the latest updates. If so we can try closing this PR and further discussions of adding StreamingtableSource and supporting WHERE clauses can be done in subsequent PRs. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5015) Add Tests/ITCase for Kafka Per-Partition Watermarks
Title: Message Title Tzu-Li (Gordon) Tai assigned an issue to Tzu-Li (Gordon) Tai Flink / FLINK-5015 Add Tests/ITCase for Kafka Per-Partition Watermarks Change By: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3149: FLINK-2168 Add HBaseTableSource
Github user ramkrish86 commented on the issue: https://github.com/apache/flink/pull/3149 @tonycox I have addressed all your latest comments including making HBaseTableSource a ProjectableTableSource. @wuchong , @fhueske Are you guys fine with the latest updates. If so we can try closing this PR and further discussions of adding StreamingtableSource and supporting WHERE clauses can be done in subsequent PRs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-5661) flink-mesos should shade its httpclient dependency away
Title: Message Title Tzu-Li (Gordon) Tai assigned an issue to Tzu-Li (Gordon) Tai Flink / FLINK-5661 flink-mesos should shade its httpclient dependency away Change By: Tzu-Li (Gordon) Tai Assignee: Tzu-Li (Gordon) Tai Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink pull request #3232: [FLINK-5183] [py] Support mulitple jobs per plan f...
Github user GEOFBOT commented on a diff in the pull request:
https://github.com/apache/flink/pull/3232#discussion_r98599687
--- Diff:
flink-libraries/flink-python/src/main/java/org/apache/flink/python/api/PythonPlanBinder.java
---
@@ -139,15 +139,28 @@ private void runPlan(String[] args) throws Exception {
String tmpPath = FLINK_PYTHON_FILE_PATH + r.nextInt();
prepareFiles(tmpPath, Arrays.copyOfRange(args, 0, split
== 0 ? args.length : split));
startPython(tmpPath, Arrays.copyOfRange(args, split ==
0 ? args.length : split + 1, args.length));
- receivePlan();
- if (env instanceof LocalEnvironment) {
- FLINK_HDFS_PATH = "file:" +
System.getProperty("java.io.tmpdir") + File.separator + "flink";
+ // Python process should terminate itself when all jobs
have been run
+ while (streamer.isPythonRunning()) {
+ try {
+ receivePlan();
+ } catch (SocketTimeoutException ste) {
+ // If the socket times out, check to
see if Python process has exited yet
+ continue;
+ }
+
+ if (env instanceof LocalEnvironment) {
+ FLINK_HDFS_PATH = "file:" +
System.getProperty("java.io.tmpdir") + File.separator + "flink";
+ }
+
+ distributeFiles(tmpPath, env);
+ JobExecutionResult jer = env.execute();
+ sendResult(jer);
+
+ environmentCounter++;
}
- distributeFiles(tmpPath, env);
- JobExecutionResult jer = env.execute();
- sendResult(jer);
+ clearPath(tmpPath);
--- End diff --
The files necessary for plan mode (Flink python module, plan file, etc.)
are copied with `prepareFiles` on L140. If `clearPath` is left at L203 inside
of `distributeFiles`, those necessary Python files will be removed after the
first job is executed and subsequent jobs can't be run.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] (FLINK-5183) [py] Support multiple jobs per Python plan file
Title: Message Title
ASF GitHub Bot commented on FLINK-5183
Re: [py] Support multiple jobs per Python plan file
Github user GEOFBOT commented on a diff in the pull request:
https://github.com/apache/flink/pull/3232#discussion_r98599687
— Diff: flink-libraries/flink-python/src/main/java/org/apache/flink/python/api/PythonPlanBinder.java — @@ -139,15 +139,28 @@ private void runPlan(String[] args) throws Exception { String tmpPath = FLINK_PYTHON_FILE_PATH + r.nextInt(); prepareFiles(tmpPath, Arrays.copyOfRange(args, 0, split == 0 ? args.length : split)); startPython(tmpPath, Arrays.copyOfRange(args, split == 0 ? args.length : split + 1, args.length));
receivePlan();
if (env instanceof LocalEnvironment) {
FLINK_HDFS_PATH = "file:" + System.getProperty("java.io.tmpdir") + File.separator + "flink"; + // Python process should terminate itself when all jobs have been run + while (streamer.isPythonRunning())
Unknown macro: { + try { + receivePlan(); + } catch (SocketTimeoutException ste) { + // If the socket times out, check to see if Python process has exited yet + continue; + } + + if (env instanceof LocalEnvironment) { + FLINK_HDFS_PATH = "file:" + System.getProperty("java.io.tmpdir") + File.separator + "flink"; + } + + distributeFiles(tmpPath, env); + JobExecutionResult jer = env.execute(); + sendResult(jer); + + environmentCounter++; }
distributeFiles(tmpPath, env);
JobExecutionResult jer = env.execute();
sendResult(jer); + clearPath(tmpPath);
End diff –
The files necessary for plan mode (Flink python module, plan file, etc.) are copied with `prepareFiles` on L140. If `clearPath` is left at L203 inside of `distributeFiles`, those necessary Python files will be removed after the first job is executed and subsequent jobs can't be run.
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title
ASF GitHub Bot commented on FLINK-1731
Re: Add kMeans clustering algorithm to machine learning library
Github user skonto commented on a diff in the pull request:
https://github.com/apache/flink/pull/3192#discussion_r98570744
— Diff: flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/clustering/KMeans.scala — @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.clustering + +import org.apache.flink.api.java.functions.FunctionAnnotation.ForwardedFields +import org.apache.flink.api.scala. {DataSet, _}
+import org.apache.flink.ml._ +import org.apache.flink.ml.common. {LabeledVector, _}
+import org.apache.flink.ml.math.Breeze._ +import org.apache.flink.ml.math. {BLAS, Vector}
+import org.apache.flink.ml.metrics.distances.EuclideanDistanceMetric +import org.apache.flink.ml.pipeline._ + + +/** + * Implements the KMeans algorithm which calculates cluster centroids based on set of training data + * points and a set of k initial centroids. + * + * [[KMeans]] is a [[Predictor]] which needs to be trained on a set of data points and can then be + * used to assign new points to the learned cluster centroids. + * + * The KMeans algorithm works as described on Wikipedia + * (http://en.wikipedia.org/wiki/K-means_clustering): + * + * Given an initial set of k means m1(1),…,mk(1) (see below), the algorithm proceeds by alternating + * between two steps: + * + * ===Assignment step:=== + * + * Assign each observation to the cluster whose mean yields the least within-cluster sum of + * squares (WCSS). Since the sum of squares is the squared Euclidean distance, this is intuitively + * the "nearest" mean. (Mathematically, this means partitioning the observations according to the + * Voronoi diagram generated by the means). + * + * `S_i^(t) = { x_p : || x_p - m_i^(t) ||^2 ≤ || x_p - m_j^(t) ||^2 \forall j, 1 ≤ j ≤ k}
`, + * where each `x_p` is assigned to exactly one `S^ {(t)}`, even if it could be assigned to two or + * more of them. + * + * ===Update step:=== + * + * Calculate the new means to be the centroids of the observations in the new clusters. + * + * `m^{(t+1)}_i = ( 1 / |S^{(t)}
i| ) \sum{x_j \in S^ {(t)}
_i} x_j` + * + * Since the arithmetic mean is a least-squares estimator, this also minimizes the within-cluster + * sum of squares (WCSS) objective. + * + * @example + * {{ { + * val trainingDS: DataSet[Vector] = env.fromCollection(Clustering.trainingData) + * val initialCentroids: DataSet[LabledVector] = env.fromCollection(Clustering.initCentroids) + * + * val kmeans = KMeans() + * .setInitialCentroids(initialCentroids) + * .setNumIterations(10) + * + * kmeans.fit(trainingDS) + * + * // getting the computed centroids + * val centroidsResult =
[GitHub] flink pull request #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's alg...
Github user skonto commented on a diff in the pull request:
https://github.com/apache/flink/pull/3192#discussion_r98570744
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/clustering/KMeans.scala
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.clustering
+
+import
org.apache.flink.api.java.functions.FunctionAnnotation.ForwardedFields
+import org.apache.flink.api.scala.{DataSet, _}
+import org.apache.flink.ml._
+import org.apache.flink.ml.common.{LabeledVector, _}
+import org.apache.flink.ml.math.Breeze._
+import org.apache.flink.ml.math.{BLAS, Vector}
+import org.apache.flink.ml.metrics.distances.EuclideanDistanceMetric
+import org.apache.flink.ml.pipeline._
+
+
+/**
+ * Implements the KMeans algorithm which calculates cluster centroids
based on set of training data
+ * points and a set of k initial centroids.
+ *
+ * [[KMeans]] is a [[Predictor]] which needs to be trained on a set of
data points and can then be
+ * used to assign new points to the learned cluster centroids.
+ *
+ * The KMeans algorithm works as described on Wikipedia
+ * (http://en.wikipedia.org/wiki/K-means_clustering):
+ *
+ * Given an initial set of k means m1(1),â¦,mk(1) (see below), the
algorithm proceeds by alternating
+ * between two steps:
+ *
+ * ===Assignment step:===
+ *
+ * Assign each observation to the cluster whose mean yields the least
within-cluster sum of
+ * squares (WCSS). Since the sum of squares is the squared Euclidean
distance, this is intuitively
+ * the "nearest" mean. (Mathematically, this means partitioning the
observations according to the
+ * Voronoi diagram generated by the means).
+ *
+ * `S_i^(t) = { x_p : || x_p - m_i^(t) ||^2 ⤠|| x_p - m_j^(t) ||^2
\forall j, 1 ⤠j ⤠k}`,
+ * where each `x_p` is assigned to exactly one `S^{(t)}`, even if it
could be assigned to two or
+ * more of them.
+ *
+ * ===Update step:===
+ *
+ * Calculate the new means to be the centroids of the observations in the
new clusters.
+ *
+ * `m^{(t+1)}_i = ( 1 / |S^{(t)}_i| ) \sum_{x_j \in S^{(t)}_i} x_j`
+ *
+ * Since the arithmetic mean is a least-squares estimator, this also
minimizes the within-cluster
+ * sum of squares (WCSS) objective.
+ *
+ * @example
+ * {{{
+ * val trainingDS: DataSet[Vector] =
env.fromCollection(Clustering.trainingData)
+ * val initialCentroids: DataSet[LabledVector] =
env.fromCollection(Clustering.initCentroids)
+ *
+ * val kmeans = KMeans()
+ * .setInitialCentroids(initialCentroids)
+ * .setNumIterations(10)
+ *
+ * kmeans.fit(trainingDS)
+ *
+ * // getting the computed centroids
+ * val centroidsResult = kmeans.centroids.get.collect()
+ *
+ * // get matching clusters for new points
+ * val testDS: DataSet[Vector] =
env.fromCollection(Clustering.testData)
+ * val clusters: DataSet[LabeledVector] = kmeans.predict(testDS)
+ * }}}
+ *
+ * =Parameters=
+ *
+ * - [[org.apache.flink.ml.clustering.KMeans.NumIterations]]:
+ * Defines the number of iterations to recalculate the centroids of the
clusters. As it
+ * is a heuristic algorithm, there is no guarantee that it will converge
to the global optimum. The
+ * centroids of the clusters and the reassignment of the data points will
be repeated till the
+ * given number of iterations is reached.
+ * (Default value: '''10''')
+ *
+ * - [[org.apache.flink.ml.clustering.KMeans.InitialCentroids]]:
+ * Defines the initial k centroids of the k clusters. They are used as
start off point of the
+ * algorithm for clustering the data set. The centroids are recalculated
as often as set in
+ * [[org.apache.flink.ml.clustering.KMeans.NumIterations]]. The choice of
the initial centroids
+
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title
ASF GitHub Bot commented on FLINK-1731
Re: Add kMeans clustering algorithm to machine learning library
Github user skonto commented on a diff in the pull request:
https://github.com/apache/flink/pull/3192#discussion_r98557895
— Diff: flink-libraries/flink-ml/src/test/scala/org/apache/flink/ml/clustering/ClusteringData.scala — @@ -0,0 +1,277 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.clustering + +import org.apache.flink.ml.common.LabeledVector +import org.apache.flink.ml.math. {DenseVector, Vector}
+import org.apache.flink.ml.metrics.distances.EuclideanDistanceMetric + +/** + * Trainings- and test-data set for the K-Means implementation — End diff –
I would prefer to keep all comments with the same format. Again I see the rest of the code base has a mix of scaladocs and javadocs.
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink pull request #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's alg...
Github user skonto commented on a diff in the pull request:
https://github.com/apache/flink/pull/3192#discussion_r98557895
--- Diff:
flink-libraries/flink-ml/src/test/scala/org/apache/flink/ml/clustering/ClusteringData.scala
---
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.clustering
+
+import org.apache.flink.ml.common.LabeledVector
+import org.apache.flink.ml.math.{DenseVector, Vector}
+import org.apache.flink.ml.metrics.distances.EuclideanDistanceMetric
+
+/**
+ * Trainings- and test-data set for the K-Means implementation
--- End diff --
I would prefer to keep all comments with the same format. Again I see the
rest of the code base has a mix of scaladocs and javadocs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title ASF GitHub Bot commented on FLINK-1731 Re: Add kMeans clustering algorithm to machine learning library Github user skonto commented on the issue: https://github.com/apache/flink/pull/3192 Correct, just saying we will need to create a set of examples for the ml library. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's algorithm)
Github user skonto commented on the issue: https://github.com/apache/flink/pull/3192 Correct, just saying we will need to create a set of examples for the ml library. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-3849) Add FilterableTableSource interface and translation rule
Title: Message Title ASF GitHub Bot commented on FLINK-3849 Re: Add FilterableTableSource interface and translation rule Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3166 Scalar functions can of course not be executed by the source. However, sources need to group the filter conditions into supported and unsupported expressions in any case. Unsupported would also include "I don't know this _expression_". So I think, there would not be additional implementation overhead on the TableSource side but potential gains in performance when pushing everything down. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3166: [FLINK-3849] Add FilterableTableSource interface and Rule...
Github user fhueske commented on the issue: https://github.com/apache/flink/pull/3166 Scalar functions can of course not be executed by the source. However, sources need to group the filter conditions into supported and unsupported expressions in any case. Unsupported would also include "I don't know this expression". So I think, there would not be additional implementation overhead on the TableSource side but potential gains in performance when pushing everything down. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title ASF GitHub Bot commented on FLINK-1731 Re: Add kMeans clustering algorithm to machine learning library Github user sachingoel0101 commented on the issue: https://github.com/apache/flink/pull/3192 Just fyi, that is not an example of the usage of machine learning library. It is just a standalone implementation of the linear regression model. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's algorithm)
Github user sachingoel0101 commented on the issue: https://github.com/apache/flink/pull/3192 Just fyi, that is not an example of the usage of machine learning library. It is just a standalone implementation of the linear regression model. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title
ASF GitHub Bot commented on FLINK-1731
Re: Add kMeans clustering algorithm to machine learning library
Github user skonto commented on a diff in the pull request:
https://github.com/apache/flink/pull/3192#discussion_r98511916
— Diff: flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/clustering/KMeans.scala — @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.clustering + +import org.apache.flink.api.java.functions.FunctionAnnotation.ForwardedFields +import org.apache.flink.api.scala. {DataSet, _}
+import org.apache.flink.ml._ +import org.apache.flink.ml.common. {LabeledVector, _}
+import org.apache.flink.ml.math.Breeze._ +import org.apache.flink.ml.math. {BLAS, Vector}
+import org.apache.flink.ml.metrics.distances.EuclideanDistanceMetric +import org.apache.flink.ml.pipeline._ + + +/** + * Implements the KMeans algorithm which calculates cluster centroids based on set of training data + * points and a set of k initial centroids. — End diff –
:+1:
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink pull request #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's alg...
Github user skonto commented on a diff in the pull request:
https://github.com/apache/flink/pull/3192#discussion_r98511916
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/clustering/KMeans.scala
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.clustering
+
+import
org.apache.flink.api.java.functions.FunctionAnnotation.ForwardedFields
+import org.apache.flink.api.scala.{DataSet, _}
+import org.apache.flink.ml._
+import org.apache.flink.ml.common.{LabeledVector, _}
+import org.apache.flink.ml.math.Breeze._
+import org.apache.flink.ml.math.{BLAS, Vector}
+import org.apache.flink.ml.metrics.distances.EuclideanDistanceMetric
+import org.apache.flink.ml.pipeline._
+
+
+/**
+ * Implements the KMeans algorithm which calculates cluster centroids
based on set of training data
+ * points and a set of k initial centroids.
--- End diff --
:+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title ASF GitHub Bot commented on FLINK-1731 Re: Add kMeans clustering algorithm to machine learning library Github user skonto commented on the issue: https://github.com/apache/flink/pull/3192 Ok ... there is one [example](https://github.com/apache/flink/blob/master/flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/ml/LinearRegression.scala). Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's algorithm)
Github user skonto commented on the issue: https://github.com/apache/flink/pull/3192 Ok ... there is one [example](https://github.com/apache/flink/blob/master/flink-examples/flink-examples-batch/src/main/scala/org/apache/flink/examples/scala/ml/LinearRegression.scala). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-1702) Authenticate via Kerberos from the client only
Title: Message Title Eron Wright commented on FLINK-1702 Re: Authenticate via Kerberos from the client only I think this is already done (for YARN mode); delegation tokens are obtained on the client and passed to the JM/TM. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-3606) Expose Configuration from RichFunctions
Title: Message Title Márton Balassi commented on FLINK-3606 Re: Expose Configuration from RichFunctions If the API is getting cleaned up that way, then this issue is irrelevant. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5662) Alias in front of output fails
Title: Message Title
Timo Walther commented on FLINK-5662
Re: Alias in front of output fails
You are right. This issue is not trivial. Actually we need a `ProjectRemoveRule` that pushes the field names in the input node, such that the field names are not getting lost. Fabian Hueske and I already discussed about having an easier architecture with conversion mappers at the beginning and end, and only operators handling row types in the middle. Maybe it is time for that now. We could then have a additional rule that merges two conversion mappers if nothing is in between.
Table t = tEnv.fromDataSet(text, "text").groupBy("text").select("text AS line"); even fails with a CodeGen exception.
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5688) Unclosed RandomAccessFile in StaticFileServerHandler#respondAsLeader()
Title: Message Title
Ted Yu created an issue
Flink / FLINK-5688
Unclosed RandomAccessFile in StaticFileServerHandler#respondAsLeader()
Issue Type:
Bug
Assignee:
Unassigned
Created:
30/Jan/17 17:09
Priority:
Minor
Reporter:
Ted Yu
final RandomAccessFile raf;
try {
raf = new RandomAccessFile(file, "r");
}
catch (FileNotFoundException e) {
sendError(ctx, NOT_FOUND);
return;
}
long fileLength = raf.length();
The RandomAccessFile, raf, should be closed upon return from the method.
Add Comment
[jira] (FLINK-5682) Fix scala version in flink-streaming-scala POM file
Title: Message Title Aljoscha Krettek commented on FLINK-5682 Re: Fix scala version in flink-streaming-scala POM file Did you observe an actual problem? If I'm not mistaken the entry in the root pom should have a version entry and this should be picked up by the dependency declarations in flink-streaming-scala. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5687) Missing null check for return value from listFiles() in JarListHandler
Title: Message Title Ted Yu commented on FLINK-5687 Re: Missing null check for return value from listFiles() in JarListHandler Similar problem is in JarDeleteHandler#handleJsonRequest() Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5687) Missing null check for return value from listFiles() in JarListHandler
Title: Message Title
Ted Yu created an issue
Flink / FLINK-5687
Missing null check for return value from listFiles() in JarListHandler
Issue Type:
Bug
Assignee:
Unassigned
Created:
30/Jan/17 17:04
Priority:
Minor
Reporter:
Ted Yu
Here is related code:
File[] list = jarDir.listFiles(new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
return name.endsWith(".jar");
}
});
for (File f : list) {
Return value from listFiles() may be null. Common practice is to check before iterating over the Files:
if (files == null) {
LOG.warn("Failed to get files from " + ...
Add Comment
[GitHub] flink issue #3184: [5456] [docs] state intro and new interfaces
Github user aljoscha commented on the issue: https://github.com/apache/flink/pull/3184 Thanks for your work, @alpinegizmo. ð Could you please close this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-5456) Add docs about new state and checkpointing interfaces
Title: Message Title Aljoscha Krettek closed an issue as Fixed Implemented on master in : d27b9fee5f21997505ad3434f46e5ff1f4e225ed c9191eb8caba602d8decdbce8d4cbee30e2a8171 Implemented on release-1.2 in: 0666786ad87d3befd248cf7b8c59e686ac29af8e 65b1da8c532304142b763e9de5d00932ae438144 Flink / FLINK-5456 Add docs about new state and checkpointing interfaces Change By: Aljoscha Krettek Resolution: Fixed Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5681) Make ReaperThread for SafetyNetCloseableRegistry a singleton
Title: Message Title Stephan Ewen resolved as Fixed Fixed in 1.2.1 via c365a34b83e913d9c6c509627fae784435b056a2 1.3.0 via ec3eb593ae93123cf54cd34c452618d8bd0a7876 Flink / FLINK-5681 Make ReaperThread for SafetyNetCloseableRegistry a singleton Change By: Stephan Ewen Resolution: Fixed Fix Version/s: 1.2.1 Fix Version/s: 1.3.0 Status: Open Resolved Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-2168) Add HBaseTableSource
Title: Message Title
ASF GitHub Bot commented on FLINK-2168
Re: Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3149#discussion_r98487401
— Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSchema.java — @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.TypeExtractor; +import org.apache.flink.util.Preconditions; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; + +import java.io.Serializable; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.sql.Time; +import java.util.List; +import java.util.Map; +import java.util.HashMap; +import java.util.ArrayList; +import java.sql.Date; + +/** + * Helps to specify an HBase Table's schema + */ +public class HBaseTableSchema implements Serializable { + + // A Map with key as column family. + private final Map>> familyMap = — End diff –
> I think PushProjectIntoBatchTableSourceScanRule is not good enough for nested data types. but we can project at least family columns now.
So you mean we will project the column families. One thing to remember is that though we have two families - Eduction and Department. When we retrieve Department would come first and then Education. So what matters is how we retrieve the result.
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3149#discussion_r98487401
--- Diff:
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSchema.java
---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.sql.Time;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.ArrayList;
+import java.sql.Date;
+
+/**
+ * Helps to specify an HBase Table's schema
+ */
+public class HBaseTableSchema implements Serializable {
+
+ // A Map with key as column family.
+ private final Map>>
familyMap =
--- End diff --
> I think PushProjectIntoBatchTableSourceScanRule is not good enough for
nested data types. but we can project at least family columns now.
So you mean we will project the column families. One thing to remember is
that though we have two families - Eduction and Department. When we retrieve
Department would come first and then Education. So what matters is how we
retrieve the result.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3149#discussion_r98486712
--- Diff:
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSchema.java
---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.sql.Time;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.ArrayList;
+import java.sql.Date;
+
+/**
+ * Helps to specify an HBase Table's schema
+ */
+public class HBaseTableSchema implements Serializable {
+
+ // A Map with key as column family.
+ private final Map>>
familyMap =
+ new HashMap<>();
+
+ // Allowed types. This may change.
+ // TODO : Check if the Date type should be the one in java.util or the
one in java.sql
+ private static Class[] CLASS_TYPES = {
+ Integer.class, Short.class, Float.class, Long.class,
String.class, Byte.class, Boolean.class, Double.class, BigInteger.class,
BigDecimal.class, Date.class, Time.class, byte[].class
--- End diff --
Ok. I can do. I have not used much of these Immutblecollection. Infact in
our project we removed the dependency on this. Hence went on with a default
way.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] (FLINK-2168) Add HBaseTableSource
Title: Message Title
ASF GitHub Bot commented on FLINK-2168
Re: Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3149#discussion_r98486712
— Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSchema.java — @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.TypeExtractor; +import org.apache.flink.util.Preconditions; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; + +import java.io.Serializable; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.sql.Time; +import java.util.List; +import java.util.Map; +import java.util.HashMap; +import java.util.ArrayList; +import java.sql.Date; + +/** + * Helps to specify an HBase Table's schema + */ +public class HBaseTableSchema implements Serializable { + + // A Map with key as column family. + private final Map>> familyMap = + new HashMap<>(); + + // Allowed types. This may change. + // TODO : Check if the Date type should be the one in java.util or the one in java.sql + private static Class[] CLASS_TYPES = { + Integer.class, Short.class, Float.class, Long.class, String.class, Byte.class, Boolean.class, Double.class, BigInteger.class, BigDecimal.class, Date.class, Time.class, byte[].class — End diff –
Ok. I can do. I have not used much of these Immutblecollection. Infact in our project we removed the dependency on this. Hence went on with a default way.
Add Comment
[GitHub] flink pull request #3149: FLINK-2168 Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3149#discussion_r98486522
--- Diff:
flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSchema.java
---
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.flink.addons.hbase;
+
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.java.typeutils.TypeExtractor;
+import org.apache.flink.util.Preconditions;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+
+import java.io.Serializable;
+import java.math.BigDecimal;
+import java.math.BigInteger;
+import java.sql.Time;
+import java.util.List;
+import java.util.Map;
+import java.util.HashMap;
+import java.util.ArrayList;
+import java.sql.Date;
+
+/**
+ * Helps to specify an HBase Table's schema
+ */
+public class HBaseTableSchema implements Serializable {
+
+ // A Map with key as column family.
+ private final Map>>
familyMap =
--- End diff --
I just felt Map is not needed here. Anyway we have the Key as String. If
you are so particular then I can change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] (FLINK-5659) FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Title: Message Title
ASF GitHub Bot commented on FLINK-5659
Re: FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3219#discussion_r98486443
— Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java — @@ -148,14 +158,49 @@ public static void deleteDirectory(File directory) throws IOException { return; }
// delete the directory. this fails if the directory is not empty, meaning
// if new files got concurrently created. we want to fail then.
try { - Files.delete(directory.toPath()); - }
catch (NoSuchFileException ignored) {
// if someone else deleted this concurrently, we don't mind
// the result is the same for us, after all + java.nio.file.Path directoryPath = directory.toPath(); + if (OperatingSystem.isWindows()) { + // delete the directory. this fails if the directory is not empty, meaning + // if new files got concurrently created. we want to fail then. + try { + Files.delete(directoryPath); + }
catch (NoSuchFileException ignored) { + // if someone else deleted this concurrently, we don't mind + // the result is the same for us, after all + }
catch (AccessDeniedException e)
Unknown macro: { + // This may occur on Windows if another process is currently + // deleting the file, since the file must be opened in order + // to delete it. We double check here to make sure the file + // was actually deleted by another process. Note that this + // isn't a perfect solution, but it's better than nothing. + if (Files.exists(directoryPath)) { + throw e; + } + }
catch (DirectoryNotEmptyException e) { + // This may occur on Windows for some reason even for empty + // directories. Apparently there's a timing/visibility + // issue when concurrently deleting the contents of a directory + // and afterwards deleting the directory itself. + try { + Thread.sleep(50);
End diff –
[jira] (FLINK-2168) Add HBaseTableSource
Title: Message Title
ASF GitHub Bot commented on FLINK-2168
Re: Add HBaseTableSource
Github user ramkrish86 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3149#discussion_r98486522
— Diff: flink-connectors/flink-hbase/src/main/java/org/apache/flink/addons/hbase/HBaseTableSchema.java — @@ -0,0 +1,140 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.flink.addons.hbase; + +import org.apache.flink.api.common.typeinfo.TypeInformation; +import org.apache.flink.api.java.typeutils.TypeExtractor; +import org.apache.flink.util.Preconditions; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; + +import java.io.Serializable; +import java.math.BigDecimal; +import java.math.BigInteger; +import java.sql.Time; +import java.util.List; +import java.util.Map; +import java.util.HashMap; +import java.util.ArrayList; +import java.sql.Date; + +/** + * Helps to specify an HBase Table's schema + */ +public class HBaseTableSchema implements Serializable { + + // A Map with key as column family. + private final Map>> familyMap = — End diff –
I just felt Map is not needed here. Anyway we have the Key as String. If you are so particular then I can change.
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink pull request #3219: [FLINK-5659] Harden FileBaseUtils#deleteFileOrDire...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/3219#discussion_r98486443
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -148,14 +158,49 @@ public static void deleteDirectory(File directory)
throws IOException {
return;
}
- // delete the directory. this fails if the directory is
not empty, meaning
- // if new files got concurrently created. we want to
fail then.
- try {
- Files.delete(directory.toPath());
- }
- catch (NoSuchFileException ignored) {
- // if someone else deleted this concurrently,
we don't mind
- // the result is the same for us, after all
+ java.nio.file.Path directoryPath = directory.toPath();
+ if (OperatingSystem.isWindows()) {
+ // delete the directory. this fails if the
directory is not empty, meaning
+ // if new files got concurrently created. we
want to fail then.
+ try {
+ Files.delete(directoryPath);
+ } catch (NoSuchFileException ignored) {
+ // if someone else deleted this
concurrently, we don't mind
+ // the result is the same for us, after
all
+ } catch (AccessDeniedException e) {
+ // This may occur on Windows if another
process is currently
+ // deleting the file, since the file
must be opened in order
+ // to delete it. We double check here
to make sure the file
+ // was actually deleted by another
process. Note that this
+ // isn't a perfect solution, but it's
better than nothing.
+ if (Files.exists(directoryPath)) {
+ throw e;
+ }
+ } catch (DirectoryNotEmptyException e) {
+ // This may occur on Windows for some
reason even for empty
+ // directories. Apparently there's a
timing/visibility
+ // issue when concurrently deleting the
contents of a directory
+ // and afterwards deleting the
directory itself.
+ try {
+ Thread.sleep(50);
--- End diff --
It is intended that the thread hitting this exception never deletes the
directory. The assumption is that another thread has a correct view over the
directory and will actually delete it if possible.
We however must still check whether the directory is empty after *some*
time so that we actually fail if a new file was created concurrently, to
preserve existing behavior.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] (FLINK-5681) Make ReaperThread for SafetyNetCloseableRegistry a singleton
Title: Message Title Stephan Ewen closed an issue as Fixed Flink / FLINK-5681 Make ReaperThread for SafetyNetCloseableRegistry a singleton Change By: Stephan Ewen Status: Resolved Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink pull request #3230: [FLINK-5681] Make ReaperThread for SafetyNetClosea...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3230 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-5681) Make ReaperThread for SafetyNetCloseableRegistry a singleton
Title: Message Title ASF GitHub Bot commented on FLINK-5681 Re: Make ReaperThread for SafetyNetCloseableRegistry a singleton Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3230 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5480) User-provided hashes for operators
Title: Message Title Ufuk Celebi closed an issue as Fixed Fixed in 35da413 (release-1.2), 0de2bc3 (master). Flink / FLINK-5480 User-provided hashes for operators Change By: Ufuk Celebi Resolution: Fixed Fix Version/s: 1.3.0 Fix Version/s: 1.2.0 Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5659) FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Title: Message Title ASF GitHub Bot commented on FLINK-5659 Re: FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3219 I added a few more thoughts on how we could improve the behavior of the function. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3219: [FLINK-5659] Harden FileBaseUtils#deleteFileOrDirectory o...
Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3219 I added a few more thoughts on how we could improve the behavior of the function. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-5659) FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Title: Message Title
ASF GitHub Bot commented on FLINK-5659
Re: FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3219#discussion_r98482840
— Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java — @@ -148,14 +158,49 @@ public static void deleteDirectory(File directory) throws IOException { return; }
// delete the directory. this fails if the directory is not empty, meaning
// if new files got concurrently created. we want to fail then.
try { - Files.delete(directory.toPath()); - }
catch (NoSuchFileException ignored) {
// if someone else deleted this concurrently, we don't mind
// the result is the same for us, after all + java.nio.file.Path directoryPath = directory.toPath(); + if (OperatingSystem.isWindows()) { + // delete the directory. this fails if the directory is not empty, meaning + // if new files got concurrently created. we want to fail then. + try { + Files.delete(directoryPath); + }
catch (NoSuchFileException ignored) { + // if someone else deleted this concurrently, we don't mind + // the result is the same for us, after all + }
catch (AccessDeniedException e)
Unknown macro: { + // This may occur on Windows if another process is currently + // deleting the file, since the file must be opened in order + // to delete it. We double check here to make sure the file + // was actually deleted by another process. Note that this + // isn't a perfect solution, but it's better than nothing. + if (Files.exists(directoryPath)) { + throw e; + } + }
catch (DirectoryNotEmptyException e) { + // This may occur on Windows for some reason even for empty + // directories. Apparently there's a timing/visibility + // issue when concurrently deleting the contents of a directory + // and afterwards deleting the directory itself. + try { + Thread.sleep(50);
End diff –
[jira] (FLINK-5659) FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Title: Message Title
ASF GitHub Bot commented on FLINK-5659
Re: FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3219#discussion_r98477980
— Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java — @@ -116,6 +118,14 @@ else if (file.exists()) { } catch (NoSuchFileException e) { // if the file is already gone (concurrently), we don't mind + }
catch (AccessDeniedException e) { — End diff –
Can we special case this also for Windows? Should never occur on Unix (and should throw an exception if it happens)...
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5659) FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Title: Message Title
ASF GitHub Bot commented on FLINK-5659
Re: FileBaseUtils#deleteFileOrDirectory not thread-safe on Windows
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3219#discussion_r98478136
— Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java — @@ -148,14 +158,49 @@ public static void deleteDirectory(File directory) throws IOException { return; }
// delete the directory. this fails if the directory is not empty, meaning
// if new files got concurrently created. we want to fail then.
try { - Files.delete(directory.toPath()); - }
catch (NoSuchFileException ignored) {
// if someone else deleted this concurrently, we don't mind
// the result is the same for us, after all + java.nio.file.Path directoryPath = directory.toPath(); + if (OperatingSystem.isWindows()) { + // delete the directory. this fails if the directory is not empty, meaning
End diff –
How about moving this to a separate method for readability?
Add Comment
[GitHub] flink pull request #3219: [FLINK-5659] Harden FileBaseUtils#deleteFileOrDire...
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3219#discussion_r98478136
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -148,14 +158,49 @@ public static void deleteDirectory(File directory)
throws IOException {
return;
}
- // delete the directory. this fails if the directory is
not empty, meaning
- // if new files got concurrently created. we want to
fail then.
- try {
- Files.delete(directory.toPath());
- }
- catch (NoSuchFileException ignored) {
- // if someone else deleted this concurrently,
we don't mind
- // the result is the same for us, after all
+ java.nio.file.Path directoryPath = directory.toPath();
+ if (OperatingSystem.isWindows()) {
+ // delete the directory. this fails if the
directory is not empty, meaning
--- End diff --
How about moving this to a separate method for readability?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3219: [FLINK-5659] Harden FileBaseUtils#deleteFileOrDire...
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3219#discussion_r98482840
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -148,14 +158,49 @@ public static void deleteDirectory(File directory)
throws IOException {
return;
}
- // delete the directory. this fails if the directory is
not empty, meaning
- // if new files got concurrently created. we want to
fail then.
- try {
- Files.delete(directory.toPath());
- }
- catch (NoSuchFileException ignored) {
- // if someone else deleted this concurrently,
we don't mind
- // the result is the same for us, after all
+ java.nio.file.Path directoryPath = directory.toPath();
+ if (OperatingSystem.isWindows()) {
+ // delete the directory. this fails if the
directory is not empty, meaning
+ // if new files got concurrently created. we
want to fail then.
+ try {
+ Files.delete(directoryPath);
+ } catch (NoSuchFileException ignored) {
+ // if someone else deleted this
concurrently, we don't mind
+ // the result is the same for us, after
all
+ } catch (AccessDeniedException e) {
+ // This may occur on Windows if another
process is currently
+ // deleting the file, since the file
must be opened in order
+ // to delete it. We double check here
to make sure the file
+ // was actually deleted by another
process. Note that this
+ // isn't a perfect solution, but it's
better than nothing.
+ if (Files.exists(directoryPath)) {
+ throw e;
+ }
+ } catch (DirectoryNotEmptyException e) {
+ // This may occur on Windows for some
reason even for empty
+ // directories. Apparently there's a
timing/visibility
+ // issue when concurrently deleting the
contents of a directory
+ // and afterwards deleting the
directory itself.
+ try {
+ Thread.sleep(50);
--- End diff --
I am wondering if it makes sense to have a something like 10-20 retries
here, with 5ms delay.
The current implementation also does not remove the directory when
initially seeing the `DirectoryNotEmptyException`, it simply suppresses the
exception when the directory ends up empty 50ms later, but leaves the directory
itself there.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #3219: [FLINK-5659] Harden FileBaseUtils#deleteFileOrDire...
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/3219#discussion_r98477980
--- Diff: flink-core/src/main/java/org/apache/flink/util/FileUtils.java ---
@@ -116,6 +118,14 @@ else if (file.exists()) {
}
catch (NoSuchFileException e) {
// if the file is already gone (concurrently),
we don't mind
+ } catch (AccessDeniedException e) {
--- End diff --
Can we special case this also for Windows? Should never occur on Unix (and
should throw an exception if it happens)...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] (FLINK-5686) Print meaningful message when JM is lost
Title: Message Title Ufuk Celebi created an issue Flink / FLINK-5686 Print meaningful message when JM is lost Issue Type: Improvement Assignee: Unassigned Components: Client Created: 30/Jan/17 16:39 Priority: Minor Reporter: Ufuk Celebi When a leading JM dies, the client prints: New JobManager elected. Connecting to null This is not helpful to the user. Instead it should inform the user that no JobManager is available currently and that the client will reconnect as soon as one is re-elected. Add Comment
[jira] (FLINK-5480) User-provided hashes for operators
Title: Message Title Stefan Richter commented on FLINK-5480 Re: User-provided hashes for operators From my side yes. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5044) Converting operator and function state from Flink 1.1 for all changed operators in 1.2
Title: Message Title Aljoscha Krettek closed an issue as Fixed Flink / FLINK-5044 Converting operator and function state from Flink 1.1 for all changed operators in 1.2 Change By: Aljoscha Krettek Resolution: Fixed Fix Version/s: 1.2.0 Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5480) User-provided hashes for operators
Title: Message Title Aljoscha Krettek commented on FLINK-5480 Re: User-provided hashes for operators I think this one can be closed, right Ufuk Celebi and Stefan Richter? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title ASF GitHub Bot commented on FLINK-1731 Re: Add kMeans clustering algorithm to machine learning library Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3192 We could think about adding examples to flink-ml itself. But this is not strictly necessary if the API and its usage is documented. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's algorithm)
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3192 We could think about adding examples to flink-ml itself. But this is not strictly necessary if the API and its usage is documented. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title ASF GitHub Bot commented on FLINK-1731 Re: Add kMeans clustering algorithm to machine learning library Github user sachingoel0101 commented on the issue: https://github.com/apache/flink/pull/3192 I'm not sure about adding examples under flink-examples project. None of the other ml algorithms are there either. Also, requires adding flink-ml as a dependency in the pom. I can however add a kmeans section under ML docs Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's algorithm)
Github user sachingoel0101 commented on the issue: https://github.com/apache/flink/pull/3192 I'm not sure about adding examples under flink-examples project. None of the other ml algorithms are there either. Also, requires adding flink-ml as a dependency in the pom. I can however add a kmeans section under ML docs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title
ASF GitHub Bot commented on FLINK-1731
Re: Add kMeans clustering algorithm to machine learning library
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3192#discussion_r98471870
— Diff: flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/clustering/KMeans.scala — @@ -0,0 +1,263 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.ml.clustering + +import org.apache.flink.api.java.functions.FunctionAnnotation.ForwardedFields +import org.apache.flink.api.scala. {DataSet, _}
+import org.apache.flink.ml._ +import org.apache.flink.ml.common. {LabeledVector, _}
+import org.apache.flink.ml.math.Breeze._ +import org.apache.flink.ml.math. {BLAS, Vector}
+import org.apache.flink.ml.metrics.distances.EuclideanDistanceMetric +import org.apache.flink.ml.pipeline._ + + +/** + * Implements the KMeans algorithm which calculates cluster centroids based on set of training data + * points and a set of k initial centroids. — End diff –
For the purpose of this PR, the centroids are assumed to be provided by the user. That automatically sets a default k. I will take this into consideration after we have set up the complete initialization architecture. Without some good initialization schemes, it's not very useful to compare different values of k.
Add Comment
This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title ASF GitHub Bot commented on FLINK-1731 Re: Add kMeans clustering algorithm to machine learning library Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3192 @skonto you're right that we should also add a corresponding kmeans documentation under `docs/dev/libs/ml/kmeans.md` and link it from the `docs/dev/libs/ml/index.md` file. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's algorithm)
Github user tillrohrmann commented on the issue: https://github.com/apache/flink/pull/3192 @skonto you're right that we should also add a corresponding kmeans documentation under `docs/dev/libs/ml/kmeans.md` and link it from the `docs/dev/libs/ml/index.md` file. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-1583) TaskManager reregistration in case of a restart
Title: Message Title Till Rohrmann closed an issue as Fixed TaskManager registration is now scoped by the ResourceID which should be unique wrt the container in which the TaskManager was started. Flink / FLINK-1583 TaskManager reregistration in case of a restart Change By: Till Rohrmann Resolution: Fixed Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink pull request #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's alg...
Github user sachingoel0101 commented on a diff in the pull request:
https://github.com/apache/flink/pull/3192#discussion_r98471870
--- Diff:
flink-libraries/flink-ml/src/main/scala/org/apache/flink/ml/clustering/KMeans.scala
---
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.clustering
+
+import
org.apache.flink.api.java.functions.FunctionAnnotation.ForwardedFields
+import org.apache.flink.api.scala.{DataSet, _}
+import org.apache.flink.ml._
+import org.apache.flink.ml.common.{LabeledVector, _}
+import org.apache.flink.ml.math.Breeze._
+import org.apache.flink.ml.math.{BLAS, Vector}
+import org.apache.flink.ml.metrics.distances.EuclideanDistanceMetric
+import org.apache.flink.ml.pipeline._
+
+
+/**
+ * Implements the KMeans algorithm which calculates cluster centroids
based on set of training data
+ * points and a set of k initial centroids.
--- End diff --
For the purpose of this PR, the centroids are assumed to be provided by the
user. That automatically sets a default k.
I will take this into consideration after we have set up the complete
initialization architecture. Without some good initialization schemes, it's not
very useful to compare different values of k.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] (FLINK-5628) CheckpointStatsTracker implements Serializable but isn't
Title: Message Title ASF GitHub Bot commented on FLINK-5628 Re: CheckpointStatsTracker implements Serializable but isn't Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3215 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-1407) Enable log output (error level) for test cases
Title: Message Title Till Rohrmann closed an issue as Fixed This should already be covered by Flink's watchdog script. Flink / FLINK-1407 Enable log output (error level) for test cases Change By: Till Rohrmann Resolution: Fixed Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-2273) Register special Writable Kryo serializer
Title: Message Title Till Rohrmann closed an issue as Fixed Agreed Robert Metzger Flink / FLINK-2273 Register special Writable Kryo serializer Change By: Till Rohrmann Resolution: Fixed Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's algorithm)
Github user skonto commented on the issue: https://github.com/apache/flink/pull/3192 @sachingoel0101 please add an example at least under flink examples. Is the code documentation enough? @thvasilo ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title ASF GitHub Bot commented on FLINK-1731 Re: Add kMeans clustering algorithm to machine learning library Github user skonto commented on the issue: https://github.com/apache/flink/pull/3192 @sachingoel0101 please add an example at least under flink examples. Is the code documentation enough? @thvasilo ? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-1541) GSoC project: Distributed profiling
Title: Message Title Till Rohrmann closed an issue as Won't Fix Flink / FLINK-1541 GSoC project: Distributed profiling Change By: Till Rohrmann Resolution: Won't Fix Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink pull request #3215: [FLINK-5628] [webfrontend] Fix serialization of st...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3215 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-1731) Add kMeans clustering algorithm to machine learning library
Title: Message Title ASF GitHub Bot commented on FLINK-1731 Re: Add kMeans clustering algorithm to machine learning library Github user skonto commented on the issue: https://github.com/apache/flink/pull/3192 @sachingoel0101 I am having a look. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3192: [FLINK-1731][ml] Add KMeans clustering(Lloyd's algorithm)
Github user skonto commented on the issue: https://github.com/apache/flink/pull/3192 @sachingoel0101 I am having a look. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-5628) CheckpointStatsTracker implements Serializable but isn't
Title: Message Title Ufuk Celebi closed an issue as Fixed 3e2e49f (release-1.2), dcfa3fb (master). Flink / FLINK-5628 CheckpointStatsTracker implements Serializable but isn't Change By: Ufuk Celebi Resolution: Fixed Fix Version/s: 1.2.1 Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-3715) Move Accumulating/Discarding from Trigger to WindowOperator
Title: Message Title Aljoscha Krettek updated an issue Flink / FLINK-3715 Move Accumulating/Discarding from Trigger to WindowOperator Change By: Aljoscha Krettek Issue Type: Improvement Sub-task Parent: FLINK-4407 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5628) CheckpointStatsTracker implements Serializable but isn't
Title: Message Title ASF GitHub Bot commented on FLINK-5628 Re: CheckpointStatsTracker implements Serializable but isn't Github user uce commented on the issue: https://github.com/apache/flink/pull/3215 Thanks for review. I'm going to merge this to `release-1.2` as well. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[GitHub] flink issue #3215: [FLINK-5628] [webfrontend] Fix serialization of stats tra...
Github user uce commented on the issue: https://github.com/apache/flink/pull/3215 Thanks for review. I'm going to merge this to `release-1.2` as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] (FLINK-5415) ContinuousFileProcessingTest failed on travis
Title: Message Title Ufuk Celebi commented on FLINK-5415 Re: ContinuousFileProcessingTest failed on travis Kostas Kloudas Do you have an idea what this could be? I had it again: https://s3.amazonaws.com/archive.travis-ci.org/jobs/195820556/log.txt Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5415) ContinuousFileProcessingTest failed on travis
Title: Message Title Ufuk Celebi updated an issue Flink / FLINK-5415 ContinuousFileProcessingTest failed on travis Change By: Ufuk Celebi Labels: test-stability Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-4415) Enhance Trigger Interface with onFire()
Title: Message Title Aljoscha Krettek closed an issue as Won't Fix With the change to onMerge() that doesn't allow returning a TriggerResult anymore we don't really need this change, it would just be API sugar. Flink / FLINK-4415 Enhance Trigger Interface with onFire() Change By: Aljoscha Krettek Resolution: Won't Fix Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-1098) flatArray() operator that converts arrays to elements
Title: Message Title Aljoscha Krettek closed an issue as Won't Fix This can be done better in other places, probably Table API. Flink / FLINK-1098 flatArray() operator that converts arrays to elements Change By: Aljoscha Krettek Resolution: Won't Fix Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-3643) Improve Windowing
Title: Message Title Aljoscha Krettek closed an issue as Fixed All the subtasks where implemented at some point. Flink / FLINK-3643 Improve Windowing Change By: Aljoscha Krettek Resolution: Fixed Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-3715) Move Accumulating/Discarding from Trigger to WindowOperator
Title: Message Title Aljoscha Krettek commented on FLINK-3715 Re: Move Accumulating/Discarding from Trigger to WindowOperator I'm not sure whether this is wanted anymore. We should probably close this and deal with ACCUMULATING/DISCARDING at the Trigger DSL level. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-3715) Move Accumulating/Discarding from Trigger to WindowOperator
Title: Message Title Aljoscha Krettek updated an issue Flink / FLINK-3715 Move Accumulating/Discarding from Trigger to WindowOperator Change By: Aljoscha Krettek Issue Type: Sub-task Improvement Parent: FLINK-3643 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-1098) flatArray() operator that converts arrays to elements
Title: Message Title Timo Walther commented on FLINK-1098 Re: flatArray() operator that converts arrays to elements Yes, you are right. I would also rather add it to the Table API. +1 for closing this. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-2284) Confusing/inconsistent PartitioningStrategy
Title: Message Title Aljoscha Krettek commented on FLINK-2284 Re: Confusing/inconsistent PartitioningStrategy Stephan Ewen What can we do about solving this issue? Is it still an issue? I haven't heard anything about this for over a year now. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-4407) Add DSL for specifying Window Triggers
Title: Message Title Aljoscha Krettek updated an issue Flink / FLINK-4407 Add DSL for specifying Window Triggers Change By: Aljoscha Krettek Summary: Implement the trigger Add DSL for specifying Window Triggers Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-4407) Add DSL for specifying Window Triggers
Title: Message Title Aljoscha Krettek updated an issue Flink / FLINK-4407 Add DSL for specifying Window Triggers Change By: Aljoscha Krettek Issue Type: Sub-task New Feature Parent: FLINK-3643 Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-3237) Create AsynchronousKvStateSnapshot similar to AsynchronousStateHandle
Title: Message Title Aljoscha Krettek closed an issue as Won't Fix It's possible for keyed state backends to to asynchronous snapshots as of a while ago. Flink / FLINK-3237 Create AsynchronousKvStateSnapshot similar to AsynchronousStateHandle Change By: Aljoscha Krettek Resolution: Won't Fix Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-2983) Add StateLoader as an alternative to default KvState
Title: Message Title Aljoscha Krettek closed an issue as Won't Fix This has not come up for a very long time and it seems the current interfaces work fine. Please re-open if you think otherwise. Flink / FLINK-2983 Add StateLoader as an alternative to default KvState Change By: Aljoscha Krettek Resolution: Won't Fix Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-2923) Make it possible to mix-and-match StateBackends with KvState implementations
Title: Message Title Aljoscha Krettek closed an issue as Not A Problem This has become obsolete with changes in how the state API works. I.e. we now have different backends for keyed state and operator state. Flink / FLINK-2923 Make it possible to mix-and-match StateBackends with KvState implementations Change By: Aljoscha Krettek Resolution: Not A Problem Status: In Progress Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-5645) IOMetrics transfer through ExecGraph does not work for failed jobs
Title: Message Title Chesnay Schepler assigned an issue to Chesnay Schepler Flink / FLINK-5645 IOMetrics transfer through ExecGraph does not work for failed jobs Change By: Chesnay Schepler Assignee: Chesnay Schepler Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-4275) Generic Folding, Reducing and List states behave differently from other state backends
Title: Message Title Aljoscha Krettek commented on FLINK-4275 Re: Generic Folding, Reducing and List states behave differently from other state backends Gyula Fora Is this still an issue? I noticed that the generic state implementations where recently removed and you mentioned once that you were using them in your code. Is that still the case? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-3606) Expose Configuration from RichFunctions
Title: Message Title Aljoscha Krettek commented on FLINK-3606 Re: Expose Configuration from RichFunctions I think the general direction is towards not having access to configurations in the user function and instead putting stuff into UDF fields that will be serialised with the function. Robert MetzgerMarton Balassi, do you think we can close this issue? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-3039) Trigger KeyValueState cannot be Scala Int
Title: Message Title Aljoscha Krettek closed an issue as Won't Fix This was solved by changes in the Trigger API starting with Flink 1.1. Flink / FLINK-3039 Trigger KeyValueState cannot be Scala Int Change By: Aljoscha Krettek Resolution: Won't Fix Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-1146) Add a way to provide types for generic type variables
Title: Message Title Timo Walther commented on FLINK-1146 Re: Add a way to provide types for generic type variables I think we can close this issue for now. It is pretty old and only contains a very rough description. TypeHints have solved most of those problems. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (FLINK-1146) Add a way to provide types for generic type variables
Title: Message Title Timo Walther closed an issue as Won't Fix Flink / FLINK-1146 Add a way to provide types for generic type variables Change By: Timo Walther Resolution: Won't Fix Status: Open Closed Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
