[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679254#comment-14679254 ] ASF GitHub Bot commented on FLINK-1528: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-129222936 @balidani, I think it'd be better if you close this PR. I don't think we'll add another example after #1000 is merged. I can take over and probably reuse some of your code to add a local clustering coefficient library method. Would that be OK? Thanks! Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679264#comment-14679264 ] ASF GitHub Bot commented on FLINK-1528: --- Github user balidani closed the pull request at: https://github.com/apache/flink/pull/420 Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679263#comment-14679263 ] ASF GitHub Bot commented on FLINK-1528: --- Github user balidani commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-129223724 @vasia yes, I'm sorry about not finishing it, but I just did not have the time lately. Cheers! Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14679276#comment-14679276 ] ASF GitHub Bot commented on FLINK-1528: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-129226158 Thats' fine :) Thanks for the fast response! Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14626076#comment-14626076 ] ASF GitHub Bot commented on FLINK-1528: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-121174894 Any progress on this pull request, or should it be closed? Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570573#comment-14570573 ] ASF GitHub Bot commented on FLINK-1528: --- Github user balidani commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-108284218 Yeah, I should definitely finish this! I'll take a look tonight, sorry about that :) Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14570567#comment-14570567 ] ASF GitHub Bot commented on FLINK-1528: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-108283573 Hey @balidani! Would you like to finish this up? It's not really urgent, but it's almost finished and it'd be a pity to abandon :) Someone else could also take over of course. Just let us know! Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14377816#comment-14377816 ] ASF GitHub Bot commented on FLINK-1528: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-85487676 Hi @balidani! I think it's still a bit confusing how this example works. As far as I understand, you expect a directed graph as input, but then you convert it to an undirected one and compute the clustering coefficient of the undirected graph, right? We should either document this behavior clearly in the example description or compute the result for the input we expect. I personally prefer the second :-) Also, take a look in the other examples and write a short usage description in the beginning of the example, including the input format that you expect when args are provided. Let me know if you have questions! Thanks! Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14371614#comment-14371614 ] ASF GitHub Bot commented on FLINK-1528: --- Github user balidani commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-84068746 Hi @vasia! I fixed the algorithm, now it will convert all edges to a pair of edges and call `distinct` on the edge set. This gives the correct results now. Thanks! Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351547#comment-14351547 ] ASF GitHub Bot commented on FLINK-1528: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-77687686 Hi @balidani! Thanks for the example :)) I'm a bit confused about the directed / undirected case. I tried testing with both a directed and an undirected input and both my tests failed.. See [here](http://gist.github.com/vasia/ad68268118b6f895cf7c#file-lcc-example-tests) are the test cases I tried. I suppose the directed case doesn't work because you only consider the out-neighbors, when you should count all. And for the undirected case, I think that the division by 2 you're making is giving a wrong result, because you're not counting edges, just out-neighbors neighbors. Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[
https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351544#comment-14351544
]
ASF GitHub Bot commented on FLINK-1528:
---
Github user vasia commented on a diff in the pull request:
https://github.com/apache/flink/pull/420#discussion_r25995666
--- Diff:
flink-staging/flink-gelly/src/main/java/org/apache/flink/graph/example/LocalClusteringCoefficientExample.java
---
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * License); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an AS IS BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.graph.example;
+
+import org.apache.flink.api.common.ProgramDescription;
+import org.apache.flink.api.common.functions.MapFunction;
+import org.apache.flink.api.java.DataSet;
+import org.apache.flink.api.java.ExecutionEnvironment;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.graph.Edge;
+import org.apache.flink.graph.EdgeDirection;
+import org.apache.flink.graph.EdgesFunction;
+import org.apache.flink.graph.Graph;
+import org.apache.flink.graph.NeighborsFunctionWithVertexValue;
+import org.apache.flink.graph.Vertex;
+import org.apache.flink.graph.example.utils.LocalClusteringCoefficientData;
+import org.apache.flink.types.NullValue;
+
+import java.util.HashSet;
+
+public class LocalClusteringCoefficientExample implements
ProgramDescription {
+
+ //
+ // Program
+ //
+
+ public static void main (String [] args) throws Exception {
+
+ if(!parseParameters(args)) {
+ return;
+ }
+
+ ExecutionEnvironment env =
ExecutionEnvironment.getExecutionEnvironment();
+
+ DataSetEdgeLong, NullValue edges = getEdgeDataSet(env);
+ GraphLong, NullValue, NullValue graph =
Graph.fromDataSet(edges, env);
+
+ // Get the neighbors of each vertex in a HashSet
+ DataSetTuple2Long, HashSetLong neighborhoods = graph
+ .reduceOnEdges(new NeighborhoodEdgesFunction(),
EdgeDirection.OUT);
+
+ // Construct a new graph where the neighborhood is the vertex
value
+ GraphLong, HashSetLong, NullValue newGraph = graph
+ .mapVertices(new EmptyVertexMapFunction())
+ .joinWithVertices(neighborhoods, new
NeighborhoodVertexMapFunction());
+
+ // Calculate clustering coefficient
+ DataSetTuple2Long, Double clusteringCoefficients = newGraph
+ .reduceOnNeighbors(new
ClusteringCoefficientNeighborsFunction(), EdgeDirection.OUT);
+
+ // Emit results
+ if(fileOutput) {
+ clusteringCoefficients.writeAsCsv(outputPath, \n,
,);
+ } else {
+ clusteringCoefficients.print();
+ }
+
+ env.execute(Local Clustering Coefficient Example);
+ }
+
+ //
+ // Clustering Coefficient Functions
+ //
+
+ private static final class NeighborhoodEdgesFunction
+ implements EdgesFunctionLong, NullValue, Tuple2Long,
HashSetLong {
+
+ @Override
+ public Tuple2Long, HashSetLong iterateEdges(
+ IterableTuple2Long, EdgeLong, NullValue
edges) throws Exception {
+
+ Long vertexId = null;
+ HashSetLong neighbors = new HashSetLong();
+
+ for (Tuple2Long, EdgeLong, NullValue edge : edges) {
+
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333524#comment-14333524 ] ASF GitHub Bot commented on FLINK-1528: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-75586167 Does anyone have an idea about this? Is there a way to pass the `HashSetK` type? In any case, even if not, I think it doesn't matter in this case. I believe we shouldn't add this as a library method, as it is a quite naive implementation of local clustering coefficient. I'd prefer if we try to keep the library methods as efficient as possible. However, I would definitely add this as an example, since it very nicely demonstrates how to use neighborhood methods and `joinWithVertices` (which are missing from the other examples). So, I would suggest we change this to an example that uses a sample dataset, with e.g. Long ids and also allows file input. What do you think @balidani? Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333538#comment-14333538 ] ASF GitHub Bot commented on FLINK-1528: --- Github user balidani commented on the pull request: https://github.com/apache/flink/pull/420#issuecomment-75588338 Hi! I agree. I will change the example then :) Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14327431#comment-14327431 ] ASF GitHub Bot commented on FLINK-1528: --- Github user balidani closed the pull request at: https://github.com/apache/flink/pull/400 Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[
https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322906#comment-14322906
]
ASF GitHub Bot commented on FLINK-1528:
---
Github user vasia commented on the pull request:
https://github.com/apache/flink/pull/400#issuecomment-74527767
Right, so let me copy:
The problem is here:
```
DataSetTuple2K, HashSetK neighborhoods = input.reduceOnEdges(new
NeighborhoodEdgesFunctionK(), EdgeDirection.OUT);
```
and we try to get the return type `Tuple2K, HashSetK` like this:
```
public TypeInformationT getProducedType() {
return TypeExtractor.createTypeInfo(EdgesFunction.class,
function.getClass(), 2, null, null);
}
```
Anyone have an idea? Is it because of the nested type parameter in
`Tuple2K, HashSetK` ?
Add local clustering coefficient library method and example
---
Key: FLINK-1528
URL: https://issues.apache.org/jira/browse/FLINK-1528
Project: Flink
Issue Type: Task
Components: Gelly
Reporter: Vasia Kalavri
Assignee: Daniel Bali
Add a gelly library method and example to compute the local clustering
coefficient.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322649#comment-14322649 ] ASF GitHub Bot commented on FLINK-1528: --- Github user vasia commented on the pull request: https://github.com/apache/flink/pull/400#issuecomment-74493011 Hi @balidani! Thanks for the PR. The commit history is a bit messed up, but we'll fix that. It's just the last commit that's relevant, right? If I recall correctly, you are having some type issues. Could you please give us some details on what is the error you are getting and what seems to be causing it? Thanks! Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1528) Add local clustering coefficient library method and example
[ https://issues.apache.org/jira/browse/FLINK-1528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322148#comment-14322148 ] ASF GitHub Bot commented on FLINK-1528: --- GitHub user balidani opened a pull request: https://github.com/apache/flink/pull/400 [FLINK-1528] Added local clustering coefficient example (failing) As @vasia requested, here is the current version of local clustering coefficient for Gelly. It causes exceptions, so this PR is only a draft. You can merge this pull request into a Git repository by running: $ git pull https://github.com/balidani/flink master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/400.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #400 commit c2b19492932e1fe4f8ade1319d8dd4786bfc257e Author: Daniel Bali balijanosdan...@gmail.com Date: 2015-02-13T11:35:50Z [FLINK-1528] Added local clustering coefficient example (failing) commit 39f88893ecf47dbcab3f1057f526bbf014c3086f Author: Gyula Fora gyf...@apache.org Date: 2015-02-11T13:19:22Z [FLINK-1517] [streaming] Added indexed input iterator for streaming commit c519ddfc64bc33474f6b8575fc051a50ab7fb0e5 Author: mbalassi mbala...@apache.org Date: 2015-02-12T21:20:14Z [streaming] Added ITCase for streaming classloading commit 7c35e748f1b17fe86c7faea4712f673fecbeea8f Author: Henry Saputra henry.sapu...@gmail.com Date: 2015-02-13T17:09:26Z Small cleanups to add space between if-else keyword and parentheses to be consistent as I found them. Also fix some comments typos. commit b0d4e94a709e40a38bba778dd32b97c72a407d06 Author: Stephan Ewen se...@apache.org Date: 2015-02-11T18:01:41Z [FLINK-1529] [jobmanager] Improve error handling on JobManager startup This closes #385 commit 7d5aa836e9af88b46957e08986073db3fa4a7ef6 Author: Stephan Ewen se...@apache.org Date: 2015-02-13T11:22:35Z [FLINK-1532] [tests] Fix spurious failure in AggregatorsITCase (plus minor cleanups) commit aeaa82c7a874a8e65c3615e3b4872c80dc426a6b Author: Till Rohrmann trohrm...@apache.org Date: 2015-02-10T11:21:33Z [FLINK-1508] [runtime] Removes AkkaUtil.ask and replaces respective calls with explicit future handling. Removes blocking calls for ActorRef retrieval in actors. This closes #384. commit 2052be6d864b0d163b3212ed20625a77399029df Author: Till Rohrmann trohrm...@apache.org Date: 2015-02-12T10:49:09Z [FLINK-1543] [runtime] Adds try and catch blocks around all method calls from an actor's receive method which can throw an exception. Sets the StoppingSupervisorStrategy as default for all guardians. This closes #394. commit 1162266a270a3c4508393073bf27dc8c5a656ed2 Author: Chiwan Park chiwanp...@icloud.com Date: 2015-02-08T06:36:19Z [FLINK-1179] [jobmanager] Add button to JobManager web interface to request stack trace of a TaskManager This closes #374 Add local clustering coefficient library method and example --- Key: FLINK-1528 URL: https://issues.apache.org/jira/browse/FLINK-1528 Project: Flink Issue Type: Task Components: Gelly Reporter: Vasia Kalavri Assignee: Daniel Bali Add a gelly library method and example to compute the local clustering coefficient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
