Re: Gelly Scatter-Gather Iteration, In a single superstep, GatherFunction.updateVertex invoked more then once

Would you perchance have an example program to demonstrate the unexpected behavior? Does this issue always manifest or are you only seeing duplicate calls under specific circumstances? On Mon, Oct 22, 2018 at 8:33 AM 曹建华 wrote: > Hi: > According to the code comment, in Scatter-Gather Iteration,

Re: Plain text SSL passwords in Log file

With the current method you always have the risk, no matter which keywords you filter on ("secret", "password", etc.), that the key name is mistyped and inadvertently logged. Perhaps we could implement something like TravisCI's encryption keys [ https://docs.travis-ci.com/user/encryption-keys/]

Re: Unable to see more than 5 jobs on Flink Dashboard

What version of Flink are you running? Deployment method? Referenced section of flink-conf.yaml? On Wed, Mar 28, 2018 at 4:34 PM, Vinay Patil wrote: > Hi, > > I am not able to see more than 5 jobs on Flink Dashboard. > I have set web.history to 50 in flink-conf.yaml

Re: TaskManager crashes with PageRank algorithm in Gelly

Termination of the TaskManager by the Linux OOM killer indicates an overallocation of memory and you have set "taskmanager.heap.mb: 139264” on machines with 136 GB. Even though you were able to (temporarily?) resolve the issue by enabling preallocation, you may see degraded performance if

Re: Gelly: akka.ask.timeout

Hi Alieh, Are you able to run the example WordCount application without losing TaskManagers? Greg > On Jan 8, 2018, at 7:48 AM, Alieh Saeedi wrote: > > Hey all, > I have an iterative algorithm implemented in Gelly. As long as I upgraded > everything to flink-1.3.1 from

Re: Flink Batch Performance degradation at scale

Hi Garrett, In the Web UI, when viewing a job under overview / subtasks, selecting the checkbox "Aggregate task statistics by TaskManager” will reduce the number of displayed rows (though in your case only by half). The following documents profiling a Flink job with Java Flight Recorder:

Re: Flink memory usage

I’ve used the following simple script to capture Flink metrics by running: python -u ./statsd_server.py 9020 > statsd_server.log >>> flink-conf.yaml metrics.reporters: statsd_reporter metrics.reporter.statsd_reporter.class: org.apache.flink.metrics.statsd.StatsDReporter

Re: Task Manager was lost/killed due to full GC

Late response, but a common reason for disappearing TaskManagers is termination by the Linux out-of-memory killer, with the recommendation to decrease the allotted memory. > On Sep 5, 2017, at 9:09 AM, ShB wrote: > > Hi, > > I'm running a Flink batch job that

Re: TypeInformation in Custom Deserializer

You should be able to implement this using a TypeHint (see the Creating a TypeInformation or TypeSerializer section from the linked page): return TypeInformation.of(new TypeHint>(){}); > On Aug 13, 2017, at 10:31 AM, AndreaKinn

Re: PageRank iteration

PageRank is using a bulk iteration via DataSet#iterate whereas a delta iteration would start with DataSet#iterateDelta. > On Aug 13, 2017, at 10:30 AM, Kaepke, Marc wrote: > > Hi everyone, > > does PageRank use bulk or delta iteration? > > I mean the

Re: Standalone cluster - taskmanager settings ignored

Hi Marc, By chance did you edit the slaves file before shutting down the cluster? If so, then the removed worker would not be stopped and would reconnect to the restarted JobManager. Greg > On Aug 11, 2017, at 11:25 AM, Kaepke, Marc wrote: > > Hi, > > I have a

Re: Gelly PageRank implementations in 1.2 to 1.3

> >> Thanks for your explanation. >> >> The vertex-centric, sg and gsa PageRank need a Double as vertex value. A >> VertexDegree function generate a vertex with a LongValue as value. >> Maybe I can iterate over the graph and remove all edges with a degree of &

Re: Gelly PageRank implementations in 1.2 to 1.3

rg 1, val 2 > src 2, trg 1, val 3 > src 2, trg 4, val 6 > > Ranking output > vertex-centric > id 4 with 1.16 > id 1 with 1.103 > id 2 with 0.815 > id 3 with 0 > sg and gsa > id 4 with 2.208 > id 1 with 2.114 > id 2 with 1.546 > id 3 with 0 > new PageRank in

Re: Gelly PageRank implementations in 1.2 to 1.3

Hi Marc, PageRank and GSAPageRank were moved to the flink-gelly-examples jar in the org.apache.flink.graph.examples package. A library algorithm was added that supports both source and sink vertices. This limitation of the old algorithms was noted in the class documentation and I understand to

Re: delta iteration

Hi Alieh, From a rich function call getIterationRuntimeContext().getSuperstepNumber() Greg > On Jul 12, 2017, at 9:56 AM, Alieh wrote: > > Hello all, > > I need iteration number in delta iteration (or any kind of counter). Is there > anyway to implement or

Re: Problem with Summerization

(0002,(2,1)) > (0008,(8,1)) > (0004,(9,7)) > (000a,(10,1)) > > g.getEdges.print(); //error!! > > As I said, I also tested when edges have values other than null > and the same problem appears.

Re: Problem with Summerization

Hi Ali, Could you print and include a gellyGraph which results in this error. Greg > On Jun 27, 2017, at 2:48 PM, rost...@informatik.uni-leipzig.de wrote: > > Dear All, > > I do not understand what the error in the following code can be? > > Graph gellyGraph =

Re: Performance Improvement on Flink 1.2.0

Some documentation on application profiling with Flink 1.3 (can be manually inserted into the scripts for Flink 1.2): https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/application_profiling.html > On Jun 22, 2017, at 9:24 AM, Stefan Richter

Re: Kafka and Flink integration

TypeInformation was specified. Greg > On Jun 21, 2017, at 6:21 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > Greg: > Can you clarify he last part? > Should it be: the concrete type cannot be known ? > > Original message > From: Greg Hogan <c...@greghogan.com

Re: Kafka and Flink integration

The recommendation has been to avoid Kryo where possible. General data exchange: avro or thrift. Flink internal data exchange: POJO (or Tuple, which are slightly faster though less readable, and there is an outstanding PR to narrow or close the performance gap). Kryo is useful for types which

Re: coGroup exception or something else in Gelly job

gt; Flink is try to optimize the execution graph, but that’s the issue. > Maybe Flink provides an influence by the programmer? > > > Best and thanks > Marc > > > Am 10.06.2017 um 00:49 schrieb Greg Hogan <c...@greghogan.com>: > > Have you looked at org.apache

Re: coGroup exception or something else in Gelly job

Have you looked at org.apache.flink.gelly.GraphExtension.CustomVertexValue.createInitSemiCluster(CustomVertexValue.java:51)? > On Jun 9, 2017, at 4:53 PM, Kaepke, Marc wrote: > > Hi everyone, > > I don’t have any exceptions if I execute my Gelly job in my IDE

Re: Flink and swapping question

Hi Flavio, Flink handles interrupts so the only silent killer I am aware of is Linux's OOM killer. Are you seeing such a message in dmesg? Greg On Wed, May 24, 2017 at 3:18 AM, Flavio Pompermaier wrote: > Hi to all, > I'd like to know whether memory swapping could cause

Re: Production Deployments

We recently added documentation (and official builds) for Docker and K8s. Feedback is appreciated! https://ci.apache.org/projects/flink/flink-docs-release-1.2/setup/kubernetes.html I’m not aware of a

Re: [E] Re: unable to add more servers in zookeeper quorum peers in flink 1.2

s://urldefense.proofpoint.com/v2/url?u=http-3A__twitter.com_verizon=DwMD-g=udBTRvFvXC5Dhqg7UHpJlPps3mZ3LRxpb6__0PomBTQ=vykaCRzOoltktnhRTDFL-6nA55ddeOEzGwp9gbKhlfRrUojHU0nRm-mt29N20zWv=8K9Z7ArBnvYZL6fAHHu5u1r_-ezNyqdayG6uUQ8Wpqc=tqZxl_Osm7Nl_TMH5fn6uLqMmYwzWMRmIY0a6Cv94XE=> > > <https://urldefense.proofpoin

Re: [POLL] Who still uses Java 7 with Flink ?

Robert, Thanks for the report. Shouldn’t we be revisiting this decision at the beginning of the new release cycle rather than near the end? There is currently little cost to staying with Java 7 since no Flink code or pull requests have been written for Java 8. Greg > On Mar 23, 2017, at

Re: Variable Tuple Type

Hi Max, Belated response but this looks to be the same problem I am working to solve in Gelly with graph data in FLINK-3695 [0]. These arrays allow for object reuse. Interface is here [1]. Additional Value types are easy to add but Long, Int, and String are most common to Gelly. Suggestions are

Re: Apache Flink 1.1.4 - Gelly - LocalClusteringCoefficient - Returning values above 1?

113,1.0) >> (5113,9748,1.0) >> (10191,5113,1.0) >> (5113,10191,1.0) >> (6064,5113,1.0) >> (5113,6064,1.0) >> (6065,5113,1.0) >> (5113,6065,1.0) >> (6279,5113,1.0) >> (5113,6279,1.0) >> (4907,5113,1.0) >> (5113,4907,1.0) >> (6465,

Re: parallelism and slots allocated

Hi Bernard and Kurt, Chaining affects how subtasks operate within slots. Resource groups segregate subtasks into different slots. https://ci.apache.org/projects/flink/flink-docs-release-1.2/concepts/runtime.html#task-slots-and-resources

Re: Questions about the V-C Iteration in Gelly

Hi Xingcan, FLINK-1885 looked into adding a bulk mode to Gelly's iterative models. As an alternative you could implement your algorithm with Flink operators and a bulk iteration. Most of the Gelly library is written with native operators. Greg On Fri, Feb 10, 2017 at 5:02 AM, Xingcan Cui

Re: Remove old <1.0 documentation from google

See FLINK-5575. https://issues.apache.org/jira/browse/FLINK-5575 Looks like release-0.8 and older are not automatically rebuilt. https://ci.apache.org/builders/ On Thu, Feb 9, 2017 at 7:17 AM, Jonas wrote: > Maybe add "This documentation is outdated. Please switch to a

Re: start-cluster.sh issue

Hi Lior, Try adding this to your flink-conf.yaml: env.ssh.opts: FLINK_CONF_DIR=/tmp/parallelmachine/lior/flink/conf I think this is expected and not a bug (setting FLINK_CONF_DIR from the environment is supported for YARN). Please do file a JIRA for this feature as I think it would be a nice

Re: Flink configuration

Has anyone reported decreased performance with hyper-threading? On Tue, Jan 24, 2017 at 11:18 AM, Aljoscha Krettek wrote: > Hi, > that wording is from a time where no-one though about VMs with virtual > cores. IMHO this maps directly to virtual cores so you should set it >

Re: Improving Flink performance

Hi Jonas, It looks like the mailing list has removed your formatting and/or attachments. Greg On Mon, Jan 23, 2017 at 6:08 AM, Jonas wrote: > Hello! > > I'm having performance problems with a Flink job. If there is anything > valuable missing, please ask and I will try to

Re: Release 1.2?

Support for Hadoop 1 was dropped in FLINK-4895 [1]. [1] https://issues.apache.org/jira/browse/FLINK-4895 On Mon, Jan 23, 2017 at 11:09 AM, wrote: > I notice that for Flink 1.2.0 there is no x.y.z-hadoop1 folder (Cf. Apache > staging > repo >

Re: Apache Flink 1.1.4 - Gelly - LocalClusteringCoefficient - Returning values above 1?

Hi Miguel, The '--output print' option describes the values and also displays the local clustering coefficient value. You're running the undirected algorithm on a directed graph. In 1.2 there is an option '--simplify true' that will add reverse edges and remove duplicate edges and self-loops.

Re: Calculating stateful counts per key

Hi Mäki, This is the expected output. Your RichFlatMapFunction is opened once per task and you are sharing counterValue for all keys processed by that task. Greg On Mon, Dec 19, 2016 at 11:38 AM, Mäki Hanna wrote: > Hi, > > > > I'm trying to calculate stateful counts

Re: [DISCUSS] "Who's hiring on Flink" monthly thread in the mailing lists?

Google indexes the mailing list. Anyone can filter the messages to trash in a few clicks. This will also be a means for the community to better understand which and how companies are using Flink. On Fri, Dec 9, 2016 at 8:27 AM, Felix Neutatz wrote: > Hi, > > I wonder

Re: How to analyze space usage of Flink algorithms

This does sound like a nice feature, both per-job and per-taskmanager bytes written to and read from disk. On Fri, Dec 9, 2016 at 8:51 AM, Chesnay Schepler wrote: > We do not measure how much data we are spilling to disk. > > > On 09.12.2016 14:43, Fabian Hueske wrote: > >

Re: S3 checkpointing in AWS in Frankfurt

a breakdown unless that's something I can configure somewhere. > > Regards, > Jonathan > > > On 23 November 2016 at 16:29, Greg Hogan <c...@greghogan.com> wrote: > >> Hi Jonathan, >> >> Which S3 storage class are you using? Do you have a breakdown of

Re: spark vs flink batch performance

"For csv reading, i deliberately did not use csv reader since i want to run same code across spark and flink." If your objective deviates from writing and running the fastest Spark and fastest Flink programs, then your comparison is worthless. On Fri, Nov 18, 2016 at 5:37 AM, CPC

Re: Retrieving a single element from a DataSet

The tickets are in Flink's Jira: https://issues.apache.org/jira/browse/FLINK-4965 https://issues.apache.org/jira/browse/FLINK-4966 Are you looking to process temporal graphs with the DataStream API? On Fri, Nov 4, 2016 at 5:52 AM, otherwise777 wrote: > Cool, thnx

Re: NotSerializableException: jdk.nashorn.api.scripting.NashornScriptEngine

Hi Pedro, Which problem are you having, the NotSerializableException or not seeing open() called on a RichFunction? Greg On Wed, Nov 2, 2016 at 10:47 AM, PedroMrChaves wrote: > Hello, > > I'm having the exact same problem. > I'm using a filter function on a

Re: Looping over a DataSet and accesing another DataSet

By 'loop' do you refer to an iteration? The output of a bulk iteration is processed as the input of the following iteration. Values updated in an iteration are available in the next iteration just as values updated by an operator are available to the following operator. Your chosen algorithm may

Re: Looping over a DataSet and accesing another DataSet

The DataSet API only supports binary joins but one can simulate an n-ary join by chaining successive join operations. Your algorithm requires a global ordering on edges, requiring a parallelism of 1, and will not scale in a distributed processing system. Flink excels at processing bulk (larger

Re: Retrieving a single element from a DataSet

I created FLINK-4965 "AllPairsShortestPaths" and FLINK-4966 "BetweennessCentrality". On Wed, Oct 26, 2016 at 4:39 PM, Greg Hogan <c...@greghogan.com> wrote: > It sounds like you want to use an all-pairs shortest paths algorithm. This > would be a great

Re: TIMESTAMP TypeInformation

Could be. I had thought TypeInfoParser was closely related to TypeExtractor. On Thu, Oct 27, 2016 at 10:20 AM, Fabian Hueske <fhue...@gmail.com> wrote: > Wouldn't that be orthogonal to adding it to the TypeInfoParser? > > 2016-10-27 15:22 GMT+02:00 Greg Hogan <c...@greghogan

Re: TIMESTAMP TypeInformation

Fabian, Should we instead add this as a registered TypeInfoFactory? Greg On Thu, Oct 27, 2016 at 3:55 AM, Fabian Hueske wrote: > Yes, I think you are right. > TypeInfoParser needs to be extended to parse the java.sql.* types into the > corresponding TypeInfos. > > Can you

Re: Retrieving a single element from a DataSet

It sounds like you want to use an all-pairs shortest paths algorithm. This would be a great contribution to Gelly! https://en.wikipedia.org/wiki/Shortest_path_problem#All-pairs_shortest_paths On Wed, Oct 26, 2016 at 9:29 AM, otherwise777 wrote: > That is indeed not the

Re: Parallelism vs task manager allocation

Is the query stream also a Flink job? Is this use case not supported by keeping state within a single Flink job? https://ci.apache.org/projects/flink/flink-docs-master/dev/state.html FLINK-3779 recently added "queryable state" to allow external processes access to operator state.

Re: SQL for Flink

Hi Deepak, There are many open tickets for Flink's SQL API. Documentation is at https://ci.apache.org/projects/flink/flink-docs-master/dev/table_api.html.

Re: Flink Iterations vs. While loop

Hi Dan, Where are you reading the 200 GB "data" from? How much memory per node? If the DataSet is read from a distributed filesystem and if with iterations Flink must spill to disk then I wouldn't expect much difference. About how many iterations are run in the 30 minutes? I don't know that this

Re: Metrics not reported to graphite

Have you copied the required jar files into your lib/ directory? Only JMX support is provided in the distribution. On Thu, Sep 1, 2016 at 5:07 PM, Jack Huang wrote: > Hi all, > > I followed the instruction for reporting metrics to a Graphite server on > the official document

Re: Setting number of TaskManagers

The number of TaskManagers will be equal to the number of entries in the conf/slaves file. On Wed, Aug 24, 2016 at 3:04 PM, Foster, Craig wrote: > Is there a way to set the number of TaskManagers using a configuration > file or environment variable? I'm looking at the docs

Re: Performance issues with GroupBy?

Hi Robert, Are you able to simplify the your function input / output types? Flink aggressively serializes the data stream and complex types such as ArrayList and BitSet will be much slower to process. Are you able to reconstruct the lists to be groupings on elements? Greg On Mon, Jul 25, 2016

Re: Modifying start-cluster scripts to efficiently spawn multiple TMs

nayake <esal...@gmail.com> > wrote: > >> I meant, I'll check when current jobs are done and will let you know. >> >> On Mon, Jul 11, 2016 at 12:19 PM, Saliya Ekanayake <esal...@gmail.com> >> wrote: >> >>> I am running some jobs now. I'll s

Re: Parameters to Control Intra-node Parallelism

:55 PM, Saliya Ekanayake <esal...@gmail.com> wrote: > Greg, > > where did you see the OOM log as shown in this mail thread? In my case > none of the TaskManagers nor JobManger reports an error like this. > > On Sun, Jul 10, 2016 at 8:45 PM, Greg Hogan <c...@greghogan.com&

Re: Parameters to Control Intra-node Parallelism

These symptoms sounds similar to what I was experiencing in the following thread. Flink can have some unexpected memory usage which can result in an OOM kill by the kernel, and this becomes more pronounced as the cluster size grows. https://www.mail-archive.com/dev@flink.apache.org/msg06346.html

Re: Modifying start-cluster scripts to efficiently spawn multiple TMs

Hi Saliya, Would you happen to have pdsh (parallel distributed shell) installed? If so the TaskManager startup in start-cluster.sh will run in parallel. As to running 24 TaskManagers together, are these running across multiple NUMA nodes? I had filed FLINK-3163 (

Re: sampling function

Hi Do, DataSet provides a stable @Public interface. DataSetUtils is marked @PublicEvolving which is intended for public use, has stable behavior, but method signatures may change. It's also good to limit DataSet to common methods whereas the utility methods tend to be used for specific

Re: Does Flink allows for encapsulation of transformations?

"The question is how to encapsulate numerous transformations into one object or may be a function in Apache Flink Java setting." Implement CustomUnaryOperation. This can then be applied to a DataSet by calling `DataSet result = DataSet.runOperation(new MyOperation<>(...));`. On Mon, Jun 6, 2016

Re: BloomFilter Exception

Flavio, It looks like your build is older than some recent fixes in that code. https://github.com/apache/flink/commits/2e6e4de5d1d2b5123f4311493763fd84f52779ab/flink-runtime/src/main/java/org/apache/flink/runtime/operators/hash/MutableHashTable.java Greg On Fri, Sep 4, 2015 at 10:34 AM, Flavio

max-fan

the number of file handles per operator, but may cause intermediate merging/partitioning, if set too small (DEFAULT: 128). Greg Hogan