Re: Flink Kafka TimeoutException

Have you tried increasing the request.timeout.ms parameter (Kafka) ? Which Flink / Kafka release are you using ? Cheers On Thu, Jul 5, 2018 at 5:39 AM Amol S - iProgrammer wrote: > Hello, > > I am using flink with kafka and getting below exception. > >

Re: error: object connectors is not a member of package org.apache.flink.streaming

any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > &g

Re: [Flink-9407] Question about proposed ORC Sink !

For #1, the word exist should be exit, right ?Thanks Original message From: zhangminglei <18717838...@163.com> Date: 6/23/18 10:12 AM (GMT+08:00) To: sagar loke Cc: dev , user Subject: Re: [Flink-9407] Question about proposed ORC Sink ! Hi, Sagar. 1. It solves the issue

Re: Exception while submitting jobs through Yarn

Since you're using a vendor's distro, I would suggest asking on their user forum. Cheers Original message From: Garvit Sharma Date: 6/19/18 3:34 AM (GMT-08:00) To: trohrm...@apache.org Cc: Amit Jain , Chesnay Schepler , Ted Yu , user@flink.apache.org Subject: Re: Exception

Re: Exception while submitting jobs through Yarn

The error for core-default.xml is interesting.  Flink doesn't have this file. Probably it came with Yarn. Please check the hadoop version Flink was built with versus the hadoop version in your cluster. Thanks Original message From: Garvit Sharma Date: 6/16/18 7:23 AM

Re: Replaying logs with microsecond delay

Please see the following: http://www.rationaljava.com/2015/10/measuring-microsecond-in-java.html https://stackoverflow.com/questions/11498585/how-to-suspend-a-java-thread-for-a-small-period-of-time-like-100-nanoseconds On Tue, May 15, 2018 at 10:40 AM, Dhruv Kumar wrote:

Re: A strange exception while consumption using a multi topic Kafka Connector

FLINK-9349 was logged. FYI On Sat, May 12, 2018 at 7:52 AM, Ted Yu <yuzhih...@gmail.com> wrote: > I took a look at ./flink-connectors/flink-connector-kafka-0.9/src/main/ > java/org/apache/flink/streaming/connectors/kafka/ > internal/Kafka09Fetcher.java > &

Re: A strange exception while consumption using a multi topic Kafka Connector

I took a look at ./flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/Kafka09Fetcher.java It seems the List subscribedPartitionStates was being modified when runFetchLoop iterated the List. This can happen if, e.g., FlinkKafkaConsumer

Re: Dynamically deleting kafka topics does not remove partitions from kafkaConsumer

It seems the correct JIRA should be FLINK-9303 On Sun, May 6, 2018 at 8:29 PM, Tzu-Li (Gordon) Tai wrote: > Hi Edward, > > Thanks for brining this up, and I think your suggestion makes sense. > The problem is that the Kafka

Re: Use gradle with flink

Currently only maven build is supported. Original message From: Georgi Stoyanov Date: 4/24/18 2:17 AM (GMT-08:00) To: user@flink.apache.org Subject: Use gradle with flink Hi guys, I’m wondering is it possible to setup my java flink application with

Re: Kafka 0.11

For #4, there was past thread: http://search-hadoop.com/m/Flink/VkLeQPK4Ek2dYiZS1?subj=Re+Flink+on+Azure+HDInsight You can find related information on Azure Table in: docs/dev/batch/connectors.md FYI On Mon, Apr 23, 2018 at 4:13 AM, m@xi wrote: > Hey Michael! Thanks a

Re: Checkpointing barriers

barrier n appearing in all the streams serves as synchronization point. As explained in the subsequent paragraph: bq. Otherwise, it would mix records that belong to snapshot *n*and with records that belong to snapshot *n+1*. Cheers On Mon, Apr 23, 2018 at 7:21 AM, Alexander Smirnov <

Re: WordCount with Flink Docker

. Please check job manager / task manager log to see if you can find the complete stack trace. On Sun, Apr 22, 2018 at 6:55 AM, Ted Yu <yuzhih...@gmail.com> wrote: > Looking at flink-examples/flink-examples-batch/src/main/java/ > org/apache/flink/examples/java/wordcount/util/WordCo

Re: WordCount with Flink Docker

length limit and was truncated. > > The corresponding line 70 in Wordcount.java is > > System.out.println("Executing WordCount example with default input data > set."); > > I got the docker image from https://github.com/docker-flink/docker-flink > and still get this message even wit

Re: WordCount with Flink Docker

bq. To enable Queryable State, please move the flink-queryable-state-runtime jar from the opt to the lib folder. Have you noticed the above and taken corresponding action ? Thanks On Sat, Apr 21, 2018 at 3:15 PM, JP de Vooght wrote: > Hello, > > When I submit a WordCount job

Re: managin order to use epoll (tasker.network.netty.transport: epoll), is it required that linux version is 4.0.16 or newer or not

I think you should upgrade Linux to said version or newer. Cheers On Fri, Apr 20, 2018 at 6:35 AM, makeyang wrote: > my flink veriso is 1.4.2 > my jdk version is 1.8.0.20 > my linux version is:3.10.0 > > I try to use epoll with setting:

Re: Flink / Kafka unit testing with scalatest-embedded-kafka

> Hi, > > > > withRunningKafka launch a kafka broker. This is one of the advantage of > this library. > > > > I test to consume / produce messages with kafka command line, and it seems > alright. > > > > Thanks > > > > *De :* Ted Yu [mailto:yuzhih...@gm

Re: java.lang.Exception: TaskManager was lost/killed

Please see the last comment on this issue: https://github.com/facebook/rocksdb/issues/3216 FYI On Tue, Apr 10, 2018 at 12:25 AM, Lasse Nedergaard < lassenederga...@gmail.com> wrote: > > This graph shows Non-Heap . If the same pattern exists it make sense that > it will try to allocate more

Re: RocksDBMapState example?

For KeyedState, apart from https://ci.apache.org/projects/flink/flink-docs- release-1.4/dev/stream/state/state.html#keyed-state-and-operator-state , you can refer to docs/dev/migration.md : public void initializeState(FunctionInitializationContext context) throws Exception { counter =

Re: java.lang.Exception: TaskManager was lost/killed

Can you use third party site for the graph ? I cannot view it. Thanks Original message From: Lasse Nedergaard Date: 4/10/18 12:25 AM (GMT-08:00) To: Ken Krugler Cc: user , Chesnay Schepler

Re: RocksDBMapState example?

Hi, Have you looked at the ctor call in : flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java around line 1261 ? Cheers

Re: [DISCUSS] Dropping Scala 2.10

+1 Original message From: Hai Zhou Date: 9/20/17 12:44 AM (GMT-08:00) To: Aljoscha Krettek , d...@flink.apache.org, user Subject: Re: [DISCUSS] Dropping Scala 2.10 +1 > 在 2017年9月19日，17:56，Aljoscha Krettek

Re: Using Azure Blob Storage with Flink

fs.AbstractFileSystem.wasb.Impl > org.apache.hadoop.fs.azure.Wasb > > > However, I’m wondering if it’s possible to achieve the same result from > within the job since it’s difficult to modify files on the task manager in > our configuration. > > On Aug 29, 2017, at 5:

Re: Using Azure Blob Storage with Flink

Was hadoop-azure jar on the classpath ? Please also see the following from https://hadoop.apache.org/docs/current/hadoop-azure/index.html : The built jar file, named hadoop-azure.jar, also declares transitive dependencies on the additional artifacts it requires, notably the Azure Storage SDK for

Re: Null Pointer Exception on Trying to read a message from Kafka

wrote: > Kafka Version is 0.10.0 > > On Tue, Aug 29, 2017 at 6:43 AM, Sridhar Chellappa <flinken...@gmail.com> > wrote: > >> 1.3.0 >> >> On Mon, Aug 28, 2017 at 10:09 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Which Flink version are y

Re: Example build error

Looking at: https://github.com/kl0u/flink-examples/blob/master/src/main/java/com/dataartisans/flinksolo/simple/StreamingWordCount.java there is no line 56. Which repo do you get StreamingWordCount from ? On Mon, Aug 28, 2017 at 3:58 PM, Jakes John wrote: > When I am

Re: Null Pointer Exception on Trying to read a message from Kafka

} finally { > return MyKafkaMessage; > } > } > > @Override > public boolean isEndOfStream(MyKafkaMessage nextElement) { > return false; > } > > @Override > public TypeInformation getProducedType() { > return null; >

Re: Null Pointer Exception on Trying to read a message from Kafka

Which version of Flink / Kafka are you using ? Can you show the snippet of code where you create the DataStream ? Cheers On Mon, Aug 28, 2017 at 7:38 AM, Sridhar Chellappa wrote: > Folks, > > I have a KafkaConsumer that I am trying to read messages from. When I try > to

Re: [Error]TaskManager -RECEIVED SIGNAL 1: SIGHUP. Shutting down as requested

See http://docs.oracle.com/cd/E19253-01/816-5166/6mbb1kq04/index.html Cheers On Sun, Aug 27, 2017 at 11:47 PM, Samim Ahmed <samim1...@gmail.com> wrote: > Hello Ted Yu, > > Thanks for your response and a sincere apology for let reply. > > OS version : Solaris10. > Flink V

Re: Flink-HTM integration

Can you give us some detail on how flink-HTM is integrated in your project ? Cheers On Sun, Aug 27, 2017 at 6:28 AM, AndreaKinn wrote: > I think this is not a good idea, I don't know if it's a bug or a my fault. > I believe I integrated correctly flink-HTM in my project

Re: Flink Elastic Sink AWS ES

If port 9300 in the following example is replaced by 9200, would that work ? https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/connectors/elasticsearch.html Please use Flink 1.3.1+ On Sat, Aug 26, 2017 at 3:00 PM, ant burton wrote: > Hello, > > Has anybody

Re: Security Control of running Flink Jobs on Flink UI

bq. introduce a special config flag to disable the Cancel functionality +1 Similar config is used in other project(s) such as hbase. On Fri, Aug 25, 2017 at 2:54 PM, Robert Metzger wrote: > Hi Raja, > > you can actually disable the UI by setting the port to a negative

Re: Possible conflict between in Flink connectors

ndency? Doesn't the > "flink-hbase" dependency transitively pull in hbase? > > On Fri, Aug 25, 2017 at 6:35 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> If Guava 18.0 is used to build hbase 1.3, there would be compilation >> errors such as the following: >> >&g

Re: Possible conflict between in Flink connectors

If Guava 18.0 is used to build hbase 1.3, there would be compilation errors such as the following: [ERROR] /mnt/disk2/a/1.3-h/hbase-server/src/main/java/org/ apache/hadoop/hbase/replication/regionserver/ReplicationSource.java:[271,25] error: cannot find symbol [ERROR] symbol: method

Re: [Error]TaskManager -RECEIVED SIGNAL 1: SIGHUP. Shutting down as requested

Can you provide more information ? OS version Flink version Anything interesting in dmesg output around this time ? On Thu, Aug 24, 2017 at 4:53 AM, Samim Ahmed wrote: > Hi All, > > From last two days I am getting below error and the worker server are > killed with below

Re: Support for multiple HDFS

Would HDFS-6584 help with your use case ? On Wed, Aug 23, 2017 at 11:00 AM, Vijay Srinivasaraghavan < vijikar...@yahoo.com.invalid> wrote: > Hello, > Is it possible for a Flink cluster to use multiple HDFS repository (HDFS-1 > for managing Flink state backend, HDFS-2 for syncing results from

Re: Issue with Scala API when using CEP's "notFollowedBy"

Please take a look at FLINK-7306, fixed for 1.4.0 release. On Mon, Aug 21, 2017 at 1:03 PM, Gehad Elrobey wrote: > Hi there, > > I have an issue using the Scala API for the CEP library, the notFollowedBy API > call doesn't return a Pattern it returns a Unit instead, So I

Re: Change state backend.

bq. we add the key-group to the heap format (1-2 bytes extra per key). This seems to be better choice among the two. bq. change the heap backends to write in the same way as RocksDB +1 on above. These two combined would give users flexibility in state backend migration. On Thu, Aug 17, 2017

Re: hadoop

Can you check the following config in yarn-site.xml ? yarn.resourcemanager.proxy-user-privileges.enabled (true) Cheers On Wed, Aug 16, 2017 at 4:48 PM, Raja.Aravapalli wrote: > > > Hi, > > > > I triggered an flink yarn-session on a running Hadoop cluster… and >

Re:Re:Re:Re:How to verify the data to Elasticsearch whether correct or not ?

Did you use image for the code ?Can you send plain code again ?Cheers Original message From: mingleizhang <18717838...@163.com> Date: 8/16/17 6:16 PM (GMT-08:00) To: mingleizhang <18717838...@163.com> Cc: "Tzu-Li (Gordon) Tai" , user@flink.apache.org

Re: Reload DistributedCache file?

For hdfs, there is iNotify mechanism. https://issues.apache.org/jira/browse/HDFS-6634 https://www.slideshare.net/Hadoop_Summit/keep-me-in-the-loop-inotify-in-hdfs FYI On Wed, Aug 16, 2017 at 9:41 AM, Conrad Crampton < conrad.cramp...@secdata.com> wrote: > Hi, > > I have a simple text file

Re: Change state backend.

I guess shashank meant switching state backend w.r.t. savepoints. On Wed, Aug 16, 2017 at 4:00 AM, Biplob Biswas wrote: > Could you clarify a bit more? Do you want an existing state on a running > job > to be migrated from FsStateBackend to RocksDbStateBackend? > > Or

Re: TypeInformation in Custom Deserializer

Please take a look at the following in flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopInputFormat.java public TypeInformation> getProducedType() { return new TupleTypeInfo<>(TypeExtractor.createTypeInfo(keyClass),

Re: Apache beam and Flink

I found this: http://search-hadoop.com/m/Beam/gfKHFfzU1k2VKWlv?subj=Re+CEP+Pattern+matching+on+top+of+Beam+pipeline Consider asking on Beam mailing list. On Sun, Aug 13, 2017 at 6:53 AM, Basanth Gowda wrote: > I wasn't able to find much info on Flink and Beam wrt CEP &

Re: No file system found with scheme s3

Shouldn't the config key be : org.apache.hadoop.fs.s3.S3FileSystem Cheers On Fri, Aug 11, 2017 at 5:38 PM, ant burton wrote: > Hello, > > After following the instructions to set the S3 filesystem in the > documentation (https://ci.apache.org/projects/flink/flink-docs- >

Re: Null pointer exception while trying to serialize a protobuf message

return this; > } > } > > public CustomerMessage.Builder clearDeviceIdLoggedInFrom() { > this.deviceIdLoggedInFrom_ = CustomerMessage.DEFAULT_INSTANCE. > getDeviceIdLoggedInFrom(); > this.onChanged(); > return this; >

Re: Null pointer exception while trying to serialize a protobuf message

Can you show how CustomerMessage is defined ? Thanks On Fri, Aug 4, 2017 at 7:22 AM, Sridhar Chellappa wrote: > Folks, > > I wrote a custom Data source to test me CEP logic. The custom data source > looks like : > > public class CustomerDataSource extends

Re: Customer inputformat

>> FileInputFormat also supports reading all files in a directory. Simply >> specify the path of the directory. >> >> Check StreamExecutionEnvironment.createFileInput() which takes a several >> parameters such as a FileInputFormat and a time interval in which the >> dir

Re: Customer inputformat

For #1, you can find quite a few classes which extend FileInputFormat. e.g. flink-connectors/flink-avro/src/main/java/org/apache/flink/api/java/io/AvroInputFormat.java:public class AvroInputFormat extends FileInputFormat implements ResultTypeQuer

Re: Graph Analytics on HBase With HGraphDB and Apache Flink Gelly

Interesting blog. >From your experience, is there anything on hbase side which you see room for improvement ? Which hbase release are you using ? Cheers On Thu, Jul 27, 2017 at 3:11 PM, Robert Yokota wrote: > In case anyone is interested, I wrote a blog on how to analyze

Re: Logback user class

Please take a look at FLINK-6767. On Wed, Jul 26, 2017 at 3:53 AM, nragon wrote: > Hi, > > I executing the following snippet on two different environments. > > > StreamExecutionEnvironment streamEnv = >

Re: How can I set charset for flink sql？

Logged CALCITE-1903 for this bug. FYI On Tue, Jul 25, 2017 at 6:39 PM, 程骥 wrote: > OK，thanks for remind me. > > My sql like this(contain a Chinese word): > > SELECT > 'HIGH' AS LEVEL, > 'Firewall uplink bandwidth exception:greater than 1' AS content, >

Re: Flink rolling upgrade support

This was the other thread, right ? http://search-hadoop.com/m/Flink/VkLeQ0dXIf1SkHpY?subj=Re+Does+job+restart+resume+from+last+known+internal+checkpoint+ On Wed, Jul 19, 2017 at 9:02 AM, Moiz Jinia wrote: > Yup! Thanks. > > Moiz > > — > sent from phone > > On 19-Jul-2017,

Re: How to send local files to a flink job on YARN

I went back to commit 6e38eb8: [FLINK-1436] [docs] update command line documentation A search in the repo for "yarnship" ended up with no hit in the code (same with commit bf6b9aaab89e2e04678784525a42a19f099aa7f5 which is at top of git repo). Wondering whether it is supported. On Thu, Jul 13,

Re: [POLL] Who still uses Java 7 with Flink ?

+1 on dropping support for Java 1.7 Original message From: Robert Metzger Date: 7/12/17 8:36 AM (GMT-08:00) To: d...@flink.apache.org Cc: user Subject: Re: [POLL] Who still uses Java 7 with Flink ? +1 to drop Java 7 support I

Re: Error when set RocksDBStateBackend option in Flink?

Since MRocksDBFactory doesn't add any option, it seems rocksDBBackEnd.setOptions() call can be skipped. If you choose to keep the call, please take a look at (OptionsFactory extends java.io.Serializable): https://docs.oracle.com/javase/7/docs/api/java/io/Serializable.html On Thu, Jun 29, 2017

Re: Appending Windowed Aggregates to Events

In case no-sql store is considered, please take a look at base Cheers Original message From: "Jain, Ankit" Date: 6/26/17 12:41 PM (GMT-08:00) To: Tim Stearn , user@flink.apache.org Subject: Re: Appending Windowed Aggregates to Events

Re: Kafka and Flink integration

Greg:Can you clarify he last part?Should it be: the concrete type cannot be known ? Original message From: Greg Hogan Date: 6/21/17 3:10 AM (GMT-08:00) To: nragon Cc: user@flink.apache.org Subject: Re: Kafka and Flink

Re: Possible Data Corruption?

See this thread: http://search-hadoop.com/m/Flink/VkLeQm2nZm1Wa7Ny1?subj=Re+Painful+KryoException+java+lang+IndexOutOfBoundsException+on+Flink+Batch+Api+scala which mentioned FLINK-6398 fixed in 1.2.2 / 1.3 On Mon, Jun 19, 2017 at 5:53 PM,

Re: Cannot write record to fresh sort buffer. Record too large.

For #2, XmlInputFormat was involved. Is it possible to prune (unneeded) field(s) so that heap requirement is lower ? On Wed, Jun 14, 2017 at 8:47 AM, Sebastian Neef < gehax...@mailbox.tu-berlin.de> wrote: > Hi Ted, > > sure. > > Here's the stack strace with .distinct() with the Exception in the

Re: Cannot write record to fresh sort buffer. Record too large.

For the 'Requested array size exceeds VM limit' error, can you pastebin the full stack trace ? Thanks On Wed, Jun 14, 2017 at 3:22 AM, Sebastian Neef < gehax...@mailbox.tu-berlin.de> wrote: > Hi, > > I removed the .distinct() and ran another test. > > Without filtering duplicate entries, the

Re: Cannot write record to fresh sort buffer. Record too large.

Sebastian: Are you using jdk 7 or jdk 8 ? For jdk 7, there was bug w.r.t. code cache getting full which affects performance. https://bugs.openjdk.java.net/browse/JDK-8051955 https://bugs.openjdk.java.net/browse/JDK-8074288 http://blog.andresteingress.com/2016/10/19/java-codecache Cheers On

Re: Java 8 lambdas for CEP patterns won't compile

ill be fixed. > > This is the JIRA to track it: > https://issues.apache.org/jira/browse/FLINK-6897 > > Kostas > > On Jun 12, 2017, at 11:51 AM, Ted Yu <yuzhih...@gmail.com> wrote: > > Do know which JIRA / discussion thread had the context for this decision ? > > I d

Re: Java 8 lambdas for CEP patterns won't compile

. I will update it today. > Java8 Lambdas are NOT supported by CEP in Flink 1.3. > > Hopefully this will change soon. I will open a JIRA for this. > > Cheers, > Kostas > > On Jun 11, 2017, at 11:55 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > >

Re: Java 8 lambdas for CEP patterns won't compile

Looking at docs/dev/libs/cep.md , there're 3 examples using lambda. Here is one: Pattern pattern = Pattern.begin("start").where(evt -> evt.getId() == 42) Your syntax should be supported. I haven't found such example in test code, though. FYI On Sun, Jun 11, 2017 at 2:42 PM, David

Re: SingleOutputStreamOperator addsink Error

bq. new SinkFunction(){ Note the case in JsonObject. It should be JSONObject FYI On Thu, Jun 8, 2017 at 1:27 PM, G.S.Vijay Raajaa wrote: > Hi, > > I am trying to pass the SingleOutputStreamOperator to a custom sink. I am > getting an error while implementing the same.

Re: Flink 1.3 - Checkpointing failing

( The UnsupportedOperationException thrown by snapshotConfiguration() should be caught without proceeding to nestedSerializersAndConfigs.add(). On Fri, Jun 2, 2017 at 7:02 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Your case doesn't seem like FLINK-5462 since there was no > CancellationException > in the stack t

Re: Flink 1.3 - Checkpointing failing

Your case doesn't seem like FLINK-5462 since there was no CancellationException in the stack trace you posted. The exception from TraversableSerializer.snapshotConfiguration() was added by FLINK-6178 FYI On Fri, Jun 2, 2017 at 4:04 PM, MAHESH KUMAR wrote: > Hi

Re: [ANNOUNCE] Apache Flink 1.3.0 released

Robert: Do you know when maven artifacts would be populated ? Currently I don't see 1.3.0 here: https://mvnrepository.com/artifact/org.apache.flink/flink-core Thanks On Thu, Jun 1, 2017 at 7:48 AM, Robert Metzger wrote: > The Apache Flink community is pleased to announce

Re: trying to externalize checkpoint to s3

641) > > at org.apache.flink.streaming.runtime.tasks.StreamTask. > performCheckpoint(StreamTask.java:586) > > at org.apache.flink.streaming.runtime.tasks.StreamTask. > triggerCheckpoint(StreamTask.java:529) > > at org.apache.flink.runtime.tas

Re: Production Deployments

This seems related FLINK-5966 On Fri, May 12, 2017 at 4:29 PM, Samuel Doyle wrote: > Is there some documentation describing best practices regarding production > deployments specifically recommended configuration and setup? From what I > have seen, it appears that YARN is

Re: UnknownHostException during start

Dominique: Which hadoop release are you using ? Please pastebin the classpath. Cheers On Thu, May 11, 2017 at 7:27 AM, Till Rohrmann wrote: > Hi Dominique, > > I’m not exactly sure but this looks more like a Hadoop or a Hadoop > configuration problem to me. Could it be

Re: Periodic flush sink?

MySink.this.mutator.flush(); >> } catch (Exception e) { >> // Ignore >> } >> }}, HBASE_BUFFER_AUTO_FLUSH_INTERVAL, HBASE_BUFFER_AUTO_FLUSH_INTERVAL); >> } >> >> @Override >> public void close() throws IOException { >

Re: Weird serialization bug?

Have you tried making fooo static ? Cheers On Sat, Apr 29, 2017 at 4:26 AM, Sebastian Neef < gehax...@mailbox.tu-berlin.de> wrote: > Hello Apache Flink users, > > I implemented a FilterFunction some months ago that worked quite well > back then. However, I wanted to check it out right now and

Re: Periodic flush sink?

I expect Flink expert to answer your question. bq. I get a flush of the buffers atleast every few seconds >From hbase point of view, during low traffic period, the above may result in many small hfiles, leading to more work for the compaction. FYI On Sat, Apr 29, 2017 at 7:32 AM, Niels Basjes

Re: flink testing

Please give more context by describing what spark-test-base does :-) > On Apr 22, 2017, at 10:57 PM, Georg Heiler wrote: > > Hi, > > is there something like spark-testing-base for flink as well? > > Cheers, > Georg

Re: Why TimerService interface in ProcessFunction doesn't have deleteEventTimeTimer

> https://www.confluent.io/blog/apache-kafka-purgatory- > hierarchical-timing-wheels/ > > http://www.cs.columbia.edu/~nahum/w6998/papers/sosp87-timing-wheels.pdf > > Cheers, > Gyula > > On Sat, Apr 22, 2017, 00:35 Ted Yu <yuzhih...@gmail.com> wrote: > >>

Re: Why TimerService interface in ProcessFunction doesn't have deleteEventTimeTimer

of timers > that were deleted. But it is eventually cleaned up. > > Gyula > > Ted Yu <yuzhih...@gmail.com> ezt írta (időpont: 2017. ápr. 21., P, 21:38): > >> A bit curious: wouldn't using "tombstone" markers constitute some memory >> leak (since Timers are

Re: Why TimerService interface in ProcessFunction doesn't have deleteEventTimeTimer

A bit curious: wouldn't using "tombstone" markers constitute some memory leak (since Timers are not released) ? Cheers On Fri, Apr 21, 2017 at 12:23 PM, Gyula Fóra wrote: > Hi! > > I thought I would drop my opinion here maybe it is relevant. > > We have used the Flink

Re: Flink errors out and job fails--IOException from CollectSink.open()

40787610437508121973762 > 49737935721565982386290786,}}'}, starting state set to the restored > sequence number LATEST_SEQUENCE_NUM > > 2017-04-13 23:28:23,472 INFO org.apache.flink.runtime. > taskmanager.Task - Freeing task resources for Source: > Custom Source (1/1) (8a

Re: Flink errors out and job fails--IOException from CollectSink.open()

Can you give us a bit more information ? release of flink snippet of your code Thanks

Re: 20 times higher throughput with Window function vs fold function, intended?

The 1,2million seems to be European notation. You meant 1.2 million, right ? > On Mar 31, 2017, at 1:19 AM, Kamil Dziublinski > wrote: > > Hi, > > Thanks for the tip man. I tried playing with this. > Was changing fetch.message.max.bytes (I still have 0.8 kafka)

Re: 20 times higher throughput with Window function vs fold function, intended?

Kamil: In the upcoming hbase 2.0 release, there are more write path optimizations which would boost write performance further. FYI > On Mar 30, 2017, at 1:07 AM, Kamil Dziublinski > wrote: > > Hey guys, > > Sorry for confusion it turned out that I had a bug in

Re: flink-1.2 and unit testing / flinkspector

Nancy: You can start another thread for the failed unit tests. You can pass "-DskipTests" to get over the install command. Cheers On Thu, Mar 23, 2017 at 11:06 AM, Nancy Estrada wrote: > Hi Tarandeep and Ted, > > I am in this route now. I am trying to use

Re: Scala vs Java

Sure you can. Examples come with Java and Scala versions. e.g. https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/datastream_api.html On Tue, Mar 21, 2017 at 9:46 AM, Madabhattula Rajesh Kumar < mrajaf...@gmail.com> wrote: > Hi, > > How stable scala API's when compare to Java

Re: flink-1.2 and unit testing / flinkspector

You're welcome. Maybe send out pull request to flink-spector so that other people don't have to go thru this route. On Fri, Mar 17, 2017 at 7:16 PM, Tarandeep Singh <tarand...@gmail.com> wrote: > Thank you Ted. It worked! > > Best, > Tarandeep > > On Fri, Mar 17, 2017 at

Re: flink-1.2 and unit testing / flinkspector

Could not resolve dependencies for project > org.flinkspector:flinkspector-core_2.11:jar:0.6-SNAPSHOT: Could not find > artifact org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in > central (https://repo.maven.apache.org/maven2) -> [Help 1] > > Thanks, > Tarandeep > > > On Fri, M

Re: flink-1.2 and unit testing / flinkspector

Can you post the patch for flink-specter where the mini cluster is replaced ? I assume you upgraded the version of Flink in the pom. Cheers > On Mar 17, 2017, at 4:26 PM, Tarandeep Singh wrote: > > Hi, > > Is someone using flinkspector unit testing framework with

Re: Java 7 -> 8 - Association failed error

ing. I ended up changing > yarn.nodemanager.vmem-check-enabled to false. Is this a bug? is there a > side effect? > > On Fri, Feb 24, 2017 at 2:54 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Can you give more detail ? >> >> I found: >> https://issues.apa

Re: Java 7 -> 8 - Association failed error

Can you give more detail ? I found: https://issues.apache.org/jira/browse/YARN-4714 Cheers On Fri, Feb 24, 2017 at 2:41 PM, Mohit Anchlia wrote: > Figured out. It looks like there is a virtual memory limit check enforced > in yarn which just surfaced with java 8 > > On

Re: Compiling Flink on macOS.

m/master, origin/master, origin/HEAD, master) > [FLINK-5836] [flip6] Fix race condition between offer slot and submit task > (5 hours ago) > > I am on *d6aed38.* > > *(btw - on linux there is no issue)* > > > On Wed, Feb 22, 2017 at 10:50 AM, Ted Yu <yuzhih...@gmail.com>

Re: Compiling Flink on macOS.

int) > [ERROR] constructor org.apache.flink.storm.wrappers.SpoutWrapper. > SpoutWrapper(backtype.storm.topology.IRichSpout,java.lang.String[],java.lang.Integer) > is not applicable > [ERROR] (argument mismatch; > org.apache.flink.storm.wordcount.operators.WordCountInMemorySpout &g

Re: Compiling Flink on macOS.

Can you show the error(s) you got ? What is the Java version you're using ? Thanks On Wed, Feb 22, 2017 at 10:24 AM, Lior Amar wrote: > Hi all, > > My name is Lior and I an new to this community. > I am working on enhancing the python API of Flink (batch). > >

Re: Checkpointing with RocksDB as statebackend

>>>> >>>>>> Can you please point me where am I doing wrong. I have tried different >>>>>> configuration options but each time the Task Manager is getting killed >>>>>> after some time :) >>>>>> >>>>>> Regards

Re: Akka 2.4

Please see FLINK-3662 On Thu, Feb 16, 2017 at 9:01 AM, Dmitry Golubets wrote: > Hi, > > Can I force Flink to use Akka 2.4 (recompile if needed)? > Is it going to misbehave in a subtle way? > > > Best regards, > Dmitry >

Re: Flink jdbc

See the tutorial at the beginning of: flink-connectors/flink-jdbc/src/main/java/org/apache/flink/api/java/io/jdbc/JDBCInputFormat.java Looks like plugging in "org.h2.Driver" should do. On Wed, Feb 15, 2017 at 4:59 PM, Punit Tandel wrote: > Hi All > > Does flink jdbc

Re: Checkpointing with RocksDB as statebackend

What did the More Details link say ? Thanks > On Feb 15, 2017, at 3:11 AM, vinay patil wrote: > > Hi, > > I have kept the checkpointing interval to 6secs and minimum pause between > checkpoints to 5secs, while testing the pipeline I have observed that that > for some

Re: Hadoop 2.7.3

Dean: Can you pastebin the stack trace around the MethodMissing error ? If there was no stack trace, please tell us the what the log said. Thanks On Fri, Feb 10, 2017 at 2:26 PM, Dean Wampler wrote: > This is completely unrelated, but I just debugged a

Re: Javadoc

> There are two 404s on the page which is likely causing the issue: > > https://ci.apache.org/projects/flink/flink-docs- > master/api/java/stylesheet.css > https://ci.apache.org/projects/flink/flink-docs- > master/api/java/script.js > > Any idea who to report to? > > > On

Re: Javadoc

Have you looked under https://ci.apache.org/projects/flink/flink -docs-master/api/java ? e.g. suppose you look for API for ByteArrayOutputStreamWithPos:

  1   2   >