Re: Flink Streaming Job Task is getting cancelled silently and causing job to restart
Hi Erik, Are your suggesting all options together ? Which of version of flink has this solved ? I am currently using 1.5.5 . -Thanks Sohi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: BucketingSink - Could not invoke truncate while recovering from state
Any help ? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
Re: StandAlone job on k8s fails with "Unknown method truncate" on restore
Any one ? I am sure there are hadoop 2.6 integrations with 1.7.1 OR I am overlooking something... On Fri, Feb 15, 2019 at 2:44 PM Vishal Santoshi wrote: > Not sure, but it seems this > https://issues.apache.org/jira/browse/FLINK-10203 may be a connected > issue. > > On Fri, Feb 15, 2019 at 11:57 AM Vishal Santoshi < > vishal.santo...@gmail.com> wrote: > >> That log does not appear. It looks like we have egg and chicken issue. >> >> 2019-02-15 16:49:15,045 DEBUG org.apache.hadoop.hdfs.DFSClient >> - Connecting to datanode 10.246.221.10:50010 >> >> 2019-02-15 16:49:15,045 DEBUG >> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.SaslDataTransferClient >> - SASL client skipping handshake in unsecured configuration for >> >> addr = /10.246.221.10, datanodeId = DatanodeInfoWithStorage[ >> 10.246.221.10:50010,DS-c57a7667-f697-4f03-9fb1-532c5b82a9e8,DISK] >> >> 2019-02-15 16:49:15,072 DEBUG >> org.apache.flink.runtime.fs.hdfs.HadoopFsFactory - >> Instantiating for file system scheme hdfs Hadoop File System >> org.apache.hadoop.hdfs.DistributedFileSystem >> >> 2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal >> - dfs.client.use.legacy.blockreader.local = false >> >> 2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal >> - dfs.client.read.shortcircuit = false >> >> 2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal >> - dfs.client.domain.socket.data.traffic = false >> >> 2019-02-15 16:49:15,072 DEBUG org.apache.hadoop.hdfs.BlockReaderLocal >> - dfs.domain.socket.path = >> >> 2019-02-15 16:49:15,076 DEBUG org.apache.hadoop.io.retry.RetryUtils >> - multipleLinearRandomRetry = null >> >> 2019-02-15 16:49:15,076 DEBUG org.apache.hadoop.ipc.Client >> - getting client out of cache: >> org.apache.hadoop.ipc.Client@31920ade >> >> 2019-02-15 16:49:15,076 DEBUG >> org.apache.hadoop.hdfs.protocol.datatransfer.sasl.DataTransferSaslUtil - >> DataTransferProtocol not using SaslPropertiesResolver, no QOP found in >> configuration for dfs.data.transfer.protection >> >> 2019-02-15 16:49:15,080 INFO >> org.apache.flink.streaming.api.functions.sink.filesystem.Buckets - >> Subtask 3 initializing its state (max part counter=58). >> >> 2019-02-15 16:49:15,081 DEBUG >> org.apache.flink.streaming.api.functions.sink.filesystem.Buckets - >> Subtask 3 restoring: BucketState for >> bucketId=ls_kraken_events/dt=2019-02-14/evt=ad_fill and >> bucketPath=hdfs://nn-crunchy:8020/tmp/kafka-to-hdfs/ls_kraken_events/dt=2019-02-14/evt=ad_fill, >> has open part file created @ 1550247946437 >> >> 2019-02-15 16:49:15,085 DEBUG org.apache.hadoop.ipc.Client >> - IPC Client (1270836494) connection to >> nn-crunchy.bf2.tumblr.net/10.246.199.154:8020 from root sending #56 >> >> 2019-02-15 16:49:15,188 DEBUG org.apache.hadoop.ipc.Client >> - IPC Client (1270836494) connection to >> nn-crunchy.bf2.tumblr.net/10.246.199.154:8020 from root got value #56 >> >> 2019-02-15 16:49:15,196 INFO org.apache.flink.runtime.taskmanager.Task >> - Source: Custom Source -> (Sink: Unnamed, Process -> >> Timestamps/Watermarks) (4/4) (f73403ac4763c99e6a244cba3797f7e9) switched >> from RUNNING to FAILED. >> >> java.io.IOException: Missing data in tmp file: >> hdfs://nn-crunchy:8020/tmp/kafka-to-hdfs/ls_kraken_events/dt=2019-02-14/evt=ad_fill/ >> .part-3-32.inprogress.da2a75d1-0c83-47bc-9c83-950360c55c86 >> >> at >> org.apache.flink.runtime.fs.hdfs.HadoopRecoverableFsDataOutputStream.(HadoopRecoverableFsDataOutputStream.java:93) >> >> >> >> >> >> >> I do see >> >> >> 2019-02-15 16:47:33,582 INFO >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner >> - Current Hadoop/Kerberos user: root >> >> 2019-02-15 16:47:33,582 INFO >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner >> - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - >> 1.8/25.181-b13 >> >> 2019-02-15 16:47:33,582 INFO >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner >> - Maximum heap size: 1204 MiBytes >> >> 2019-02-15 16:47:33,582 INFO >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner >> - JAVA_HOME: /docker-java-home >> >> 2019-02-15 16:47:33,585 INFO >> org.apache.flink.runtime.taskexecutor.TaskManagerRunner >> - Hadoop version: 2.7.5 >> >> >> >> which has to be expected given that we are running the hadoop27flink >> 1.7.1 version. >> >> >> >> Does it make sense to go with a hadoop less version and inject the >> required jar files ? Has that been done by anyone ? >> >> >> >> >> >> >> On Fri, Feb 15, 2019 at 2:33 AM Yun Tang wrote: >> >>> Hi >>> >>> When 'RollingSink' try to initialize state, it would first check current >>> file system supported truncate method. If file system not supported, it >>> would use another work-around solution, which means you
flink sink es
flink sink es exception，也暂时没有思路。2019-02-23 19:46:34.460 +0800 [Source:
order_label_map -> Flat Map -> Map (1/1)] WARN
[com.imdada.parse.OrderLabelParse] [AbstractParseModel.scala:20] - fail parse
json
object:{"eventType":1,"execTime":1550820023000,"schema":"order_detail","table":"order_label_map","tableName":"order_label_map","columnValues":{"last_update_time":"2019-02-22
15:20:23","create_time":"2019-02-22
15:20:23","is_del":"0","only_sign":"","id":"2635475064","order_id":"280572309811180","label_id":"398"},"updatedValues":{}}
org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
Could not forward element to next operator at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:596)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:712)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:690)
at
org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
at
com.imdada.dw.parse.AbstractParseModel$class.flatMap(AbstractParseModel.scala:24)
at com.imdada.parse.OrderLabelParse.flatMap(OrderLabelParse.scala:13) at
com.imdada.parse.OrderLabelParse.flatMap(OrderLabelParse.scala:13) at
org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:50)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:712)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:690)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
at
org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
at
org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:89)
at
org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:154)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:738)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:94)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:58)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) at
java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException:
Buffer pool is destroyed. at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.pushToRecordWriter(RecordWriterOutput.java:110)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:89)
at
org.apache.flink.streaming.runtime.io.RecordWriterOutput.collect(RecordWriterOutput.java:45)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:712)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:690)
at
org.apache.flink.streaming.api.operators.StreamMap.processElement(StreamMap.java:41)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
... 26 common frames omitted Caused by: java.lang.IllegalStateException:
Buffer pool is destroyed. at
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestMemorySegment(LocalBufferPool.java:230)
at
org.apache.flink.runtime.io.network.buffer.LocalBufferPool.requestBufferBuilderBlocking(LocalBufferPool.java:204)
at
org.apache.flink.runtime.io.network.api.writer.RecordWriter.requestNewBufferBuilder(RecordWriter.java:213)
at
flink sink es
阿里大神，这问题无思路了。偶先。暂时不清楚会不会丢数据。从falcon对es的监控来看，es正常。2019-02-23 19:46:30.965 +0800
[Source: jd_mall_order -> Flat Map -> (Sink: Unnamed, Sink: Unnamed) (1/1)]
WARN [com.imdada.parse.PublishTimeParse] [AbstractParseModel.scala:20] - fail
parse json
object:{"eventType":1,"execTime":1550879841000,"schema":"plat_order","table":"jd_mall_order","tableName":"jd_mall_order","columnValues":{"update_time":"2019-02-23
07:57:21","create_time":"2019-02-23
07:57:21","jd_type":"9","deliver_company_name":"?","jd_order_status":"0","order_source":"1","jd_express_no":"88048920204","id":"723109463","order_id":"280587990738384","jd_order_no":"88048920204","deliver_company_id":"566399"},"updatedValues":{}}
org.apache.flink.streaming.runtime.tasks.ExceptionInChainedOperatorException:
Could not forward element to next operator at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:596)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:649)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$BroadcastingOutputCollector.collect(OperatorChain.java:602)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:712)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:690)
at
org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
at
com.imdada.dw.parse.AbstractParseModel$class.flatMap(AbstractParseModel.scala:24)
at com.imdada.parse.PublishTimeParse.flatMap(PublishTimeParse.scala:13)
at com.imdada.parse.PublishTimeParse.flatMap(PublishTimeParse.scala:13)
at
org.apache.flink.streaming.api.operators.StreamFlatMap.processElement(StreamFlatMap.java:50)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:554)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:534)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:712)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:690)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collectWithTimestamp(StreamSourceContexts.java:111)
at
org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordWithTimestamp(AbstractFetcher.java:398)
at
org.apache.flink.streaming.connectors.kafka.internal.Kafka010Fetcher.emitRecord(Kafka010Fetcher.java:89)
at
org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:154)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:738)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:94)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:58)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:99)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:300)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:711) at
java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: An
error occurred in ElasticsearchSink. at
org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.checkErrorAndRethrow(ElasticsearchSinkBase.java:379)
at
org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.invoke(ElasticsearchSinkBase.java:304)
at
org.apache.flink.streaming.api.functions.sink.SinkFunction.invoke(SinkFunction.java:52)
at
org.apache.flink.streaming.api.operators.StreamSink.processElement(StreamSink.java:56)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:579)
... 28 common frames omitted Caused by:
org.elasticsearch.client.transport.NoNodeAvailableException: None of the
configured nodes were available:
[{basedata04}{EQmKCsxLQZmuAHxNPS2Qwg}{qT-kaeaJRFiG3inAed7Ulg}{10.9.45.166}{10.9.45.166:9300},
