Re: ValidationException using DataTypeHint in Scalar Function

Hi Steve, Thanks for reaching out to the Flink community. I am pulling in Timo who might be able to help you with this question. Best, Jark On Mon, 26 Oct 2020 at 23:10, Steve Whelan wrote: > Hi, > > I have a column of type *RAW('java.util.Map', ?)* that I want to pass to > a scalar function

Re: How to understand NOW() in SQL when using Table & SQL API to develop a streaming app?

Hi Longdexin, In traditional batch sql, NOW() is executed and determined before the job is submitted and will not change for every processed record. However, this doesn't make much sense in streaming sql, therefore, NOW() function in Flink is executed for every record. Best, Jark On Fri, 23 Oct

Re: Rocksdb - Incremental vs full checkpoints

Hi Sudharsan Once enable the incremental checkpoint, the delta size is the same as the size of newly uploaded sst files. Which might not be always the same considering RocksDB's compression ratio, compaction times and time to flush. If you really want to check the details, you could login the m

Re: [SURVEY] Remove Mesos support

Hi Piyush, Thanks a lot for sharing the information. It would be a great relief that you are good with Flink on Mesos as is. As for the jira issues, I believe the most essential ones should have already been resolved. You may find some remaining open issues here [1], but not all of them are neces

Re: Flink checkpointing state

Hi Boris Please refer to FLINK-12884[1] for current progress of native HA support of k8s which targets for release-1.12. [1] https://issues.apache.org/jira/browse/FLINK-12884 Best Yun Tang From: Boris Lublinsky Sent: Tuesday, October 27, 2020 2:56 To: user Su

Re: FLINK 1.11 Graphite Metrics

Hi Chesnay, Another log message: 2020-10-26 23:33:08,516 WARN org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory (org.apache.flink.metrics.graphite.GraphiteReporterFactory) could not be found for reporter grph. Available factories: [org.apache.flink.runtime.metrics.ReporterSetu

Re: how to enable metrics in Flink 1.11

This is what I see on the WebUI. 23:19:24.263 [flink-akka.actor.default-dispatcher-1865] ERROR org.apache.flink.runtime.rest.handler.taskmanager.TaskManagerLogFileHandler - Failed to transfer file from TaskExecutor container_1603649952937_0002_01_04. java.util.concurrent.CompletionException: o

EMR Logging Woes

Hello, After lots of testing in local environments we're now trying to get our cluster running on AWS EMR. We followed much of the documentation from both AWS and Flink and have gotten to the point of creating a yarn session and submitting jobs. We successfully get back a Job ID and in the Yarn Ti

Getting UnsupportedException in Kyro for proto maps

Hi, I have a proto which has a map field in it. When i try to use it in Flink using Kyro serializer, im getting following error when pushing the output to Kafka Caused by: java.lang.UnsupportedOperationException at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057) at com.eso

Re: FLINK 1.11 Graphite Metrics

Hi Chesnay, I have the same, and I am exporting the flinkconf like below, where i have flink-conf.yaml with configuration you have given.What else can I try ? export FLINK_CONF_DIR=${app_install_path}/flinkconf/ regards, Vijay On Sun, Oct 25, 2020 at 8:03 AM Chesnay Schepler wrote: > Ah wait,

Re: Getting UnsupportedException in Kyro for proto maps

Are you registering the protobuf serializer with Kryo? (See https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html#protobuf-via-kryo) From: Sudan S Date: Monday, October 26, 2020 at 11:44 AM To: User-Flink Subject: Getting UnsupportedException in Kyro for proto maps Hi,

Re: [SURVEY] Remove Mesos support

Hi Xintong, Do you have any jiras that cover any of the items on 1 or 2? I can reach out to folks internally and see if I can get some folks to commit to helping out. To cover the other qs: * Yes, we’ve not got a plan at the moment to get off Mesos. We use Yarn for some our Flink workloads

Flink checkpointing state

This is from Flink 1.8: "Job Manager keeps some state related to checkpointing in it’s memory. This state would be lost on Job Manager crashes, which is why this state is persisted in ZooKeeper. This means that even though there is no real need for the leader election and -discovery part of Fli

RE: Runtime Dependency Issues Upgrading to Flink 1.11.2 from 1.9.2

Hi Leonard, Chesnay, thanks for having a look. I was able to sort this out -it was because of the change in default Class Loading policy becoming child-first introduced in 1.10 through https://issues.apache.org/jira/browse/FLINK-13749 . Once I changed it back to parent-first, I was able to subm

RestClusterClient and classpath

Hi to all, I was trying to use the RestClusterClient to submit my job to the Flink cluster. However when I submit the job Flink cannot find the classes contained in the "fat" jar..what should I do? Am I missing something in my code? This is the current client code I'm testing: public static void m

Re: Some questions regarding operator IDs

Hey Kevin, setting the uid is not needed for exactly-once guarantees. It is used if you want to restore the operator state manually using a savepoint. This blog blog post (there are probably a lot more explaining this) could be helpful to understand how the checkpointing ensures exactly once desp

Re: HA on AWS EMR

Hey Averell, you should be able to migrate savepoints from Flink 1.10 to 1.11. Is there a simple way for me to reproduce this issue locally? This seems to be a rare, but probably valid issue. Are you using any special operators? (like the new source API?) Best, Robert On Wed, Oct 21, 2020 at 11

Re: Dependency vulnerabilities with flink 1.11.1 version

Hey Suchithra, thanks a lot for this report. I'm in the process of closing all the tickets Till has created (by pushing version upgrades to Flink). The fixes will be released with the upcoming Flink 1.12 release. I have decided against backporting the fixes to the 1.11 line of Flink, because they

ValidationException using DataTypeHint in Scalar Function

Hi, I have a column of type *RAW('java.util.Map', ?)* that I want to pass to a scalar function UDF. I'm using DataTypeHints but hitting an exception. What would be the proper DataTypeHint and data type param to achieve this? @FunctionHint( input = {@DataTypeHint("RAW"), @DataTypeHint(

Feature request: Removing state from operators

We use the feature for removing stateful operators via the allowNonRestoredState relatively often and it works great. However, there doesn’t seem to be anything like that for removing state from an existing operator (that we want to keep). Say my operator defines a MapState and a ValueState. Lat

Re: KryoException UnsupportedOperationException when writing Avro GenericRecords to Parquet

Hello Till, Adding GenericRecordAvroTypeInfo(schema) does help. Thanks and best regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Un-ignored Parsing Exceptions in the CsvFormat

Hey Austin, I assigned the ticket, that would be great if you could fix it! Regards, Roman On Thu, Oct 22, 2020 at 5:08 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey Roman, > > Sorry to miss this -- thanks for the confirmation and making the ticket. > I'm happy to propose a

Re: Flink - Kafka topic null error; happens only when running on cluster

Hi Timo, Sure, I have opened this issue on Jira. On Fri, Oct 23, 2020 at 4:09 PM Timo Walther wrote: > Hi Manas, > > that is a good point. Feel free to open an issue for this. It is not the > first time that your question appeared on the mailin