Re: Correct way to package application.properties file with Flink JAR

Okay, I solved the other issue with viewing logs which proved that correct, non-null values are being loaded. I believe I have a different issue altogether so will create a separate thread for that. Thanks for the help Chesnay! On Thu, Oct 22, 2020 at 11:30 AM Manas Kale wrote: > Hi Chesnay, > T

Re: Logs printed when running with minicluster but not printed when submitted as a job

Thank you Chesnay. I found the logs being printed in the standalone session when I used CLI to submit the job. However this only deepens the mystery of the configuration file on the other thread - I see from the logs that the configuration values are being read correctly, but when these values are

Re: Correct way to package application.properties file with Flink JAR

Hi Chesnay, The Config reader has everything static, so I tried using Config.class.getClassLoader().getResourceAsStream("pipeline.properties"); Here's the .pom file for that file: src/main/resources pipeline.properties log4j.properties

Flink Table SQL and MongoDB connector?

Has anyone connected these two? Looking through previous emails and the Flink docs, I've see two mentions of how to hook up MongoDB to Flink. 1) https://github.com/okkam-it/flink-mongodb-test 2) Debezium->Kafka->Flink https://debezium.io/documentation/reference/1.3/connectors/mongodb.html The de

Re: Trying to run Flink tests

1) I don't see anything useful in it . 2) This PR . Thanks for replying, Xintong! On Wed, Oct 21, 2020 at 7:11 PM Xintong Song wrote: > Hi Dan, > > It looks like whil

flink job will restart over and over again if a taskmanager's disk damages

Hi everyone: I met this Exception when a hard disk was damaged: https://issues.apache.org/jira/secure/attachment/13009035/13009035_flink_disk_error.png I checked the code and found that flink will create a temp file when Record length > 5 MB: // SpillingAdaptiveSpanningRecordDeserializer.java

Flink Job Manager Memory Usage Keeps on growing when enabled checkpoint

Hi all, I have a flink job running version 1.10.2, it simply read from a kafka topic with 96 partitions, and output to another kafka topic. It is running in k8s, with 1 JM (not in HA mode), 12 task managers each has 4 slots. The checkpoint persists the snapshot to azure blob storage, checkpoints

Re: Trying to run Flink tests

Hi Dan, It looks like while your tests are executed and passed, the java processes executing those tests did not exit properly. - Could you try execute the command manually and see if there's any useful outputs? You can find the commands by searching "Command was" in the maven logs. - Quick questi

Re: Trying to run Flink tests

Sure, here's a link to the output. I think for this one I used either: - `mvn package -e -X -DfailIfNoTests=false` - or added a `clean` before package. On Wed, Oct 21, 2020 at 2:24 AM Xintong Song wrote: > Woul

Re: Job Restart Failure

Hi All, Any feedback on how this can be resolved? This is causing downtime in production. Thanks On Tue, Oct 20, 2020 at 4:39 PM Navneeth Krishnan wrote: > Hi All, > > I'm facing an issue in our flink application. This happens in version > 1.4.0 and 1.7.2. We have both versions and we are se

expected behavior when Flink job cluster exhausted all restarts

Hi experts, I am running a flink job cluster, the application jar is packaged together with flink in a docker image. The flink job cluster is running in kubernetes, the restart strategy is below restart-strategy: failure-rate restart-strategy.failure-rate.max-failures-per-interval: 20 restart-stra

Re: is it possible to upgrade zookeeper without having job managers restart

Great, thank you :) -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

savepoint failure

Hello all, I am running a Flink job that performs data enrichment. My job has 7 kafka consumers that receive messages for dml statements performed for 7 db tables. Job setup: - Flink is run in k8s in a similar way as it is described here

Re: is it possible to upgrade zookeeper without having job managers restart

Hello Barisa, Seems like you asked the same question as me just ~ a week ago! http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ZooKeeper-connection-SUSPENDING-td38779.html We will have to wait until the feature mentioned in the JIRA ticket is implemented. Best, Kenzyme Le

is it possible to upgrade zookeeper without having job managers restart

We are running flink 1.10 cluster, with 3 zookeepers for HA. I noticed that trying to upgrade the zookeepers, even 1 by one, causes jobmanagers to crash ``` java.net.UnknownHostException: foo-host: Name or service not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)

Re: What does Kafka Error sending fetch request mean for the Kafka source?

And yes my downstream is handling the duplicates in an idempotent way so we are good on that point. But just curious what the behaviour is on the source consumer when that error happens. On Wed, 21 Oct 2020 at 12:04, John Smith wrote: > Hi, running Flink 1.10.0 we see these logs once in a while.

What does Kafka Error sending fetch request mean for the Kafka source?

Hi, running Flink 1.10.0 we see these logs once in a while... 2020-10-21 15: 48:57,625 INFO org.apache.kafka.clients.FetchSessionHandler - [Consumer clientId=consumer-2, groupId=xx-import] Error sending fetch request (sessionId=806089934, epoch=INITIAL) to node 0: org.apache.kafka.common.errors

Configurable Parser

Hi there, In my usecase, I read data from Kafka where in each kafka partition, I have ascending timestamps. Currently, I parse the data from Kafka with a custom deserialization schema so that after parsing, the FlinkKafkaConsumerBase can extract the eventtime ascending timestamps and create p

(iceberg testing) DDL parser error when create hive catalog

Below is my error, thanks for help, I would like to know ,is it a bug ? I followed the website to carry on: https://github.com/apache/iceberg/pull/1464/files Flink SQL> CREATE CATALOG hive_catalog WITH ( > 'type'='iceberg', > 'catalog-type'='hive', > 'uri'='thrift://localhost:9083', >

Re: Logs printed when running with minicluster but not printed when submitted as a job

I see, thanks for that clarification - I incorrectly assumed both methods of submission produce logs in the same place. I will have an update tomorrow! On Wed, Oct 21, 2020 at 6:12 PM Chesnay Schepler wrote: > Hold on, let us clarify how you submit the job. > > Do you upload the jar via the WebU

Re: Logs printed when running with minicluster but not printed when submitted as a job

Hold on, let us clarify how you submit the job. Do you upload the jar via the WebUI, or with the CLI (e.g., ./bin/flink run ...)? If it is the former, then it show up in the JM logs. If it is the latter, then it should appear in the logs of the client (i.e., log/flink-???-client-???.log). O

Re: Correct way to package application.properties file with Flink JAR

You could bundle said file in the jar and retrieve it via getClass().getClassLoader().getResource(""). On 10/21/2020 2:24 PM, Manas Kale wrote: Hi, I have a Flink job that I am packaging as a JAR that is submitted to the Flink cluster runtime. However, this JAR reads a few configuration value

Correct way to package application.properties file with Flink JAR

Hi, I have a Flink job that I am packaging as a JAR that is submitted to the Flink cluster runtime. However, this JAR reads a few configuration values from a .properties file. What is the recommended way to package this properties file when submitting to a cluster? Do I have to copy it to a folder

Re: Logs printed when running with minicluster but not printed when submitted as a job

Hi Chesnay, I checked the JobManager logs - it's not there either. On Wed, Oct 21, 2020 at 3:51 PM Chesnay Schepler wrote: > The main method is executed in the JobManager process and never reaches > the TaskExecutors (only the individual functions do). > As such you have to take a peek into the

KryoException UnsupportedOperationException when writing Avro GenericRecords to Parquet

Hi, I'm trying to convert a stream of JSON string to a stream of Avro GenericRecords, and write this to parquet files, but I get the exception. This exception came at the line /out.collect(genericRecord)/. If there's no sink then there's no error. /KryoException: java.lang.UnsupportedOperationExce

Re: Logs printed when running with minicluster but not printed when submitted as a job

The main method is executed in the JobManager process and never reaches the TaskExecutors (only the individual functions do). As such you have to take a peek into the JobManager logs. On 10/21/2020 11:37 AM, Manas Kale wrote: Hi, I have the following pattern: public static void main(String[] a

Logs printed when running with minicluster but not printed when submitted as a job

Hi, I have the following pattern: public static void main(String[] args) { // Get the exec environment. This could be a cluster or a mini-cluster used for local development. StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); // Make the Flink

Exception in thread "main" org.apache.flink.table.api.ValidationException: Could not resolve over call.

I'm learning Official document-Over Window Aggregation My pojo is: https://paste.ubuntu.com/p/kPydGd2Cwd/ Completed test code is: https://paste.ubuntu.com/p/KyCVxBz254/ The error I got is: Exception in thread "main" org.apache.flink.table.api.ValidationException: Could not resolve over c

Re: Building Flink on VirtualBox VM failing

Hmm Even when setting the forkcounts to 1 things fail. I wonder why there seem to be five of these JVM crashes. There should be one JVM at time. And Maven should fail after the 1st fail? ~/apache-maven-3.2.5/bin/mvn -Dflink.forkCount=1 -Dflink.forkCountTestPackage=1 clean verify [INFO]

Re: Trying to run Flink tests

Would you be able to share the complete maven logs and the command? And what is the maven version? Thank you~ Xintong Song On Wed, Oct 21, 2020 at 1:37 AM Dan Hill wrote: > Hi Xintong! > > No changes. I tried -X and no additional log information is logged. > -DfailIfNoTests=false does not h

Re: HA on AWS EMR

Hello Roman, Thanks for the answer. I have already had that high-availability.storageDir configured to an S3 location. Our service is not critical enough, so to save the cost, we are using the single-master EMR setup. I understand that we'll not get YARN HA in that case, but what I expect here is

Re: Building Flink on VirtualBox VM failing

Hi, You're right, I thought about this also after writing the last comment - for example on Linux, the Kernel by default overcommits memory allocations and this approach doesn't work (doesn't make JVM crash right when it starts). I dug a little deeper. It seems that for ci-environments there are

Re: Issue with Flink - unrelated executeSql causes other executeSqls to fail.

Alright. Thanks for letting me know. I will take a look at the PR. Cheers, Till On Tue, Oct 20, 2020 at 8:00 PM Dan Hill wrote: > -others. > > I can't add you as a watcher to the issue (I lack permissions). > https://issues.apache.org/jira/projects/FLINK/issues/FLINK-19721?filter=reportedbyme >