Thanks Dian - I’m restricted to 1.13 for the moment because this is running inside AWS Kinesis. Is there a way to manually bypass that issue?
J Sent from my iPhone On 17 Jun 2022, at 04:59, Dian Fu <dian0511...@gmail.com> wrote: >> This error generally occurs in jobs where there are transfers between Table >> and datastream. AFAIK, this issue should have already been fixed, see https://issues.apache.org/jira/browse/FLINK-26920 and https://issues.apache.org/jira/browse/FLINK-23133 for more details. Regards, Dian On Fri, Jun 17, 2022 at 10:17 AM Xingbo Huang <hxbks...@gmail.com<mailto:hxbks...@gmail.com>> wrote: Hi John, Because I can't see your code, I can only provide some possible reasons for this error: 1. This error generally occurs in jobs where there are transfers between Table and datastream. But given that you said you just used the sql + python udf, this shouldn't be the case. 2. The default value of `taskmanager.memory.managed.consumer-weights` is `OPERATOR:70,STATE_BACKEND:70,PYTHON:30`, so in your case, there is actually no need to set it to `PYTHON:30` 3. In fact, for pure sql+python udf jobs, if you don't set error value `PYTHON:0` in `taskmanager.memory.managed.consumer-weights`, I really can't think of any situation where this problem will occur. Best, Xingbo John Tipper <john_tip...@hotmail.com<mailto:john_tip...@hotmail.com>> 于2022年6月16日周四 19:41写道： Hi Xingbo, Yes, there are a number of temporary views being created, where each is being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit calls to the Table and DataStream APIs. Is this a good pattern or are there caveats I should be aware of please? Many thanks, John ________________________________ From: Xingbo Huang <hxbks...@gmail.com<mailto:hxbks...@gmail.com>> Sent: 16 June 2022 12:34 To: John Tipper <john_tip...@hotmail.com<mailto:john_tip...@hotmail.com>> Cc: user@flink.apache.org<mailto:user@flink.apache.org> <user@flink.apache.org<mailto:user@flink.apache.org>> Subject: Re: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s Hi John, Does your job logic include conversion between Table and DataStream? For example, methods such as `create_temporary_view(path: str, data_stream: DataStream): -> Table` are used. Best, Xingbo John Tipper <john_tip...@hotmail.com<mailto:john_tip...@hotmail.com>> 于2022年6月16日周四 18:31写道： Hi Xingbo, I’m afraid I can’t share my code but Flink is 1.13. The main Flink code is running inside Kinesis on AWS so I cannot change the version. Many thanks, John Sent from my iPhone On 16 Jun 2022, at 10:37, Xingbo Huang <hxbks...@gmail.com<mailto:hxbks...@gmail.com>> wrote: Hi John, Could you provide the code snippet and the version of pyflink you used? Best, Xingbo John Tipper <john_tip...@hotmail.com<mailto:john_tip...@hotmail.com>> 于2022年6月16日周四 17:05写道： Hi all, I'm trying to run a PyFlink unit test to test some PyFlink SQL and where my code uses a Python UDF. I can't share my code but the test case is similar to the code here: https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py When I have some simple SQL everything is fine. When I add a more complex query I get an error, which looks like it's memory related. java.lang.IllegalArgumentException: The configured managed memory fraction for Python worker process must be within (0, 1], was: %s. It may be because the consumer type "Python" was missing or set to 0 for the config option "taskmanager.memory.managed.consumer-weights".0.0 In my test case setUp(), I try to set that value like this, but it seems to have no effect: self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights", "PYTHON:30") Am I not setting it correctly, or is there something else I need to do to fix this error? Many thanks, John
