Hi John, A simple workaround is by setting configuration option `python.fn-execution.memory.managed` as `false`.
Regards, Dian On Fri, Jun 17, 2022 at 6:50 PM John Tipper <john_tip...@hotmail.com> wrote: > Thanks Dian - I’m restricted to 1.13 for the moment because this is > running inside AWS Kinesis. Is there a way to manually bypass that issue? > > J > > Sent from my iPhone > > On 17 Jun 2022, at 04:59, Dian Fu <dian0511...@gmail.com> wrote: > > > >> This error generally occurs in jobs where there are transfers between > Table and datastream. > AFAIK, this issue should have already been fixed, see > https://issues.apache.org/jira/browse/FLINK-26920 and > https://issues.apache.org/jira/browse/FLINK-23133 for more details. > > Regards, > Dian > > On Fri, Jun 17, 2022 at 10:17 AM Xingbo Huang <hxbks...@gmail.com> wrote: > >> Hi John, >> >> Because I can't see your code, I can only provide some possible reasons >> for this error: >> 1. This error generally occurs in jobs where there are transfers between >> Table and datastream. But given that you said you just used the sql + >> python udf, this shouldn't be the case. >> 2. The default value of `taskmanager.memory.managed.consumer-weights` is >> `OPERATOR:70,STATE_BACKEND:70,PYTHON:30`, so in your case, there is >> actually no need to set it to `PYTHON:30` >> 3. In fact, for pure sql+python udf jobs, if you don't set error value >> `PYTHON:0` in `taskmanager.memory.managed.consumer-weights`, I really can't >> think of any situation where this problem will occur. >> >> Best, >> Xingbo >> >> John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 19:41写道： >> >>> Hi Xingbo, >>> >>> Yes, there are a number of temporary views being created, where each is >>> being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit >>> calls to the Table and DataStream APIs. >>> >>> Is this a good pattern or are there caveats I should be aware of please? >>> >>> Many thanks, >>> >>> John >>> >>> >>> ------------------------------ >>> *From:* Xingbo Huang <hxbks...@gmail.com> >>> *Sent:* 16 June 2022 12:34 >>> *To:* John Tipper <john_tip...@hotmail.com> >>> *Cc:* user@flink.apache.org <user@flink.apache.org> >>> *Subject:* Re: The configured managed memory fraction for Python worker >>> process must be within (0, 1], was: %s >>> >>> Hi John, >>> >>> Does your job logic include conversion between Table and DataStream? For >>> example, methods such as `create_temporary_view(path: str, data_stream: >>> DataStream): -> Table` are used. >>> >>> Best, >>> Xingbo >>> >>> John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 18:31写道： >>> >>> Hi Xingbo, >>> >>> I’m afraid I can’t share my code but Flink is 1.13. The main Flink code >>> is running inside Kinesis on AWS so I cannot change the version. >>> >>> Many thanks, >>> >>> John >>> >>> Sent from my iPhone >>> >>> On 16 Jun 2022, at 10:37, Xingbo Huang <hxbks...@gmail.com> wrote: >>> >>> >>> Hi John, >>> >>> Could you provide the code snippet and the version of pyflink you used? >>> >>> Best, >>> Xingbo >>> >>> >>> John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 17:05写道： >>> >>> Hi all, >>> >>> I'm trying to run a PyFlink unit test to test some PyFlink SQL and where >>> my code uses a Python UDF. I can't share my code but the test case is >>> similar to the code here: >>> https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py >>> When >>> I have some simple SQL everything is fine. When I add a more complex query >>> I get an error, which looks like it's memory related. >>> >>> java.lang.IllegalArgumentException: The configured managed memory fraction >>> for Python worker process must be within (0, 1], was: %s. It may be because >>> the consumer type "Python" was missing or set to 0 for the config option >>> "taskmanager.memory.managed.consumer-weights".0.0 >>> >>> >>> >>> In my test case setUp(), I try to set that value like this, but it seems >>> to have no effect: >>> >>> self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights", >>> "PYTHON:30") >>> >>> >>> Am I not setting it correctly, or is there something else I need to do >>> to fix this error? >>> >>> Many thanks, >>> >>> John >>> >>>
