>> This error generally occurs in jobs where there are transfers between Table and datastream. AFAIK, this issue should have already been fixed, see https://issues.apache.org/jira/browse/FLINK-26920 and https://issues.apache.org/jira/browse/FLINK-23133 for more details.
Regards, Dian On Fri, Jun 17, 2022 at 10:17 AM Xingbo Huang <hxbks...@gmail.com> wrote: > Hi John, > > Because I can't see your code, I can only provide some possible reasons > for this error: > 1. This error generally occurs in jobs where there are transfers between > Table and datastream. But given that you said you just used the sql + > python udf, this shouldn't be the case. > 2. The default value of `taskmanager.memory.managed.consumer-weights` is > `OPERATOR:70,STATE_BACKEND:70,PYTHON:30`, so in your case, there is > actually no need to set it to `PYTHON:30` > 3. In fact, for pure sql+python udf jobs, if you don't set error value > `PYTHON:0` in `taskmanager.memory.managed.consumer-weights`, I really can't > think of any situation where this problem will occur. > > Best, > Xingbo > > John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 19:41写道： > >> Hi Xingbo, >> >> Yes, there are a number of temporary views being created, where each is >> being created using SQL (CREATE TEMPORARY VIEW ...) rather than explicit >> calls to the Table and DataStream APIs. >> >> Is this a good pattern or are there caveats I should be aware of please? >> >> Many thanks, >> >> John >> >> >> ------------------------------ >> *From:* Xingbo Huang <hxbks...@gmail.com> >> *Sent:* 16 June 2022 12:34 >> *To:* John Tipper <john_tip...@hotmail.com> >> *Cc:* user@flink.apache.org <user@flink.apache.org> >> *Subject:* Re: The configured managed memory fraction for Python worker >> process must be within (0, 1], was: %s >> >> Hi John, >> >> Does your job logic include conversion between Table and DataStream? For >> example, methods such as `create_temporary_view(path: str, data_stream: >> DataStream): -> Table` are used. >> >> Best, >> Xingbo >> >> John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 18:31写道： >> >> Hi Xingbo, >> >> I’m afraid I can’t share my code but Flink is 1.13. The main Flink code >> is running inside Kinesis on AWS so I cannot change the version. >> >> Many thanks, >> >> John >> >> Sent from my iPhone >> >> On 16 Jun 2022, at 10:37, Xingbo Huang <hxbks...@gmail.com> wrote: >> >> >> Hi John, >> >> Could you provide the code snippet and the version of pyflink you used? >> >> Best, >> Xingbo >> >> >> John Tipper <john_tip...@hotmail.com> 于2022年6月16日周四 17:05写道： >> >> Hi all, >> >> I'm trying to run a PyFlink unit test to test some PyFlink SQL and where >> my code uses a Python UDF. I can't share my code but the test case is >> similar to the code here: >> https://github.com/apache/flink/blob/f8172cdbbc27344896d961be4b0b9cdbf000b5cd/flink-python/pyflink/testing/test_case_utils.py >> When >> I have some simple SQL everything is fine. When I add a more complex query >> I get an error, which looks like it's memory related. >> >> java.lang.IllegalArgumentException: The configured managed memory fraction >> for Python worker process must be within (0, 1], was: %s. It may be because >> the consumer type "Python" was missing or set to 0 for the config option >> "taskmanager.memory.managed.consumer-weights".0.0 >> >> >> >> In my test case setUp(), I try to set that value like this, but it seems >> to have no effect: >> >> self.t_env.get_config().get_configuration().set_string("taskmanager.memory.managed.consumer-weights", >> "PYTHON:30") >> >> >> Am I not setting it correctly, or is there something else I need to do to >> fix this error? >> >> Many thanks, >> >> John >> >>
