Hi Stefan,

I have upgraded to 1.6.1. I saw the warnings are gone but my issue remains: the 
scenarios: Set[Scenario] cannot be passed as a method argument in order to be 
used in the map function.

But it is working if I just directly use the object variable scenarios: 
Set[Scenario] instead of passing it as a method argument.

Does it have anything to do with the class Scenario?

Many Thanks.

Best regards/祝好,

Chang Liu 刘畅


> On 22 Jan 2019, at 16:34, Chang Liu <[email protected]> wrote:
> 
> Hi Stefan,
> 
> Thanks. I am using 1.6.0. I will upgrade to 1.6.1 and see whether the problem 
> remains.
> 
> Best regards/祝好,
> 
> Chang Liu 刘畅
> 
> 
>> On 22 Jan 2019, at 16:10, Stefan Richter <[email protected] 
>> <mailto:[email protected]>> wrote:
>> 
>> Hi,
>> 
>> Which version of Flink are you using? This issue 
>> https://issues.apache.org/jira/browse/FLINK-10283 
>> <https://issues.apache.org/jira/browse/FLINK-10283> shows that a similar 
>> problem was fixed in 1.6.1 and 1.7. If you use a newer version and still 
>> encounter the problem, you can reopen the issue and describe how this is 
>> still a problem for you.
>> 
>> Best,
>> Stefan 
>> 
>>> On 22. Jan 2019, at 13:49, Chang Liu <[email protected] 
>>> <mailto:[email protected]>> wrote:
>>> 
>>> I have tried another way, it is not working as well:
>>> 
>>> def evaluationStream(
>>>     indicatorsStream: DataStream[Indicators],
>>>     scenarios: Set[Scenario]): DataStream[Evaluation] =
>>>   indicatorsStream.map(new IndicatorsToTxEval(scenarios))
>>> 
>>> class IndicatorsToTxEval(
>>>     scenarios: Set[Scenario])
>>>   extends MapFunction[Indicators, Evaluation] {
>>> 
>>>   override def map(inds: Indicators): Evaluation =
>>>     Evaluation(indicators.id <http://indicator.id/>, 
>>> evaluateScenarios(indicators, scenarios))
>>> }
>>> 
>>> Best regards/祝好,
>>> 
>>> Chang Liu 刘畅
>>> 
>>> 
>>>> On 22 Jan 2019, at 13:33, Chang Liu <[email protected] 
>>>> <mailto:[email protected]>> wrote:
>>>> 
>>>> Ok, I think I found where is the issue, but I don’t understand why.
>>>> 
>>>> I have a method:
>>>> 
>>>> def evaluationStream(
>>>>     indicatorsStream: DataStream[Indicators],
>>>>     scenarios: Set[Scenario]): DataStream[Evaluation] =
>>>>   indicatorsStream.map { indicators =>
>>>>     Evaluation(indicators.id <http://indicator.id/>, 
>>>> evaluateScenarios(indicators, scenarios))
>>>>   }
>>>> 
>>>> And this is how I am using it:
>>>> 
>>>> lazy indicatorsStream: DataStream[Indicators] = ...
>>>> 
>>>> lazy val scenarios: Set[Scenario] = loadScenarios(...)
>>>> 
>>>> lazy val evalStream: DataStream[Evaluation] = 
>>>> evaluationStream(indicatorsStream, scenarios).print()
>>>> 
>>>> 
>>>> The problem is caused by the scenarios, which is passed as an argument of 
>>>> the method evaluationStream. But is is not working.
>>>> 
>>>> It will work if I do it in the following way:
>>>> 
>>>> lazy val scenarios: Set[Scenario] = Set(S1, S2, ...)
>>>> 
>>>> def evaluationStream(indicatorsStream: DataStream[Indicators]): 
>>>> DataStream[Evaluation] =
>>>>   indicatorsStream.map { indicators =>
>>>>     Evaluation(indicators.id <http://indicator.id/>, 
>>>> evaluateScenarios(indicators, scenarios))
>>>>   }
>>>> 
>>>> where the scenarios is not passed as a method argument but is a static 
>>>> object variable.
>>>> 
>>>> But this is not what I want, I would like to have a configurable scenarios 
>>>> which I can load from config file instead of a static object variable.
>>>> 
>>>> Any idea why this is happening? I also have other codes where I am also 
>>>> passing arguments and use them as part of my data flow and they are just 
>>>> working fine.
>>>> 
>>>> Many Thanks.
>>>> 
>>>> Best regards/祝好,
>>>> 
>>>> Chang Liu 刘畅
>>>> 
>>>> 
>>>>> On 22 Jan 2019, at 10:47, Chang Liu <[email protected] 
>>>>> <mailto:[email protected]>> wrote:
>>>>> 
>>>>> Dear community,
>>>>> 
>>>>> I am having a problem releasing the job.
>>>>> 
>>>>> 2019-01-22 10:42:50.098  WARN [Source: Custom Source -> Kafka -> 
>>>>> ConstructTxSepa -> FilterOutFailures -> ObtainActualTxSepa -> 
>>>>> TxSepaStream -> TxStream -> IndicatorsEvalStream -> TxEvalStream -> Sink: 
>>>>> Print to Std. Out (2/4)] [FileCache] - improper use of releaseJob() 
>>>>> without a matching number of createTmpFiles() calls for jobId 
>>>>> 9e1557723a065925e01c7749899547fb
>>>>>  
>>>>> I searched online but only found this: 
>>>>> https://stackoverflow.com/questions/52135604/fix-improper-use-of-releasejob-without-a-matching-number-of-createtmpfiles
>>>>>  
>>>>> <https://stackoverflow.com/questions/52135604/fix-improper-use-of-releasejob-without-a-matching-number-of-createtmpfiles>
>>>>> 
>>>>> However, this warnings are keeping popping up and the job cannot be 
>>>>> released so that my data flow is not working.
>>>>> 
>>>>> But if I remove my last operator, it will work just fine. But my last 
>>>>> operator is justing doing some map operation. I am wondering what could 
>>>>> be the cause of this issue?
>>>>> 
>>>>> Many Thanks :)
>>>>> 
>>>>> Best regards/祝好,
>>>>> 
>>>>> Chang Liu 刘畅
>>>>> 
>>>>> 
>>>> 
>>> 
>> 
> 

Reply via email to