[jira] [Created] (FLINK-21107) Move test base classes from flink-tests to flink-test-utils

[Zhenqiu Huang (Jira)]

Zhenqiu Huang created FLINK-21107:
-

 Summary: Move test base classes from flink-tests to 
flink-test-utils
 Key: FLINK-21107
 URL: https://issues.apache.org/jira/browse/FLINK-21107
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Affects Versions: 1.12.0
Reporter: Zhenqiu Huang


There are several test help classes are defined in flink-tests and used in 
modules in flink-streaming-scala, flink-table-planner-blink:

For example:

org.apache.flink.test.streaming.runtime.util.TestListResultSink
org.apache.flink.api.scala.completeness.ScalaAPICompletenessTestBase
org.apache.flink.test.util.SuccessException


But actually, flink-tests module is a place for ITCases. If we want to add 
ITCase for classes (for example remote udf) of flink-table-planner-blink into 
flink-tests, the cycle reference will be created.

Thus, I would suggest moving those helper classes from flink-tests to 
flink-test-utils.






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] FLIP-160: Declarative scheduler

[Xintong Song]

Thanks for preparing the FLIP and starting the discussion, Till.

I have a few questions trying to understand the design.

## What is the relationship between the current and new schedulers?
IIUC, the new declarative scheduler will coexist with the current
scheduler, as an alternative that the user needs to explicitly switch to.
Then does it require any changes to the scheduler interfaces and how the
JobMaster interacts with it?

## Is `SlotAllocator` aware of `ExecutionGraph`?
Looking at the interfaces, it seems to me that `SlotAllocator` only takes
`JobInformation` and `VertexInformation` as topology inputs. However, it
generates `ParallelismAndResourceAssignments` which maps slots to
`ExecutionVertexID`s. It's a bit confusing that `ExecutionGraph` is
generated outside `SlotAllocator` while `ExecutionVertexID`s are generated
inside `SlotAllocator`.

## About `ScaleUpController#canScaleUp`

### What is cumulative parallelism?
The interface shows that the cumulative parallelism is a single number per
job graph. I assume this is the sum of parallelism of all vertices?

### Is this interface expected to be called before or after
`SlotAllocator#determineParallelism`?
IIUC, on new resources appear, the scheduler always calls
`SlotAllocator#determineParallelism` to generate a new plan based on the
available resources, and `ScaleUpController#canScaleUp` is then called to
decide whether to apply the new plan, according to whether the increasement
is significant, how long the job has been running since last restart, etc.
Is that correct?

### The cumulative parallelism may not be enough for deciding whether the
job should be scaled up.
I'm assuming my understanding for the above questions are correct. Please
correct me if not.

I noticed Chesnay's comment on the wiki page about making the decision
based on when the job entered the execution state last time.

In addition to that, there could also be situations where jobs may not
benefit much an increment in cumulative parallelism.
E.g., for a topology A -> B, where A and B are in different slot sharing
groups, and the current parallelism for both A and B are 2. When 1 new slot
appears, `SlotAllocator` may suggest increasing parallelism of A to 3. But
this may not be a significant beneficial change for the job because the
overall performance is still bottlenecked by the parallelism of B.

## Cluster vs. Job configuration
I'm not entirely sure about specifying which scheduler to be used via a
cluster level configuration option. Each job in a Flink cluster has its own
JobMaster, and which scheduler to use is internal to that JobMaster. I
understand that the declarative scheduler requires the cluster to use
declarative resource management. However, other jobs should still be
allowed to use other scheduler implementations that also support
declarative resource management (e.g. the current `DefaultScheduler`).
Maybe we should consider the cluster level configuration option as a
default scheduler, and allow the job to specify a different scheduler in
its execution config. This is similar to how we specify which state backend
to be used.

## Minor: It might be better to also show in the state machine figure that
it can go from `Executing` to `Restarting` when new resources appear.

Thank you~

Xintong Song



On Sat, Jan 23, 2021 at 6:04 AM Steven Wu  wrote:

> Till, thanks a lot for the proposal.
>
> Even if the initial phase is only to support scale-up, maybe the
> "ScaleUpController" interface should be called "RescaleController" so that
> in the future scale-down can be added.
>
> On Fri, Jan 22, 2021 at 7:03 AM Till Rohrmann 
> wrote:
>
> > Hi everyone,
> >
> > I would like to start a discussion about adding a new type of scheduler
> to
> > Flink. The declarative scheduler will first declare the required
> resources
> > and wait for them before deciding on the actual parallelism of a job.
> > Thereby it can better handle situations where resources cannot be fully
> > fulfilled. Moreover, it will act as a building block for the reactive
> mode
> > where Flink should scale to the maximum of the currently available
> > resources.
> >
> > Please find more details in the FLIP wiki document [1]. Looking forward
> to
> > your feedback.
> >
> > [1] https://cwiki.apache.org/confluence/x/mwtRCg
> >
> > Cheers,
> > Till
> >
>

Re: [ANNOUNCE] New formatting rules are now in effect

[Jark Wu]

Hi Matthias,

I also have the same problem when creating a new Java class. This is quite
annoying. Thanks for looking into it and providing a patch. The
patched plugin works well for me.

Regarding the next actions, is it possible to backport this fix to
google-java-format 1.7.0 series, and release a new version for that?  The
latest version of 1.7.0.5 is released in Apr 2020. If this doesn't work,
option #2 also sounds good to me, because users need to download the plugin
anyway.

Best,
Jark



On Sun, 24 Jan 2021 at 03:32, Matthias Pohl  wrote:

> // With this one I am curious now how many of you had the same issue
> without complaining: In the engine team there were 4 out of 4.
>
> It's about the error dialog that pops up when creating a new Java class
> file. Additionally, the Java class is generated but is not formatted
> correctly. Reformatting the file helps. Confirming the dialog is annoying,
> though. I started looking into the issue and found an
> UnsupportedOperationException that led me to the actual cause: A bug in the
> google-java-format plugin version 1.7.0.5 is causing this behavior. I
> provided a more detailed description of my findings in FLINK-21106 [1]
> including a compiled version of the patched plugin.
>
> I want to open the discussion on how we want to deal with it. I see three
> options right now:
> 1. We leave it as it is right now as we consider this to be a minor thing.
> 2. We provide the patched google-java-format plugin as part of the docs.
> 3. We upgrade to Java 11 to be able to upgrade the google-java-format
> plugin as it was already mentioned earlier in the thread.
>
> None of the above options seem to be the right one to go for. Any thoughts
> on this?
>
> Best,
> Matthias
>
> [1] https://issues.apache.org/jira/browse/FLINK-21106
>
>
>
> On Wed, Dec 30, 2020 at 11:05 AM Chesnay Schepler 
> wrote:
>
>> 1) No, it is not possible to exclude certain code blocks from formatting.
>> There is a workaround though for this case; you can add en empty comment
>> (//) to the end of a line to prevent subsequent lines from being added
>> to it.
>> https://github.com/google/google-java-format/issues/137
>>
>> Note that any spotless-specific features would like not help us anyway,
>> unless we'd be fine with not using the IntelliJ plugin.
>>
>> 2) The code style has not been updated yet.
>> The indent choices with the google-java-format is either 2 spaces for
>> everything, or 4 spaces + 8 spaces for continuations.
>> In other words, 4 spaces is simply not an option.
>>
>> On 12/30/2020 9:09 AM, Jark Wu wrote:
>> > Hi,
>> >
>> > I have played with the format plugin these days and found some problems.
>> > Maybe some of them are personal taste.
>> >
>> > 1) Is it possible to disable auto-format for some code blocks?
>> > For example, the format of code generation
>> > StructuredObjectConverter#generateCode is manually
>> >   adjusted for readability. However, the format plugin breaks it and
>> it's
>> > hard to read now.
>> > See before[1] and after[2]. cc @Timo Walther 
>> >
>> > 2) Using 4 spaces or 8 spaces for continuation indent?
>> > AOSP uses 8 spaces for continuation indent.
>> > However, Flink Code Style suggests "Each new line should have one extra
>> > indentation relative to
>> > the line of the called entity" which means 4 spaces.
>> > Personally, I think 4 spaces may be more friendly for Java lambdas.  An
>> > example:
>> >
>> > 8 spaces:
>> >
>> > wrapClassLoader(
>> >  () ->
>> >  environment
>> >  .getModules()
>> >  .forEach(
>> >  (name, entry) ->
>> >  modules.put(
>> >  name,
>> >  createModule(
>> >  entry.asMap(),
>> > classLoader;
>> >
>> > 4 spaces:
>> >
>> > wrapClassLoader(
>> >  () ->
>> >  environment
>> >  .getModules()
>> >  .forEach(
>> >  (name, entry) ->
>> >  modules.put(name, createModule(entry.asMap(),
>> > classLoader;
>> >
>> >
>> >
>> > Best,
>> > Jark
>> >
>> > [1]:
>> >
>> https://github.com/apache/flink/blob/8fc6eaead00d6e98535874b0a137bc59716d260a/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/data/conversion/StructuredObjectConverter.java#L169
>> > [2]:
>> >
>> https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/data/conversion/StructuredObjectConverter.java#L170
>> > [3]:
>> >
>> https://flink.apache.org/contributing/code-style-and-quality-formatting.html#breaking-the-lines-of-too-long-statements
>> >
>> > On Tue, 29 Dec 2020 at 19:19, Chesnay Schepler 
>> wrote:
>> >
>> >> I have filed FLINK-20803
>> >>  for the issue that
>> 

Re: I want to contribute to Apache Flink

[Matthias Pohl]

Hi nicygan,
thanks for your interest in contributing to Apache Flink. There's no need
to ask for permission to contribute, though. Just browse through open Jira
issues [1] and see whether there is one that you would like to solve. You
might want to check out issues labeled as starter issues [2]. You can ask
in the comment section of the corresponding issue to be set as an assignee.
Additionally, you can check out [3] for further details on how to
contribute to Apache Flink.

Best,
Matthias

[1] https://issues.apache.org/jira/projects/FLINK
[2]
https://issues.apache.org/jira/browse/FLINK-21069?jql=project%3DFLINK%20AND%20labels%20%20%3D%20starter%20AND%20status%20NOT%20IN%20(Closed%2C%20Resolved)
[3] https://flink.apache.org/contributing/how-to-contribute.html

On Sat, Jan 23, 2021 at 2:02 AM nicygan  wrote:

> Hi Guys,
>
> I want to contribute to Apache Flink.
> Would you please give me the permission as a contributor?
> My JIRA ID is nicygan.
>
>
>
>
> | |
> nicygan
> |
> |
> read3...@163.com
> |
> 签名由网易邮箱大师定制

Re: [ANNOUNCE] New formatting rules are now in effect

[Matthias Pohl]

// With this one I am curious now how many of you had the same issue
without complaining: In the engine team there were 4 out of 4.

It's about the error dialog that pops up when creating a new Java class
file. Additionally, the Java class is generated but is not formatted
correctly. Reformatting the file helps. Confirming the dialog is annoying,
though. I started looking into the issue and found an
UnsupportedOperationException that led me to the actual cause: A bug in the
google-java-format plugin version 1.7.0.5 is causing this behavior. I
provided a more detailed description of my findings in FLINK-21106 [1]
including a compiled version of the patched plugin.

I want to open the discussion on how we want to deal with it. I see three
options right now:
1. We leave it as it is right now as we consider this to be a minor thing.
2. We provide the patched google-java-format plugin as part of the docs.
3. We upgrade to Java 11 to be able to upgrade the google-java-format
plugin as it was already mentioned earlier in the thread.

None of the above options seem to be the right one to go for. Any thoughts
on this?

Best,
Matthias

[1] https://issues.apache.org/jira/browse/FLINK-21106



On Wed, Dec 30, 2020 at 11:05 AM Chesnay Schepler 
wrote:

> 1) No, it is not possible to exclude certain code blocks from formatting.
> There is a workaround though for this case; you can add en empty comment
> (//) to the end of a line to prevent subsequent lines from being added
> to it.
> https://github.com/google/google-java-format/issues/137
>
> Note that any spotless-specific features would like not help us anyway,
> unless we'd be fine with not using the IntelliJ plugin.
>
> 2) The code style has not been updated yet.
> The indent choices with the google-java-format is either 2 spaces for
> everything, or 4 spaces + 8 spaces for continuations.
> In other words, 4 spaces is simply not an option.
>
> On 12/30/2020 9:09 AM, Jark Wu wrote:
> > Hi,
> >
> > I have played with the format plugin these days and found some problems.
> > Maybe some of them are personal taste.
> >
> > 1) Is it possible to disable auto-format for some code blocks?
> > For example, the format of code generation
> > StructuredObjectConverter#generateCode is manually
> >   adjusted for readability. However, the format plugin breaks it and it's
> > hard to read now.
> > See before[1] and after[2]. cc @Timo Walther 
> >
> > 2) Using 4 spaces or 8 spaces for continuation indent?
> > AOSP uses 8 spaces for continuation indent.
> > However, Flink Code Style suggests "Each new line should have one extra
> > indentation relative to
> > the line of the called entity" which means 4 spaces.
> > Personally, I think 4 spaces may be more friendly for Java lambdas.  An
> > example:
> >
> > 8 spaces:
> >
> > wrapClassLoader(
> >  () ->
> >  environment
> >  .getModules()
> >  .forEach(
> >  (name, entry) ->
> >  modules.put(
> >  name,
> >  createModule(
> >  entry.asMap(),
> > classLoader;
> >
> > 4 spaces:
> >
> > wrapClassLoader(
> >  () ->
> >  environment
> >  .getModules()
> >  .forEach(
> >  (name, entry) ->
> >  modules.put(name, createModule(entry.asMap(),
> > classLoader;
> >
> >
> >
> > Best,
> > Jark
> >
> > [1]:
> >
> https://github.com/apache/flink/blob/8fc6eaead00d6e98535874b0a137bc59716d260a/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/data/conversion/StructuredObjectConverter.java#L169
> > [2]:
> >
> https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/data/conversion/StructuredObjectConverter.java#L170
> > [3]:
> >
> https://flink.apache.org/contributing/code-style-and-quality-formatting.html#breaking-the-lines-of-too-long-statements
> >
> > On Tue, 29 Dec 2020 at 19:19, Chesnay Schepler 
> wrote:
> >
> >> I have filed FLINK-20803
> >>  for the issue that
> >> Till raised.
> >>
> >> google-java-format 1.8 and above require Java 11 to /run/, so we'll
> >> stick with 1.7 for the time being.
> >> This requires a manual installation of the google-java-format plugin,
> >> and remembering to not update the plugin (urgh...).
> >>
> >> Long term we may want to think about requiring Java 11 for /development/
> >> (*not* usage) of Flink.
> >>
> >> On 12/29/2020 12:13 PM, Till Rohrmann wrote:
> >>> I might have found a problem:
> >>>
> >>> In my current setup I am using the google-java-format plugin version
> >>> 1.9.0.0 which uses google-java-format 1.9.0. In our spotless
> >> configuration
> >>> we are using google-java-format 1.7.0, however. 

[jira] [Created] (FLINK-21106) google-java-format Intellij Plugin 1.7.0.5 causes UnsupportedOperationException in IntelliJ

[Matthias (Jira)]

Matthias created FLINK-21106:


 Summary: google-java-format Intellij Plugin 1.7.0.5 causes 
UnsupportedOperationException in IntelliJ
 Key: FLINK-21106
 URL: https://issues.apache.org/jira/browse/FLINK-21106
 Project: Flink
  Issue Type: Bug
Reporter: Matthias


There's a problem with {{google-java-format}} Intellij plugin version 
{{1.7.0.5}} that causes an {{UnsupportedOperationException}} when creating a 
new Java class file. Besides the exception, an error dialog pops up and the 
newly created file is not properly formatted. A simple reformat solves the 
issue.

This problem is caused by a bug that got fixed in the {{google-java-format}} 
plugin's codebase in 
[45fb41a|https://github.com/google/google-java-format/commit/45fb41a7bac3dfe0726601ceb87d1c17bbf494ec].

Unfortunately, this fix got released with the plugin version {{1.8.0.1}} which 
we cannot upgrade to due to our limitations on sticking to Java 8 for now.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (FLINK-21105) Rename ExecEdge to InputProperty

[godfrey he (Jira)]

godfrey he created FLINK-21105:
--

 Summary: Rename ExecEdge to InputProperty
 Key: FLINK-21105
 URL: https://issues.apache.org/jira/browse/FLINK-21105
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / Planner
Reporter: godfrey he
Assignee: godfrey he
 Fix For: 1.13.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)