[jira] [Created] (FLINK-12610) Introduce planner rules about aggregate
godfrey he created FLINK-12610:
--
Summary: Introduce planner rules about aggregate
Key: FLINK-12610
URL: https://issues.apache.org/jira/browse/FLINK-12610
Project: Flink
Issue Type: New Feature
Components: Table SQL / Planner
Reporter: godfrey he
Assignee: godfrey he
This issue aims to introduce planner rules about aggregate, rules include:
1. {{AggregateCalcMergeRule}}, that recognizes {{Aggregate}} on top of a
{{Calc}} and if possible aggregate through the calc or removes the calc
2. {{AggregateReduceGroupingRule}}, that reduces unless grouping columns
3. {{PruneAggregateCallRule}}, that that removes unreferenced AggregateCall
from Aggregate
4. {{FlinkAggregateRemoveRule}}, that is copied from Calcite's
AggregateRemoveRule, and supports SUM, MIN, MAX, AUXILIARY_GROUP aggregate
functions in non-empty group aggregate
5. {{FlinkAggregateJoinTransposeRule}}, that is copied from Calcite's
AggregateJoinTransposeRule, and supports Left/Right outer join and aggregate
with AUXILIARY_GROUP
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release flink-shaded 7.0, release candidate 2
I found we should also add the `slf4j-api` as a `provided` dependency for `flink-shaded-hadoop-2`. @Nico, Thanks for your check, and the https://issues.apache.org/jira/browse/ FLINK-12598 is not the blocker. @Chesnay, I remember where we already discussed this issue, the problems caused by `provided` scope, and now, It seems nice to have added the changes for the RC3! What do you think? @Chesnay Schepler The PR can be found here: https://github.com/apache/flink-shaded/pull/64 Best, Jincheng Chesnay Schepler 于2019年5月23日周四 下午5:29写道： > I found another issue: > > 1. jackson is bundled twice in the uber jar. > 2. javax annotations are included in the slim jar > > > As for 1), I messed up and modified the shading patterns only for > hadoop-uber, resulting in 2 different shading patterns for jackson. > (jackson is pulled into both jars separately; once explicitly in > shaded-hadoop, and another time in shaded-hadoop-uber as a transitive > dependency of avro) > > I've already fixed this on master. > > 2) happened because we didn't mark findbugs as provided like in the main > project. I'm not quite sure why we aren't doing that, but I'm not too > bothered by this being bundled. This kind of setups where we rely on > Flink to provide certain dependencies are a bit brittle in general. > > > On 23/05/2019 10:26, Chesnay Schepler wrote: > > Worked for me; could you try again without parallel builds? > > > > Which maven version are you using? > > > > On 23/05/2019 09:24, Nico Kruber wrote: > >> Thanks Jincheng, > >> unfortunately, I think, I found a blocker: > >> > >> -1 (non-binding) > >> > >> # not working: > >> - https://issues.apache.org/jira/browse/FLINK-12598 > >> > >> # ok: > >> - checked release notes > >> - verified signatures (sha512, asc) > >> - verified "mvn clean package -Pinclude-netty-tcnative-static" > >> > >> On 22/05/2019 07:32, Hequn Cheng wrote: > >>> Thank you Jincheng for the release! > >>> > >>> +1 (non-binding) > >>> > >>> - Release notes are correct. > >>> - Built from source archive successfully. > >>> - Signatures and hash are correct. > >>> - All artifacts(11 artifacts including flink-shaded) have been > >>> deployed to > >>> the maven central repository. > >>> > >>> One minor comment for the website pull request but I think it is not a > >>> blocker. > >>> > >>> Best, Hequn > >>> > >>> > >>> On Mon, May 20, 2019 at 9:20 AM jincheng sun > > >>> wrote: > >>> > Hi everyone, > > Please review and vote on the release candidate #2 for the version > 7.0, as > follows: > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > The complete staging area is available for your review, which > includes: > * JIRA release notes [1], > * the official Apache source release to be deployed to > dist.apache.org > [2], > which are signed with the key with fingerprint > 8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E [3], > * all artifacts to be deployed to the Maven Central Repository [4], > * source code tag "release-7.0-rc2" [5], > * website pull request listing the new release [6]. > > The vote will be open for at least 72 hours. It is adopted by majority > approval, with at least 3 PMC affirmative votes. > > Thanks, > Jincheng > > [1] > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345226=Html=12315522=Create_token=A5KQ-2QAV-T4JA-FDED%7C8ba061049bec0c5a72dc0191c47bb53a73b82cb4%7Clin > > [2] > https://dist.apache.org/repos/dist/dev/flink/flink-shaded-7.0-rc2/ > [3] https://dist.apache.org/repos/dist/release/flink/KEYS > [4] > > https://repository.apache.org/content/repositories/orgapacheflink-1218 > [5] https://github.com/apache/flink-shaded/tree/release-7.0-rc2 > [6] https://github.com/apache/flink-web/pull/209 > > > > > > >
[jira] [Created] (FLINK-12609) Align the data types of Python with Java
Dian Fu created FLINK-12609: --- Summary: Align the data types of Python with Java Key: FLINK-12609 URL: https://issues.apache.org/jira/browse/FLINK-12609 Project: Flink Issue Type: Sub-task Reporter: Dian Fu Assignee: Dian Fu Currently, there are some data types defined in Java not supported in Python such as TIMESTAMP_WITH_TIME_ZONE, TIMESTAMP_WITH_LOCAL_TIME_ZONE, etc. We should support them in Python once these types have been fully supported in Java. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12608) Add getVertexOrThrow(ExecutionVertexID) to SchedulingTopology
BoWang created FLINK-12608: -- Summary: Add getVertexOrThrow(ExecutionVertexID) to SchedulingTopology Key: FLINK-12608 URL: https://issues.apache.org/jira/browse/FLINK-12608 Project: Flink Issue Type: Sub-task Reporter: BoWang Assignee: BoWang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12607) Introduce a REST API that returns the maxParallelism of a job
Akshay Kanfade created FLINK-12607: -- Summary: Introduce a REST API that returns the maxParallelism of a job Key: FLINK-12607 URL: https://issues.apache.org/jira/browse/FLINK-12607 Project: Flink Issue Type: Improvement Components: Runtime / REST Affects Versions: 1.6.3 Reporter: Akshay Kanfade Today, Flink does not offer any way to get the maxParallelism for a job and it's operators through any of the REST APIs. Since, the internal state already tracks maxParallelism for a job and it's operators, we should expose this via the REST APIs so that application developer can get more insights on the current state. There can be two approaches on how we can do this - Approach 1 : Modify the existing rest API response model to additionally expose a new field 'maxParallelism'. Some of the REST APIs that would be affected by this |h5. */jobs/:jobid/vertices/:vertexid*| |h5. */jobs/:jobid*| Approach 2 : Create a new REST API that would only return maxParallelism for a job and it's operators. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [SURVEY] Usage of flink-ml and [DISCUSS] Delete flink-ml
+1 for the deletion. Also I think it also might be a good idea to update the roadmap for the plan of removal/development since we've reached the consensus on FLIP-39. Thanks, Rong On Wed, May 22, 2019 at 8:26 AM Shaoxuan Wang wrote: > Hi Chesnay, > Yes, you are right. There is not any active commit planned for the legacy > Flink-ml package. It does not matter delete it now or later. I will open a > PR and remove it. > > Shaoxuan > > On Wed, May 22, 2019 at 7:05 PM Chesnay Schepler > wrote: > >> I believe we can remove it regardless since users could just use the 1.8 >> version against future releases. >> >> Generally speaking, any library/connector that is no longer actively >> developed can be removed from the project as existing users can always >> rely on previous versions, which should continue to work by virtue of >> working against @Stable APIs. >> >> On 22/05/2019 12:08, Shaoxuan Wang wrote: >> > Hi Flink community, >> > >> > We plan to delete/deprecate the legacy flink-libraries/flink-ml package >> in >> > Flink1.9, and replace it with the new flink-ml interface proposed in >> FLIP39 >> > (FLINK-12470). >> > Before we remove this package, I want to reach out to you and ask if >> there >> > is any active project still uses this package. Please respond to this >> > thread and outline how you use flink-libraries/flink-ml. >> > Depending on the replies of activity and adoption >> > of flink-libraries/flink-ml, we will decide to either delete this >> package >> > in Flink1.9 or deprecate it for now & remove it in the next release >> after >> > 1.9. >> > >> > Thanks for your attention and help! >> > >> > Regards, >> > Shaoxuan >> > >> >>
[jira] [Created] (FLINK-12606) Move SourceFunction to flink-core
Aljoscha Krettek created FLINK-12606:
Summary: Move SourceFunction to flink-core
Key: FLINK-12606
URL: https://issues.apache.org/jira/browse/FLINK-12606
Project: Flink
Issue Type: Improvement
Components: API / DataStream
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek
As part of the efforts to merge the new Blink-based Table API runner
(FLINK-11439) we need to make {{TableSource}}/{{TableSink}} interfaces
available in {{flink-table-common}}. This requires, among other things, that
the "physical" source interfaces are available in {{flink-core}}, independent
of DataSet/DataStream dependencies. For {{InputFormat}} this is already the
case, we need to do the same for {{SourceFunction}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-12605) Describe new contribution process
Robert Metzger created FLINK-12605: -- Summary: Describe new contribution process Key: FLINK-12605 URL: https://issues.apache.org/jira/browse/FLINK-12605 Project: Flink Issue Type: Task Components: Project Website Reporter: Robert Metzger Assignee: Robert Metzger The community has decided to change the contribution process to seek consensus in Jira first. Update the website to reflect this change. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12604) Register TableSource/Sink as CatalogTable
Dawid Wysakowicz created FLINK-12604: Summary: Register TableSource/Sink as CatalogTable Key: FLINK-12604 URL: https://issues.apache.org/jira/browse/FLINK-12604 Project: Flink Issue Type: Improvement Components: Table SQL / API, Table SQL / Legacy Planner Affects Versions: 1.9.0 Reporter: Dawid Wysakowicz Assignee: Dawid Wysakowicz Fix For: 1.9.0 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12603) Remove getOwningTaskName method from InputGate
zhijiang created FLINK-12603: Summary: Remove getOwningTaskName method from InputGate Key: FLINK-12603 URL: https://issues.apache.org/jira/browse/FLINK-12603 Project: Flink Issue Type: Sub-task Components: Runtime / Network Reporter: zhijiang Assignee: zhijiang Current `InputGate#getOwningTaskName` is only used for logging in related components such as `BarrierBuffer`, `StreamInputProcessor`, etc. We could put this name in the structure of `TaskInfo`, then the related components could get task name directly from `RuntimeEnvironment#getTaskInfo`. To do so, we could simplify the interface of `InputGate`. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: [DISCUSS] Correct the flink pom `artifactId` config before flink 1.9 release
Oh, Thanks for your quick response, I have copied the content to JIRA.
Chesnay Schepler 于2019年5月23日周四 下午7:03写道：
> Please post your mail as a comment into the JIRA to consolidate the
> discussion there.
>
> On 23/05/2019 12:55, jincheng sun wrote:
> > Thank you for confirming this is an issue，and thanks a lot for your
> double
> > check. Due to the script check logic is incorrect, most of the outputs
> are
> > incorrect.
> >
> > You are right and I just pointed out the problem of the scala-free check
> > and not mentioned the final solution which I think we should discuss it.
> >
> > I tried to understand you and have modified the script a bit and have got
> > the same results as you mentioned(In my local).
> > The main changes are as follows:
> > 1) Adds test modules which should be checked:
> > !flink-fs-tests,!flink-yarn-tests,!flink-tests
> > 2) Marks the modules as infected which depend on scala trasitively or
> > depend on modules suffixed with `_{scala_version}`
> >
> > I think we should discuss the rule of how to check whether a module is
> > scala-free or not.
> > If I understand you correctly, the rule in your mind may be as follows:
> >
> > 1) All the modules should check the dependencies(excluding the
> dependencies
> > introduced by the test code). Regarding to modules flink-fs-tests,
> > flink-yarn-tests and flink-tests, the dependencies introduced by the
> > test code should also be checked.
> > 2) The checking rule is whether a module depends on scala trasitively or
> > depends on modules suffixed with `_{scala_version}`.
> > Following the above rules, we can get results you mentioned(Only 3
> modules
> > with incorrect artifact id).
> >
> > Open question:
> >
> > Currently, all the test code are also released into the repository, such
> as
> >
> http://central.maven.org/maven2/org/apache/flink/flink-avro/1.8.0/flink-avro-1.8.0-tests.jar
> > .
> > Users can also depend on these jars. My question is why we need to check
> > the test dependencies for modules flink-fs-tests, flink-yarn-tests and
> > flink-tests,
> > but not check the test dependencies for other modules?
> >
> > The solution:
> > If we follow the rule you mentioned, the change is as follows:
> > 1) Correct the check logic for the script
> > 2) Correct the artifact id for modules: flink-connector-hive,
> > flink-queryable-state-client-java
> > 3) Add the scala dependencies for the module flink-table-api-scala due we
> > plan to add the scala code(I discussed with Timo and Aljoscha)
> >
> > If we should check other test code for other modules, maybe we need more
> > changes. But it depends on the above open question.
> >
> > I have already opened the JIRA:
> > https://issues.apache.org/jira/browse/FLINK-12602
> >
> > Feel free to discuss the solution in this mail thread or in the JIRA.
> >
> > Best,
> > Jincheng
> >
> > Chesnay Schepler 于2019年5月22日周三 下午8:50写道：
> >
> >> You do have a point, but your output is mostly incorrect.
> >>
> >> There are only 3 modules that have a suffix, which don't need it:
> >>
> >> -
> >>
> >> flink-connector-hive
> >>
> >> -
> >>
> >> flink-queryable-state-client-java
> >>
> >> -
> >>
> >> flink-table-api-scala
> >>
> >>
> >> The remaining do need it since they have dependencies with a
> scala-suffix
> >> (mostly on runtime and streaming-java).
> >>
> >> Your change does make sense to me, but there's likely another issue in
> the
> >> preceding logic that determines which module is scala-free.
> >> flink-tests for example should not be considered scala-free, since it
> >> relies on flink-runtime which contains scala and hence the scala-lang
> >> dependencies, but it apparently is given that your change detects
> something.
> >>
> >> In any case, please open a JIRA:
> >>
> >> On 22/05/2019 13:31, jincheng sun wrote:
> >>
> >> Hi all,
> >>
> >> I find a shell issue in `verify_scala_suffixes.sh`(line 145) as follows:
> >> ```
> >> grep "${module}_\d\+\.\d\+" "{}"
> >> ```
> >> This code want to find out all modules that the module's `artifactId`
> with
> >> a `scala_binary_version` suffix.
> >> but the problem is our all `artifactId` value is in the pattern of
> >> `XXX_${scala.binary.version}`, such as:
> >> ```
> >> flink-tests_${scala.binary.version}
> >> ```
> >> then the find out always empty, so this check did not take effect.
> >>
> >> When I correct the script as follows:
> >>
> >> ```
> >> grep "${module}_\\${scala.binary.version}" "{}"
> >> ```
> >> we find there more than 10 modules have incorrect `artifactId` config.
> as
> >> follows:
> >>
> >> 1.flink-connector-hive
> >> 2.flink-fs-tests
> >> 3.flink-queryable-state-client-java
> >> 4.flink-sql-connector-elasticsearch6
> >> 5.flink-sql-connector-kafka
> >> 6.flink-sql-connector-kafka-0.10
> >> 7.flink-sql-connector-kafka-0.11
> >> 8.flink-sql-connector-kafka-0.9
> >> 9.flink-table-api-scala
> >> 10.flink-tests
> >> 11.flink-yarn-tests
> >>
> >> And to fix this issue, we need a big change, such as:
> >>
Re: [DISCUSS] Correct the flink pom `artifactId` config before flink 1.9 release
Please post your mail as a comment into the JIRA to consolidate the
discussion there.
On 23/05/2019 12:55, jincheng sun wrote:
Thank you for confirming this is an issue，and thanks a lot for your double
check. Due to the script check logic is incorrect, most of the outputs are
incorrect.
You are right and I just pointed out the problem of the scala-free check
and not mentioned the final solution which I think we should discuss it.
I tried to understand you and have modified the script a bit and have got
the same results as you mentioned(In my local).
The main changes are as follows:
1) Adds test modules which should be checked:
!flink-fs-tests,!flink-yarn-tests,!flink-tests
2) Marks the modules as infected which depend on scala trasitively or
depend on modules suffixed with `_{scala_version}`
I think we should discuss the rule of how to check whether a module is
scala-free or not.
If I understand you correctly, the rule in your mind may be as follows:
1) All the modules should check the dependencies(excluding the dependencies
introduced by the test code). Regarding to modules flink-fs-tests,
flink-yarn-tests and flink-tests, the dependencies introduced by the
test code should also be checked.
2) The checking rule is whether a module depends on scala trasitively or
depends on modules suffixed with `_{scala_version}`.
Following the above rules, we can get results you mentioned(Only 3 modules
with incorrect artifact id).
Open question:
Currently, all the test code are also released into the repository, such as
http://central.maven.org/maven2/org/apache/flink/flink-avro/1.8.0/flink-avro-1.8.0-tests.jar
.
Users can also depend on these jars. My question is why we need to check
the test dependencies for modules flink-fs-tests, flink-yarn-tests and
flink-tests,
but not check the test dependencies for other modules?
The solution:
If we follow the rule you mentioned, the change is as follows:
1) Correct the check logic for the script
2) Correct the artifact id for modules: flink-connector-hive,
flink-queryable-state-client-java
3) Add the scala dependencies for the module flink-table-api-scala due we
plan to add the scala code(I discussed with Timo and Aljoscha)
If we should check other test code for other modules, maybe we need more
changes. But it depends on the above open question.
I have already opened the JIRA:
https://issues.apache.org/jira/browse/FLINK-12602
Feel free to discuss the solution in this mail thread or in the JIRA.
Best,
Jincheng
Chesnay Schepler 于2019年5月22日周三 下午8:50写道：
You do have a point, but your output is mostly incorrect.
There are only 3 modules that have a suffix, which don't need it:
-
flink-connector-hive
-
flink-queryable-state-client-java
-
flink-table-api-scala
The remaining do need it since they have dependencies with a scala-suffix
(mostly on runtime and streaming-java).
Your change does make sense to me, but there's likely another issue in the
preceding logic that determines which module is scala-free.
flink-tests for example should not be considered scala-free, since it
relies on flink-runtime which contains scala and hence the scala-lang
dependencies, but it apparently is given that your change detects something.
In any case, please open a JIRA:
On 22/05/2019 13:31, jincheng sun wrote:
Hi all,
I find a shell issue in `verify_scala_suffixes.sh`(line 145) as follows:
```
grep "${module}_\d\+\.\d\+" "{}"
```
This code want to find out all modules that the module's `artifactId` with
a `scala_binary_version` suffix.
but the problem is our all `artifactId` value is in the pattern of
`XXX_${scala.binary.version}`, such as:
```
flink-tests_${scala.binary.version}
```
then the find out always empty, so this check did not take effect.
When I correct the script as follows:
```
grep "${module}_\\${scala.binary.version}" "{}"
```
we find there more than 10 modules have incorrect `artifactId` config. as
follows:
1.flink-connector-hive
2.flink-fs-tests
3.flink-queryable-state-client-java
4.flink-sql-connector-elasticsearch6
5.flink-sql-connector-kafka
6.flink-sql-connector-kafka-0.10
7.flink-sql-connector-kafka-0.11
8.flink-sql-connector-kafka-0.9
9.flink-table-api-scala
10.flink-tests
11.flink-yarn-tests
And to fix this issue, we need a big change, such as:
- correct the `artifactId` config.
- update the dependency of relation modules.
- release the connector into the repo.
- update some of the doc for `connector.zh.md` and `connector.md`.
- others
>From the points of my view, It's better to fix this issue before the flink
1.9 release.
What do you think?
NOTE: Please remind me if I have a missing above!
1. The script code
change:https://github.com/sunjincheng121/flink/commit/736f16a8a76aaef1018cc754f0effec119e43120
2. The CI test result: https://api.travis-ci.org/v3/job/535719615/log.txt
Regards,
Jincheng
Re: [DISCUSS] Correct the flink pom `artifactId` config before flink 1.9 release
Thank you for confirming this is an issue，and thanks a lot for your double
check. Due to the script check logic is incorrect, most of the outputs are
incorrect.
You are right and I just pointed out the problem of the scala-free check
and not mentioned the final solution which I think we should discuss it.
I tried to understand you and have modified the script a bit and have got
the same results as you mentioned(In my local).
The main changes are as follows:
1) Adds test modules which should be checked:
!flink-fs-tests,!flink-yarn-tests,!flink-tests
2) Marks the modules as infected which depend on scala trasitively or
depend on modules suffixed with `_{scala_version}`
I think we should discuss the rule of how to check whether a module is
scala-free or not.
If I understand you correctly, the rule in your mind may be as follows:
1) All the modules should check the dependencies(excluding the dependencies
introduced by the test code). Regarding to modules flink-fs-tests,
flink-yarn-tests and flink-tests, the dependencies introduced by the
test code should also be checked.
2) The checking rule is whether a module depends on scala trasitively or
depends on modules suffixed with `_{scala_version}`.
Following the above rules, we can get results you mentioned(Only 3 modules
with incorrect artifact id).
Open question:
Currently, all the test code are also released into the repository, such as
http://central.maven.org/maven2/org/apache/flink/flink-avro/1.8.0/flink-avro-1.8.0-tests.jar
.
Users can also depend on these jars. My question is why we need to check
the test dependencies for modules flink-fs-tests, flink-yarn-tests and
flink-tests,
but not check the test dependencies for other modules?
The solution:
If we follow the rule you mentioned, the change is as follows:
1) Correct the check logic for the script
2) Correct the artifact id for modules: flink-connector-hive,
flink-queryable-state-client-java
3) Add the scala dependencies for the module flink-table-api-scala due we
plan to add the scala code(I discussed with Timo and Aljoscha)
If we should check other test code for other modules, maybe we need more
changes. But it depends on the above open question.
I have already opened the JIRA:
https://issues.apache.org/jira/browse/FLINK-12602
Feel free to discuss the solution in this mail thread or in the JIRA.
Best,
Jincheng
Chesnay Schepler 于2019年5月22日周三 下午8:50写道：
> You do have a point, but your output is mostly incorrect.
>
> There are only 3 modules that have a suffix, which don't need it:
>
>-
>
>flink-connector-hive
>
>-
>
>flink-queryable-state-client-java
>
>-
>
>flink-table-api-scala
>
>
> The remaining do need it since they have dependencies with a scala-suffix
> (mostly on runtime and streaming-java).
>
> Your change does make sense to me, but there's likely another issue in the
> preceding logic that determines which module is scala-free.
> flink-tests for example should not be considered scala-free, since it
> relies on flink-runtime which contains scala and hence the scala-lang
> dependencies, but it apparently is given that your change detects something.
>
> In any case, please open a JIRA:
>
> On 22/05/2019 13:31, jincheng sun wrote:
>
> Hi all,
>
> I find a shell issue in `verify_scala_suffixes.sh`(line 145) as follows:
> ```
> grep "${module}_\d\+\.\d\+" "{}"
> ```
> This code want to find out all modules that the module's `artifactId` with
> a `scala_binary_version` suffix.
> but the problem is our all `artifactId` value is in the pattern of
> `XXX_${scala.binary.version}`, such as:
> ```
> flink-tests_${scala.binary.version}
> ```
> then the find out always empty, so this check did not take effect.
>
> When I correct the script as follows:
>
> ```
> grep "${module}_\\${scala.binary.version}" "{}"
> ```
> we find there more than 10 modules have incorrect `artifactId` config. as
> follows:
>
> 1.flink-connector-hive
> 2.flink-fs-tests
> 3.flink-queryable-state-client-java
> 4.flink-sql-connector-elasticsearch6
> 5.flink-sql-connector-kafka
> 6.flink-sql-connector-kafka-0.10
> 7.flink-sql-connector-kafka-0.11
> 8.flink-sql-connector-kafka-0.9
> 9.flink-table-api-scala
> 10.flink-tests
> 11.flink-yarn-tests
>
> And to fix this issue, we need a big change, such as:
> - correct the `artifactId` config.
> - update the dependency of relation modules.
> - release the connector into the repo.
> - update some of the doc for `connector.zh.md` and `connector.md`.
> - others
>
> >From the points of my view, It's better to fix this issue before the flink
> 1.9 release.
>
> What do you think?
>
> NOTE: Please remind me if I have a missing above!
>
> 1. The script code
> change:https://github.com/sunjincheng121/flink/commit/736f16a8a76aaef1018cc754f0effec119e43120
> 2. The CI test result: https://api.travis-ci.org/v3/job/535719615/log.txt
>
> Regards,
> Jincheng
>
>
>
>
[jira] [Created] (FLINK-12602) Correct the flink pom `artifactId` config and scala-free check logic
sunjincheng created FLINK-12602:
---
Summary: Correct the flink pom `artifactId` config and scala-free
check logic
Key: FLINK-12602
URL: https://issues.apache.org/jira/browse/FLINK-12602
Project: Flink
Issue Type: Bug
Components: Build System
Affects Versions: 1.9.0
Reporter: sunjincheng
I find a shell issue in `verify_scala_suffixes.sh`(line 145) as follows:
{code}
grep "${module}_\d\+\.\d\+" "{}"
{code}
This code want to find out all modules that the module's `artifactId` with a
`scala_binary_version` suffix.
but the problem is our all `artifactId` value is in the pattern of
`XXX_${scala.binary.version}`, such as:
{code}
flink-tests_${scala.binary.version}
{code}
then the result always empty, so this check did not take effect.
I have already initiated a discussion of the issue. Please check the Mail
thread here for details.
http://mail-archives.apache.org/mod_mbox/flink-dev/201905.mbox/%3ccajsjtkw+8mcsc0fvneyaovl_ttrr_uuosx-tfgxj5gfqp1a...@mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (FLINK-12601) Register DataStream/DataSet as DataStream/SetTableOperations in Catalog
Dawid Wysakowicz created FLINK-12601: Summary: Register DataStream/DataSet as DataStream/SetTableOperations in Catalog Key: FLINK-12601 URL: https://issues.apache.org/jira/browse/FLINK-12601 Project: Flink Issue Type: Improvement Components: Table SQL / API, Table SQL / Legacy Planner Reporter: Dawid Wysakowicz Assignee: Dawid Wysakowicz -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12600) Introduce planner rules to do deterministic rewriting on RelNode
godfrey he created FLINK-12600:
--
Summary: Introduce planner rules to do deterministic rewriting on
RelNode
Key: FLINK-12600
URL: https://issues.apache.org/jira/browse/FLINK-12600
Project: Flink
Issue Type: New Feature
Components: Table SQL / Planner
Reporter: godfrey he
Assignee: godfrey he
This issue aims to introduce planner rules to to do deterministic rewriting on
RelNode , rules include:
1. {{FlinkLimit0RemoveRule}} that rewrites `limit 0` to empty {{Values}}
2. {{FlinkRewriteSubQueryRule}} that rewrites a {{Filter}} with condition:
`(select count(*) from T) > 0` to a {{Filter}} with condition: `exists(select *
from T)`
3. {{ReplaceIntersectWithSemiJoinRule}} that rewrites distinct {{Intersect}} to
a distinct {{Aggregate}} on a SEMI {{Join}}.
4. {{ReplaceMinusWithAntiJoinRule}} that rewrite distinct {{Minus}} to a
distinct {{Aggregate}} on an ANTI {{Join}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: [VOTE] Release flink-shaded 7.0, release candidate 2
I found another issue: 1. jackson is bundled twice in the uber jar. 2. javax annotations are included in the slim jar As for 1), I messed up and modified the shading patterns only for hadoop-uber, resulting in 2 different shading patterns for jackson. (jackson is pulled into both jars separately; once explicitly in shaded-hadoop, and another time in shaded-hadoop-uber as a transitive dependency of avro) I've already fixed this on master. 2) happened because we didn't mark findbugs as provided like in the main project. I'm not quite sure why we aren't doing that, but I'm not too bothered by this being bundled. This kind of setups where we rely on Flink to provide certain dependencies are a bit brittle in general. On 23/05/2019 10:26, Chesnay Schepler wrote: Worked for me; could you try again without parallel builds? Which maven version are you using? On 23/05/2019 09:24, Nico Kruber wrote: Thanks Jincheng, unfortunately, I think, I found a blocker: -1 (non-binding) # not working: - https://issues.apache.org/jira/browse/FLINK-12598 # ok: - checked release notes - verified signatures (sha512, asc) - verified "mvn clean package -Pinclude-netty-tcnative-static" On 22/05/2019 07:32, Hequn Cheng wrote: Thank you Jincheng for the release! +1 (non-binding) - Release notes are correct. - Built from source archive successfully. - Signatures and hash are correct. - All artifacts(11 artifacts including flink-shaded) have been deployed to the maven central repository. One minor comment for the website pull request but I think it is not a blocker. Best, Hequn On Mon, May 20, 2019 at 9:20 AM jincheng sun wrote: Hi everyone, Please review and vote on the release candidate #2 for the version 7.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-7.0-rc2" [5], * website pull request listing the new release [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Jincheng [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345226=Html=12315522=Create_token=A5KQ-2QAV-T4JA-FDED%7C8ba061049bec0c5a72dc0191c47bb53a73b82cb4%7Clin [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-7.0-rc2/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1218 [5] https://github.com/apache/flink-shaded/tree/release-7.0-rc2 [6] https://github.com/apache/flink-web/pull/209
Re: [VOTE] Release flink-shaded 7.0, release candidate 2
Worked for me; could you try again without parallel builds? Which maven version are you using? On 23/05/2019 09:24, Nico Kruber wrote: Thanks Jincheng, unfortunately, I think, I found a blocker: -1 (non-binding) # not working: - https://issues.apache.org/jira/browse/FLINK-12598 # ok: - checked release notes - verified signatures (sha512, asc) - verified "mvn clean package -Pinclude-netty-tcnative-static" On 22/05/2019 07:32, Hequn Cheng wrote: Thank you Jincheng for the release! +1 (non-binding) - Release notes are correct. - Built from source archive successfully. - Signatures and hash are correct. - All artifacts(11 artifacts including flink-shaded) have been deployed to the maven central repository. One minor comment for the website pull request but I think it is not a blocker. Best, Hequn On Mon, May 20, 2019 at 9:20 AM jincheng sun wrote: Hi everyone, Please review and vote on the release candidate #2 for the version 7.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-7.0-rc2" [5], * website pull request listing the new release [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Jincheng [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345226=Html=12315522=Create_token=A5KQ-2QAV-T4JA-FDED%7C8ba061049bec0c5a72dc0191c47bb53a73b82cb4%7Clin [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-7.0-rc2/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1218 [5] https://github.com/apache/flink-shaded/tree/release-7.0-rc2 [6] https://github.com/apache/flink-web/pull/209
Re: [VOTE] Release flink-shaded 7.0, release candidate 2
Thanks Jincheng, unfortunately, I think, I found a blocker: -1 (non-binding) # not working: - https://issues.apache.org/jira/browse/FLINK-12598 # ok: - checked release notes - verified signatures (sha512, asc) - verified "mvn clean package -Pinclude-netty-tcnative-static" On 22/05/2019 07:32, Hequn Cheng wrote: > Thank you Jincheng for the release! > > +1 (non-binding) > > - Release notes are correct. > - Built from source archive successfully. > - Signatures and hash are correct. > - All artifacts(11 artifacts including flink-shaded) have been deployed to > the maven central repository. > > One minor comment for the website pull request but I think it is not a > blocker. > > Best, Hequn > > > On Mon, May 20, 2019 at 9:20 AM jincheng sun > wrote: > >> Hi everyone, >> >> Please review and vote on the release candidate #2 for the version 7.0, as >> follows: >> >> [ ] +1, Approve the release >> [ ] -1, Do not approve the release (please provide specific comments) >> >> The complete staging area is available for your review, which includes: >> * JIRA release notes [1], >> * the official Apache source release to be deployed to dist.apache.org >> [2], >> which are signed with the key with fingerprint >> 8FEA1EE9D0048C0CCC70B7573211B0703B79EA0E [3], >> * all artifacts to be deployed to the Maven Central Repository [4], >> * source code tag "release-7.0-rc2" [5], >> * website pull request listing the new release [6]. >> >> The vote will be open for at least 72 hours. It is adopted by majority >> approval, with at least 3 PMC affirmative votes. >> >> Thanks, >> Jincheng >> >> [1] >> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345226=Html=12315522=Create_token=A5KQ-2QAV-T4JA-FDED%7C8ba061049bec0c5a72dc0191c47bb53a73b82cb4%7Clin >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-shaded-7.0-rc2/ >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS >> [4] https://repository.apache.org/content/repositories/orgapacheflink-1218 >> [5] https://github.com/apache/flink-shaded/tree/release-7.0-rc2 >> [6] https://github.com/apache/flink-web/pull/209 >> > -- Nico Kruber | Solutions Architect Follow us @VervericaData Ververica -- Join Flink Forward - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Data Artisans GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen signature.asc Description: OpenPGP digital signature
[jira] [Created] (FLINK-12599) Add Jobgraph zooming with mouse wheel
Nico Kruber created FLINK-12599: --- Summary: Add Jobgraph zooming with mouse wheel Key: FLINK-12599 URL: https://issues.apache.org/jira/browse/FLINK-12599 Project: Flink Issue Type: Improvement Components: Runtime / Web Frontend Affects Versions: 1.9.0 Reporter: Nico Kruber The new web UI looks nice, but I almost didn't find a way back after zooming in: If I click twice into the job graph of a running job, the graph zooms in. After clicking multiple times, it got too large and I wanted to zoom out: * the zoom slider was not clearly visible (blue on a blue operator behind it) * the mouse wheel did not zoom (neither in nor out) Can we make this a bit more visible / accessible? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12598) Flink-shaded's shade-sources profile does not work anymore
Nico Kruber created FLINK-12598:
---
Summary: Flink-shaded's shade-sources profile does not work anymore
Key: FLINK-12598
URL: https://issues.apache.org/jira/browse/FLINK-12598
Project: Flink
Issue Type: Bug
Components: BuildSystem / Shaded
Affects Versions: shaded-7.0
Reporter: Nico Kruber
Fix For: shaded-7.0
{code}
> mvn clean package -Pshade-sources
...
[INFO] --- maven-shade-plugin:3.0.0:shade (shade-flink) @ flink-shaded-hadoop-2
---
[INFO] Excluding org.apache.commons:commons-compress:jar:1.18 from the shaded
jar.
[INFO] Excluding org.apache.avro:avro:jar:1.8.2 from the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded
jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the
shaded jar.
[INFO] Excluding com.thoughtworks.paranamer:paranamer:jar:2.7 from the shaded
jar.
[INFO] Excluding org.xerial.snappy:snappy-java:jar:1.1.4 from the shaded jar.
[INFO] Excluding org.tukaani:xz:jar:1.5 from the shaded jar.
[INFO] Excluding org.slf4j:slf4j-api:jar:1.7.7 from the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-common:jar:2.4.1 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-annotations:jar:2.4.1 in the shaded
jar.
[INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.5 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Including commons-httpclient:commons-httpclient:jar:3.1 in the shaded
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.10 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the
shaded jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding commons-el:commons-el:jar:1.0 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Including net.java.dev.jets3t:jets3t:jar:0.9.0 in the shaded jar.
[INFO] Including org.apache.httpcomponents:httpclient:jar:4.5.3 in the shaded
jar.
[INFO] Including org.apache.httpcomponents:httpcore:jar:4.4.6 in the shaded jar.
[INFO] Excluding com.jamesmurty.utils:java-xmlbuilder:jar:0.4 from the shaded
jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.7 from the
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8.1 from the shaded
jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.15 from the shaded jar.
[INFO] Including com.google.protobuf:protobuf-java:jar:2.5.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-auth:jar:2.4.1 in the shaded jar.
[INFO] Excluding com.jcraft:jsch:jar:0.1.42 from the shaded jar.
[INFO] Including com.google.code.findbugs:jsr305:jar:1.3.9 in the shaded jar.
[WARNING] Could not get sources for
com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] flink-shaded 7.0 ... SUCCESS [ 0.771 s]
[INFO] flink-shaded-force-shading 7.0 . SUCCESS [ 0.951 s]
[INFO] flink-shaded-asm-6 6.2.1-7.0 ... SUCCESS [ 1.469 s]
[INFO] flink-shaded-guava-18 18.0-7.0 . SKIPPED
[INFO] flink-shaded-netty-4 4.1.32.Final-7.0 .. SKIPPED
[INFO] flink-shaded-netty-tcnative-dynamic 2.0.25.Final-7.0 SUCCESS [ 2.195 s]
[INFO] flink-shaded-jackson-parent 2.9.8-7.0 .. SUCCESS [ 0.161 s]
[INFO] flink-shaded-jackson-2 2.9.8-7.0 ... SKIPPED
[INFO] flink-shaded-jackson-module-jsonSchema-2 2.9.8-7.0 . SKIPPED
[INFO] flink-shaded-hadoop-2 2.4.1-7.0 FAILURE [ 2.597 s]
[INFO] flink-shaded-hadoop-2-uber 2.4.1-7.0 ... SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 3.499 s (Wall Clock)
[INFO] Finished at: 2019-05-23T09:22:04+02:00
[INFO]
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade (shade-flink) on
project flink-shaded-hadoop-2: Execution shade-flink of goal
org.apache.maven.plugins:maven-shade-plugin:3.0.0:shade failed.:
NullPointerException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run
[jira] [Created] (FLINK-12597) Remove the legacy flink-libraries/flink-ml
Shaoxuan Wang created FLINK-12597: - Summary: Remove the legacy flink-libraries/flink-ml Key: FLINK-12597 URL: https://issues.apache.org/jira/browse/FLINK-12597 Project: Flink Issue Type: Sub-task Components: Library / Machine Learning Affects Versions: 1.9.0 Reporter: Shaoxuan Wang Assignee: Luo Gen As discussed in dev-ml, we decided to delete the legacy flink-libraries/flink-ml, so as to the flink-libraries/flink-ml-uber. There is not any further development planned for this legacy flink-ml package in 1.9 or even future. Users could just use the 1.8 version if their products/projects still rely on this package. [1] [http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Usage-of-flink-ml-and-DISCUSS-Delete-flink-ml-td29057.html] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12596) Unify all kinds of cluster into ClusterClient in FlinkShell
Jeff Zhang created FLINK-12596: -- Summary: Unify all kinds of cluster into ClusterClient in FlinkShell Key: FLINK-12596 URL: https://issues.apache.org/jira/browse/FLINK-12596 Project: Flink Issue Type: Improvement Components: Scala Shell Affects Versions: 1.8.0 Reporter: Jeff Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (FLINK-12595) KinesisDataFetcherTest.testOriginalExceptionIsPreservedWhenInterruptedDuringShutdown deadlocks
Dawid Wysakowicz created FLINK-12595: Summary: KinesisDataFetcherTest.testOriginalExceptionIsPreservedWhenInterruptedDuringShutdown deadlocks Key: FLINK-12595 URL: https://issues.apache.org/jira/browse/FLINK-12595 Project: Flink Issue Type: Bug Components: Connectors / Kinesis, Tests Affects Versions: 1.9.0 Reporter: Dawid Wysakowicz https://api.travis-ci.org/v3/job/535738122/log.txt -- This message was sent by Atlassian JIRA (v7.6.3#76005)
