[jira] [Created] (FLINK-18611) Include `m` as time unit expression for MINUTE in SQL

2020-07-15 Thread Q Kang (Jira)
Q Kang created FLINK-18611: -- Summary: Include `m` as time unit expression for MINUTE in SQL Key: FLINK-18611 URL: https://issues.apache.org/jira/browse/FLINK-18611 Project: Flink Issue Type:

Using md5 hash while sinking files to s3

2020-07-15 Thread nikita Balakrishnan
Hello team, I’m developing a system where we are trying to sink to an immutable s3 bucket. This bucket has server side encryption set as KMS. The DataStream sink works perfectly fine when I don’t use the immutable bucket but when I use an immutable bucket, I get exceptions regarding multipart

Performance test Flink vs Storm

2020-07-15 Thread Prasanna kumar
Hi, We are testing flink and storm for our streaming pipelines on various features. In terms of Latency,i see the flink comes up short on storm even if more CPU is given to it. Will Explain in detail. *Machine*. t2.large 4 core 16 gb. is used for Used for flink task manager and storm supervisor

[jira] [Created] (FLINK-18610) Clean up Table connector docs grammar

2020-07-15 Thread Seth Wiesman (Jira)
Seth Wiesman created FLINK-18610: Summary: Clean up Table connector docs grammar Key: FLINK-18610 URL: https://issues.apache.org/jira/browse/FLINK-18610 Project: Flink Issue Type:

Re: [DISCUSS] FLIP-129: Refactor Descriptor API to register connector in Table API

2020-07-15 Thread Leonard Xu
Thanks Jark bring this discussion and organize the FLIP document. Thanks Dawid and Timo for the feedback. Here are my thoughts. 1) I’m +1 with using column() for both cases. 2) Expression DSL vs pure SQL string for computed columns I think we can support them both and implement the pure SQL

Re: PackagedProgram and ProgramDescription

2020-07-15 Thread Chesnay Schepler
Let's try this again; the formatting went haywire for some reason... public class MetaDataUtils {     public static ExecutionConfig.GlobalJobParameters createMetaData(ParameterTool parameterTool) {     Map metaData = new HashMap<>(parameterTool.toMap());     setFromManifest(metaData,

Re: PackagedProgram and ProgramDescription

2020-07-15 Thread Chesnay Schepler
For completeness sake, here's an example of what we're doing to add the job arguments and some manifest entries to the global job parameters: (Manifests is a class from jcabi-manifests) public class MetaDataUtils { public static ExecutionConfig.GlobalJobParameters

Re: Improvement idea: Naming the maven modules.

2020-07-15 Thread Kurt Young
+1, I also like this idea. Best, Kurt On Wed, Jul 15, 2020 at 7:10 PM Niels Basjes wrote: > Ok, > > I'll put up a fix > https://issues.apache.org/jira/browse/FLINK-18607 > > Niels > > On Wed, Jul 15, 2020 at 11:23 AM Aljoscha Krettek > wrote: > > > Hi, > > > > I like the proposal! I remember

[jira] [Created] (FLINK-18609) The user-defined jdbc-source process before add window ,job finished early.

2020-07-15 Thread ZHangTianLong (Jira)
ZHangTianLong created FLINK-18609: - Summary: The user-defined jdbc-source process before add window ,job finished early. Key: FLINK-18609 URL: https://issues.apache.org/jira/browse/FLINK-18609

Re: PackagedProgram and ProgramDescription

2020-07-15 Thread Flavio Pompermaier
Thanks Chesnay for the tip. I'll try to investigate the usage of GlobalJobParameters. On Wed, Jul 15, 2020 at 2:51 PM Chesnay Schepler wrote: > The more we strive towards a model where an application can submit > multiple jobs it will become increasingly important to be able to attach > meta

Re: PackagedProgram and ProgramDescription

2020-07-15 Thread Chesnay Schepler
The more we strive towards a model where an application can submit multiple jobs it will become increasingly important to be able to attach meta data to a job/application to have any idea what is going on. But I don't think the PackagedProgram/ProgramDescription is the way to go; and I'd

[jira] [Created] (FLINK-18608) CustomizedConvertRule#convertCast drops nullability

2020-07-15 Thread Flavio Pompermaier (Jira)
Flavio Pompermaier created FLINK-18608: -- Summary: CustomizedConvertRule#convertCast drops nullability Key: FLINK-18608 URL: https://issues.apache.org/jira/browse/FLINK-18608 Project: Flink

Re: PackagedProgram and ProgramDescription

2020-07-15 Thread Flavio Pompermaier
Ok, it's not a problem for me if the community is not interested in pushing this thing forward. When we develop a Job is super useful for us to have the job describing itself somehow (what it does and which parameters it requires). If this is not in Flink I have to implement it somewhere else but

Re: Improvement idea: Naming the maven modules.

2020-07-15 Thread Niels Basjes
Ok, I'll put up a fix https://issues.apache.org/jira/browse/FLINK-18607 Niels On Wed, Jul 15, 2020 at 11:23 AM Aljoscha Krettek wrote: > Hi, > > I like the proposal! I remember that Beam also had more human-readable > names for the modules and found that helpful. Also, changing the names >

[jira] [Created] (FLINK-18607) Give the maven modules human readable names.

2020-07-15 Thread Niels Basjes (Jira)
Niels Basjes created FLINK-18607: Summary: Give the maven modules human readable names. Key: FLINK-18607 URL: https://issues.apache.org/jira/browse/FLINK-18607 Project: Flink Issue Type:

Re: Use of SinkFunction.Context without generic parameter.

2020-07-15 Thread Niels Basjes
Hi, Cool. I'm going to put up a fix ( https://issues.apache.org/jira/browse/FLINK-18606 ) I think that even people who ARE making their own Sink are not affected because the method signature of the invoke does not use the parameter. Niels. On Wed, Jul 15, 2020 at 11:32 AM Aljoscha Krettek

[jira] [Created] (FLINK-18606) Remove generic parameter from SinkFunction.Context

2020-07-15 Thread Niels Basjes (Jira)
Niels Basjes created FLINK-18606: Summary: Remove generic parameter from SinkFunction.Context Key: FLINK-18606 URL: https://issues.apache.org/jira/browse/FLINK-18606 Project: Flink Issue

[VOTE] Release 1.11.1, release candidate #1

2020-07-15 Thread Dian Fu
Hi everyone, Please review and vote on the release candidate #1 for the version 1.11.1, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes

Re: PackagedProgram and ProgramDescription

2020-07-15 Thread Aljoscha Krettek
I think no-one is interested to push this personally right now. We would need a champion that is interested and pushes this forward. Best, Aljoscha On Mon, Mar 30, 2020, at 12:45, Flavio Pompermaier wrote: > I would personally like to see a way of describing a Flink job/pipeline > (including

Re: [DISCUSS] FLIP-130: Support for Python DataStream API (Stateless Part)

2020-07-15 Thread Aljoscha Krettek
Hi, thanks for the proposal! I have some comments about the API. We should not blindly copy the existing Java DataSteam because we made some mistakes with that and we now have a chance to fix them and not forward them to a new API. I don't think we need SingleOutputStreamOperator, in the Scala

Re: Use of SinkFunction.Context without generic parameter.

2020-07-15 Thread Aljoscha Krettek
Hi, this was actually my mistake back then. :-O I'm open to removing the generic parameter from Context if we are sure that it won't break user code. I think it doesn't, because you cannot actually use it with the generic parameter, as you found. Also, I think custom sink implementations in

Re: Improvement idea: Naming the maven modules.

2020-07-15 Thread Aljoscha Krettek
Hi, I like the proposal! I remember that Beam also had more human-readable names for the modules and found that helpful. Also, changing the names shouldn't change anything for users because dependencies are referred to by group/artifactId, it really just makes build output and IDE a bit more

Re: [DISCUSS] FLIP-129: Refactor Descriptor API to register connector in Table API

2020-07-15 Thread Timo Walther
Hi Jark, thanks for working on this issue. It is time to fix this last part of inconsistency in the API. I also like the core parts of the FLIP, esp. that TableDescriptor is one entity that can be passed to different methods. Here is some feedback from my side: 1) +1 for just `column(...)`

[jira] [Created] (FLINK-18605) Refactor Descriptor API (TableEnvironment#connect)

2020-07-15 Thread Jark Wu (Jira)
Jark Wu created FLINK-18605: --- Summary: Refactor Descriptor API (TableEnvironment#connect) Key: FLINK-18605 URL: https://issues.apache.org/jira/browse/FLINK-18605 Project: Flink Issue Type: New

[jira] [Created] (FLINK-18604) HBase Descriptor can not work in Table API

2020-07-15 Thread Leonard Xu (Jira)
Leonard Xu created FLINK-18604: -- Summary: HBase Descriptor can not work in Table API Key: FLINK-18604 URL: https://issues.apache.org/jira/browse/FLINK-18604 Project: Flink Issue Type: Bug