[jira] [Created] (FLINK-12043) Null value check in array serializers classes

[Quan Shi (JIRA)]

Quan Shi created FLINK-12043:


 Summary: Null value check in array serializers classes
 Key: FLINK-12043
 URL: https://issues.apache.org/jira/browse/FLINK-12043
 Project: Flink
  Issue Type: Bug
  Components: API / Type Serialization System
Affects Versions: 1.7.2
Reporter: Quan Shi


Null pointer exception when get length of "_from"_ if _"from"_ is null in 
copy() method:

 
Involved classes:
{code:java}
// code placeholder
public String[] copy(String[] from) {
   String[] target = new String[from.length];
   System.arraycopy(from, 0, target, 0, from.length);
   return target;
}

{code}
Involved serializer classes in package 
"org.apache.flink.api.common.typeutils.base.array" 
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-37: Rework of the Table API Type System (Part 1)

[Kurt Young]

Big +1 to this! I left some comments in google doc.

Best,
Kurt


On Wed, Mar 27, 2019 at 11:32 PM Timo Walther  wrote:

> Hi everyone,
>
> some of you might have already read FLIP-32 [1] where we've described an
> approximate roadmap of how to handle the big Blink SQL contribution and
> how we can make the Table & SQL API equally important to the existing
> DataStream API.
>
> As mentioned there (Advance the API and Unblock New Features, Item 1),
> the rework of the Table/SQL type system is a crucial step for unblocking
> future contributions. In particular, Flink's current type system has
> many shortcomings which make an integration with other systems (such as
> Hive), DDL statements, and a unified API for Java/Scala difficult. We
> propose a new type system that is closer to the SQL standard, integrates
> better with other SQL vendors, and solves most of the type-related
> issues we had in the past.
>
> The design document for FLIP-37 can be found here:
>
>
> https://docs.google.com/document/d/1a9HUb6OaBIoj9IRfbILcMFPrOL7ALeZ3rVI66dvA2_U/edit?usp=sharing
>
> I'm looking forward to your feedback.
>
> Thanks,
> Timo
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
>
>

[jira] [Created] (FLINK-12044) Flink CEP discards events as late if they have timestamps <= 0

[Arseniy Tashoyan (JIRA)]

Arseniy Tashoyan created FLINK-12044:


 Summary: Flink CEP discards events as late if they have timestamps 
<= 0
 Key: FLINK-12044
 URL: https://issues.apache.org/jira/browse/FLINK-12044
 Project: Flink
  Issue Type: Bug
  Components: Library / CEP
Affects Versions: 1.7.2
Reporter: Arseniy Tashoyan


Let's make an input stream like this:

{code:scala}
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val events = env.fromElements(
  Event(timestamp = 0L, siteId),
  Event(timestamp = 500L, siteId)
)
.assignTimestampsAndWatermarks(new BoundedOutOfOrdernessTimestampExtractor{...})
{code}

For this stream the event with timestamp = 0L will be treated as late. The 
reason is AbstractKeyedCEPPatternOperator that initializes lastWatermark with 
the default value provided by the compiler: zero. Would it be more correct to 
initialize lastWatermark with Long.MIN_VALUE?

Of course, this is not a blocking defect, but I had really hard times to find 
why my unit test disregards the first event. Strictly speaking, this behavior 
seems incorrect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12045) Support force migration between incompatible state schemas/serializers

[Yu Li (JIRA)]

Yu Li created FLINK-12045:
-

 Summary: Support force migration between incompatible state 
schemas/serializers
 Key: FLINK-12045
 URL: https://issues.apache.org/jira/browse/FLINK-12045
 Project: Flink
  Issue Type: Improvement
  Components: API / Type Serialization System
Reporter: Yu Li
Assignee: Yu Li


Under certain circumstances user would choose to do force state schema 
migration, for example since we suggest to avoid using Kryo for state 
serialization, user might want to use POJO/Avro serializers instead and wish 
Flink automatically perform a force migration when required to.

This JIRA targets at supporting such force migration requirements.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-27: Refactor Source Interface

[Biao Liu]

Hi Steven,
Thank you for the feedback. Please take a look at the document FLIP-27

which
is updated recently. A lot of details of enumerator were added in this
document. I think it would help.

Steven Wu  于2019年3月28日周四 下午12:52写道：

> This proposal mentioned that SplitEnumerator might run on the JobManager or
> in a single task on a TaskManager.
>
> if enumerator is a single task on a taskmanager, then the job DAG can never
> been embarrassingly parallel anymore. That will nullify the leverage of
> fine-grained recovery for embarrassingly parallel jobs.
>
> It's not clear to me what's the implication of running enumerator on the
> jobmanager. So I will leave that out for now.
>
> On Mon, Jan 28, 2019 at 3:05 AM Biao Liu  wrote:
>
> > Hi Stephan & Piotrek,
> >
> > Thank you for feedback.
> >
> > It seems that there are a lot of things to do in community. I am just
> > afraid that this discussion may be forgotten since there so many
> proposals
> > recently.
> > Anyway, wish to see the split topics soon :)
> >
> > Piotr Nowojski  于2019年1月24日周四 下午8:21写道：
> >
> > > Hi Biao!
> > >
> > > This discussion was stalled because of preparations for the open
> sourcing
> > > & merging Blink. I think before creating the tickets we should split
> this
> > > discussion into topics/areas outlined by Stephan and create Flips for
> > that.
> > >
> > > I think there is no chance for this to be completed in couple of
> > remaining
> > > weeks/1 month before 1.8 feature freeze, however it would be good to
> aim
> > > with those changes for 1.9.
> > >
> > > Piotrek
> > >
> > > > On 20 Jan 2019, at 16:08, Biao Liu  wrote:
> > > >
> > > > Hi community,
> > > > The summary of Stephan makes a lot sense to me. It is much clearer
> > indeed
> > > > after splitting the complex topic into small ones.
> > > > I was wondering is there any detail plan for next step? If not, I
> would
> > > > like to push this thing forward by creating some JIRA issues.
> > > > Another question is that should version 1.8 include these features?
> > > >
> > > > Stephan Ewen  于2018年12月1日周六 上午4:20写道：
> > > >
> > > >> Thanks everyone for the lively discussion. Let me try to summarize
> > > where I
> > > >> see convergence in the discussion and open issues.
> > > >> I'll try to group this by design aspect of the source. Please let me
> > > know
> > > >> if I got things wrong or missed something crucial here.
> > > >>
> > > >> For issues 1-3, if the below reflects the state of the discussion, I
> > > would
> > > >> try and update the FLIP in the next days.
> > > >> For the remaining ones we need more discussion.
> > > >>
> > > >> I would suggest to fork each of these aspects into a separate mail
> > > thread,
> > > >> or will loose sight of the individual aspects.
> > > >>
> > > >> *(1) Separation of Split Enumerator and Split Reader*
> > > >>
> > > >>  - All seem to agree this is a good thing
> > > >>  - Split Enumerator could in the end live on JobManager (and assign
> > > splits
> > > >> via RPC) or in a task (and assign splits via data streams)
> > > >>  - this discussion is orthogonal and should come later, when the
> > > interface
> > > >> is agreed upon.
> > > >>
> > > >> *(2) Split Readers for one or more splits*
> > > >>
> > > >>  - Discussion seems to agree that we need to support one reader that
> > > >> possibly handles multiple splits concurrently.
> > > >>  - The requirement comes from sources where one poll()-style call
> > > fetches
> > > >> data from different splits / partitions
> > > >>--> example sources that require that would be for example Kafka,
> > > >> Pravega, Pulsar
> > > >>
> > > >>  - Could have one split reader per source, or multiple split readers
> > > that
> > > >> share the "poll()" function
> > > >>  - To not make it too complicated, we can start with thinking about
> > one
> > > >> split reader for all splits initially and see if that covers all
> > > >> requirements
> > > >>
> > > >> *(3) Threading model of the Split Reader*
> > > >>
> > > >>  - Most active part of the discussion ;-)
> > > >>
> > > >>  - A non-blocking way for Flink's task code to interact with the
> > source
> > > is
> > > >> needed in order to a task runtime code based on a
> > > >> single-threaded/actor-style task design
> > > >>--> I personally am a big proponent of that, it will help with
> > > >> well-behaved checkpoints, efficiency, and simpler yet more robust
> > > runtime
> > > >> code
> > > >>
> > > >>  - Users care about simple abstraction, so as a subclass of
> > SplitReader
> > > >> (non-blocking / async) we need to have a BlockingSplitReader which
> > will
> > > >> form the basis of most source implementations. BlockingSplitReader
> > lets
> > > >> users do blocking simple poll() calls.
> > > >>  - The BlockingSplitReader would spawn a thread (or more) and the
> > > >> thread(s) can make blocking calls and hand over data buffers via a
> > > blocking
> >

Re: [VOTE] Release 1.8.0, release candidate #4

[Tzu-Li (Gordon) Tai]

@Shaoxuan

The drop in the serializerAvro benchmark, as explained earlier in previous
voting threads of earlier RCs, was due to a slower job initialization phase
caused by slower deserialization of the AvroSerializer.
Piotr also pointed out that after the number of records was increased in
the serializer benchmarks, this drop was no longer observable before /
after the changes in mid February.
IMO, this is not critical as it does not affect the per-record performance
/ throughput, and therefore should not block this release.

On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek 
wrote:

> By now, I'm reasonably sure that the test instabilities on the end-to-end
> test are only instabilities. I pushed changes to increase timeouts to make
> the tests more stable. As in any project, there will always be bugs but I
> think we could release this RC4 and be reasonably sure that it works well.
>
> Now, we only need to have the required number of PMC votes.
>
> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
> > +1 （non-binding)
> >
> > • checked signature and checksum  ok
> > • mvn clean package -DskipTests ok
> > • Run job on yarn ok
> > • Test state migration with POJO type (both heap and rocksdb) ok
> > • - 1.6 -> 1.8
> > • - 1.7 -> 1.8
> > • - 1.8 -> 1.8
> >
> >
> > Best, Congxian
> > On Mar 27, 2019, 10:26 +0800, vino yang , wrote:
> > > +1 (non-binding)
> > >
> > > - checked JIRA release note
> > > - ran "mvn package -DskipTests"
> > > - checked signature and checksum
> > > - started a cluster locally and ran some examples in binary
> > > - checked web site announcement's PR
> > >
> > > Best,
> > > Vino
> > >
> > >
> > > Xiaowei Jiang  于2019年3月26日周二 下午8:20写道：
> > >
> > > > +1 (non-binding)
> > > >
> > > > - checked checksums and GPG files
> > > > - build from source successfully- run end-to-end precommit tests
> > > > successfully- run end-to-end nightly tests successfully
> > > > Xiaowei
> > > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
> car...@gmail.com>
> > > > wrote:
> > > >
> > > > +1 (non-binding)
> > > >
> > > > - Checked release notes: OK
> > > > - Checked sums and signatures: OK
> > > > - Source release
> > > > - contains no binaries: OK
> > > > - contains no 1.8-SNAPSHOT references: OK
> > > > - build from source: OK (8u101)
> > > > - mvn clean verify: OK (8u101)
> > > > - Binary release
> > > > - no examples appear to be missing
> > > > - started a cluster; WebUI reachable, example ran successfully
> > > > - end-to-end test (all but K8S and docker ones): OK (8u101)
> > > > - Repository appears to contain all expected artifacts
> > > >
> > > > Best Regards,
> > > > Yu
> > > >
> > > >
> > > > On Tue, 26 Mar 2019 at 14:28, Kurt Young  wrote:
> > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > Checked items:
> > > > > - checked checksums and GPG files
> > > > > - verified that the source archives do not contains any binaries
> > > > > - checked that all POM files point to the same version
> > > > > - build from source successfully
> > > > >
> > > > > Best,
> > > > > Kurt
> > > > >
> > > > >
> > > > > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
> wshaox...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > > I tested RC4 with the following items:
> > > > > > - Maven Central Repository contains all artifacts
> > > > > > - Built the source with Maven (ensured all source files have
> Apache
> > > > > > headers), and executed built-in tests via "mvn clean verify"
> > > > > > - Manually executed the tests in IntelliJ IDE
> > > > > > - Verify that the quickstarts for Scala and Java are working
> with the
> > > > > > staging repository in IntelliJ
> > > > > > - Checked the benchmark results. The perf regression of
> > > > > > tuple-key-by/statebackend/tumblingWindow are gone, but the
> regression
> > > > on
> > > > > > serializer still exists.
> > > > > >
> > > > > > Regards,
> > > > > > Shaoxuan
> > > > > >
> > > > > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
> sunjincheng...@gmail.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Aljoscha, I think you are right, increase the timeout
> config will
> > > > > fix
> > > > > > > this issue. this depends on the resource of Travis. I would
> like
> > > > share
> > > > > > > some phenomenon during my test (not the flink problem) as
> follows:
> > > > :-)
> > > > > > >
> > > > > > > During my testing, `mvn clean verify` and `nightly end-to-end
> test `
> > > > > both
> > > > > > > consume a lot of machine resources (especially
> memory/network), and
> > > > the
> > > > > > > network bandwidth requirements of `nightly end-to-end test `
> are also
> > > > > > very
> > > > > > > high. In China, need to use VPN acceleration (100~200Kb before
> > > > > > > acceleration, 3~4Mb after acceleration), I have encountered:
> [Avro
> > > > > > > Confluent Schema Registry nightly end-to-end test' failed
> after 18
> > > > > > minutes
> > > > > > > and 15 seconds! Test exited with exit Code 1] takes more

Re: [VOTE] Release 1.8.0, release candidate #4

[Tzu-Li (Gordon) Tai]

+1 (binding)

Functional checks:

- Built Flink from source (`mvn clean verify`) locally, with success
- Ran end-to-end tests locally for 5 times in a loop, no attempts failed
(Hadoop 2.8.4, Scala 2.12)
- Manually tested state schema evolution for POJO. Besides the tests that
@Congxian already did, additionally tested evolution cases with POJO
subclasses + non-registered POJOs.
- Manually tested migration of Scala stateful jobs that use case classes /
Scala collections as state types, performing the migration across Scala
2.11 to Scala 2.12.
- Reviewed release announcement PR

Misc / legal checks:

- checked checksums and signatures
- No binaries in source distribution
- Staging area does not seem to have any missing artifacts

Cheers,
Gordon

On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai 
wrote:

> @Shaoxuan
>
> The drop in the serializerAvro benchmark, as explained earlier in previous
> voting threads of earlier RCs, was due to a slower job initialization phase
> caused by slower deserialization of the AvroSerializer.
> Piotr also pointed out that after the number of records was increased in
> the serializer benchmarks, this drop was no longer observable before /
> after the changes in mid February.
> IMO, this is not critical as it does not affect the per-record performance
> / throughput, and therefore should not block this release.
>
> On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek 
> wrote:
>
>> By now, I'm reasonably sure that the test instabilities on the end-to-end
>> test are only instabilities. I pushed changes to increase timeouts to make
>> the tests more stable. As in any project, there will always be bugs but I
>> think we could release this RC4 and be reasonably sure that it works well.
>>
>> Now, we only need to have the required number of PMC votes.
>>
>> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
>> > +1 （non-binding)
>> >
>> > • checked signature and checksum  ok
>> > • mvn clean package -DskipTests ok
>> > • Run job on yarn ok
>> > • Test state migration with POJO type (both heap and rocksdb) ok
>> > • - 1.6 -> 1.8
>> > • - 1.7 -> 1.8
>> > • - 1.8 -> 1.8
>> >
>> >
>> > Best, Congxian
>> > On Mar 27, 2019, 10:26 +0800, vino yang , wrote:
>> > > +1 (non-binding)
>> > >
>> > > - checked JIRA release note
>> > > - ran "mvn package -DskipTests"
>> > > - checked signature and checksum
>> > > - started a cluster locally and ran some examples in binary
>> > > - checked web site announcement's PR
>> > >
>> > > Best,
>> > > Vino
>> > >
>> > >
>> > > Xiaowei Jiang  于2019年3月26日周二 下午8:20写道：
>> > >
>> > > > +1 (non-binding)
>> > > >
>> > > > - checked checksums and GPG files
>> > > > - build from source successfully- run end-to-end precommit tests
>> > > > successfully- run end-to-end nightly tests successfully
>> > > > Xiaowei
>> > > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
>> car...@gmail.com>
>> > > > wrote:
>> > > >
>> > > > +1 (non-binding)
>> > > >
>> > > > - Checked release notes: OK
>> > > > - Checked sums and signatures: OK
>> > > > - Source release
>> > > > - contains no binaries: OK
>> > > > - contains no 1.8-SNAPSHOT references: OK
>> > > > - build from source: OK (8u101)
>> > > > - mvn clean verify: OK (8u101)
>> > > > - Binary release
>> > > > - no examples appear to be missing
>> > > > - started a cluster; WebUI reachable, example ran successfully
>> > > > - end-to-end test (all but K8S and docker ones): OK (8u101)
>> > > > - Repository appears to contain all expected artifacts
>> > > >
>> > > > Best Regards,
>> > > > Yu
>> > > >
>> > > >
>> > > > On Tue, 26 Mar 2019 at 14:28, Kurt Young  wrote:
>> > > >
>> > > > > +1 (non-binding)
>> > > > >
>> > > > > Checked items:
>> > > > > - checked checksums and GPG files
>> > > > > - verified that the source archives do not contains any binaries
>> > > > > - checked that all POM files point to the same version
>> > > > > - build from source successfully
>> > > > >
>> > > > > Best,
>> > > > > Kurt
>> > > > >
>> > > > >
>> > > > > On Tue, Mar 26, 2019 at 10:57 AM Shaoxuan Wang <
>> wshaox...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > +1 (non-binding)
>> > > > > >
>> > > > > > I tested RC4 with the following items:
>> > > > > > - Maven Central Repository contains all artifacts
>> > > > > > - Built the source with Maven (ensured all source files have
>> Apache
>> > > > > > headers), and executed built-in tests via "mvn clean verify"
>> > > > > > - Manually executed the tests in IntelliJ IDE
>> > > > > > - Verify that the quickstarts for Scala and Java are working
>> with the
>> > > > > > staging repository in IntelliJ
>> > > > > > - Checked the benchmark results. The perf regression of
>> > > > > > tuple-key-by/statebackend/tumblingWindow are gone, but the
>> regression
>> > > > on
>> > > > > > serializer still exists.
>> > > > > >
>> > > > > > Regards,
>> > > > > > Shaoxuan
>> > > > > >
>> > > > > > On Tue, Mar 26, 2019 at 8:06 AM jincheng sun <
>> sunjincheng...@gmail.com
>> > > > >
>> > > > > > wrote:
>

Re: [DISCUSS] Python (and Non-JVM) Language Support in Flink

[Jeff Zhang]

Hi Shaoxuan & Jincheng,

Thanks for driving this initiative. Python would be a very big add-on for
flink adoption in data science world. One additional suggestion is you may
need to think about how to transfer flink Table to pandas dataframe which
is a very popular library in python. And you may be interested in apache
arrow which is a common layer to transferring data efficiently across
languages. https://arrow.apache.org/






vino yang  于2019年3月28日周四 下午2:44写道：

> Hi jincheng,
>
> Thanks for activating this discussion again.
> I personally look forward to your design draft.
>
> Best,
> Vino
>
> jincheng sun  于2019年3月28日周四 下午12:16写道：
>
> > Hi everyone,
> > Sorry to join in this discussion late.
> >
> > Thanks to Xianda Ke for initiating this discussion. I also enjoy the
> > discussions&suggestions by Max, Austin, Thomas, Shaoxuan and others.
> >
> > Recently, I did feel the desire of the community and Flink users for
> Python
> > support. Stephan also pointed out in the discussion of `Adding a mid-term
> > roadmap`: "Table API becomes primary API for analytics use cases", while
> a
> > large number of users in analytics use cases are accustomed to the Python
> > language, and the accumulation of a large number of class libraries is
> also
> > deposited in the python library.
> >
> > So I am very interested in participating in the discussion of supporting
> > Python in Flink. With regard to the three options mentioned so far, it
> is a
> > great encouragement to leverage the beam’s language portable layer on
> > Flink. For now, we can start with a step in the Flink to add a
> Py-tableAPI.
> > I believe in, in this process, we will have a deeper understanding of how
> > Flink support python. If we can quickly let users experience the first
> > version of Flink Python TableAPI, we can also receive feedback from many
> > users, and consider the long-term goals of multi-language support on
> Flink.
> >
> > So if you agree, I volunteer to draft a document that would support the
> > detailed design and implementation plan of Py-TableAPI on Flink.
> >
> > What do you think?
> >
> > Shaoxuan Wang  于2019年2月21日周四 下午10:44写道：
> >
> > > Hey guys,
> > > Thanks for your comments and sorry for the late reply.
> > > Beam Python API and Flink Python TableAPI describe the DAG/pipeline in
> > > different manners. We got a chance to communicate with Tyler Akidau
> (from
> > > Beam) offline, and explained why the Flink tableAPI needs a specific
> > design
> > > for python, rather than purely leverage Beam portability layer.
> > >
> > > In our proposal, most of the Python code is just a DAG/pipeline builder
> > for
> > > tableAPI. The majority of operators run purely in Java, while only
> python
> > > UDFs executed in Python environment during the runtime. This design
> does
> > > not affect the development and adoption of Beam language portability
> > layer
> > > with Flink runner. Flink and Beam community will still collaborate,
> > jointly
> > > develop and optimize on the JVM / Non-JVM (python,GO) bridge (data and
> > > control connections between different processes) to ensure the
> robustness
> > > and performance.
> > >
> > > Regards,
> > > Shaoxuan
> > >
> > >
> > > On Fri, Dec 21, 2018 at 1:39 PM Thomas Weise  wrote:
> > >
> > > > Interest in Python seems on the rise and so this is a good discussion
> > to
> > > > have :)
> > > >
> > > > So far there seems to be agreement that Beam's approach towards
> Python
> > > and
> > > > other non-JVM language support (language SDK, portability layer etc.)
> > is
> > > > the right direction? Specification and execution are native Python
> and
> > it
> > > > does not suffer from the shortcomings of Flink's Jython API and few
> > other
> > > > approaches.
> > > >
> > > > Overall there already is good alignment between Beam and Flink in
> > > concepts
> > > > and model. There are also few of us that are active in both
> > communities.
> > > > The Beam Flink runner has made a lot of progress this year, but work
> on
> > > > portability in Beam actually started much before that and was a very
> > big
> > > > change (originally there was just the Java SDK). Much of the code has
> > > been
> > > > rewritten as part of the effort; that's what implementing a strong
> > multi
> > > > language support story took. To have a decent shot at it, the
> > equivalent
> > > of
> > > > much of the Beam portability framework would need to be reinvented in
> > > > Flink. This would fork resources and divert focus away from things
> that
> > > may
> > > > be more core to Flink. As you can guess I am in favor of option (1) !
> > > >
> > > > We could take a look at SQL for reference. Flink community has
> > invested a
> > > > lot in SQL and there remains a lot of work to do. Beam community has
> > done
> > > > the same and we have two completely separate implementations. When I
> > > > recently learned more about the Beam SQL work, one of my first
> > questions
> > > > was if joined effort would not lead

[jira] [Created] (FLINK-12046) Increase records per invocation for serializerAvro benchmark to stabilize it

[Yu Li (JIRA)]

Yu Li created FLINK-12046:
-

 Summary: Increase records per invocation for serializerAvro 
benchmark to stabilize it
 Key: FLINK-12046
 URL: https://issues.apache.org/jira/browse/FLINK-12046
 Project: Flink
  Issue Type: Improvement
  Components: Tests
Reporter: Yu Li
Assignee: Yu Li


We have observed waved result of this benchmark and it turns out to be caused 
by testing data is too small. To stabilize the case, we need to increase the 
records per invocation for the benchmark. More details please see this [email 
thread|https://s.apache.org/E1i7]

To be specified, this change is for the 
[flink-benchmark|https://github.com/dataArtisans/flink-benchmarks] project.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12047) Savepoint connector to read / write / process savepoints

[Tzu-Li (Gordon) Tai (JIRA)]

Tzu-Li (Gordon) Tai created FLINK-12047:
---

 Summary: Savepoint connector to read / write / process savepoints
 Key: FLINK-12047
 URL: https://issues.apache.org/jira/browse/FLINK-12047
 Project: Flink
  Issue Type: New Feature
  Components: Runtime / State Backends
Reporter: Tzu-Li (Gordon) Tai


This JIRA tracks the ongoing efforts and discussions about a means to read / 
write / process state in savepoints.

There are already two known existing works (that was mentioned already in the 
mailing lists) related to this:
1. Bravo [1]
2. https://github.com/sjwiesman/flink/tree/savepoint-connector

Essentially, the two tools both provide a connector to read or write a Flink 
savepoint, and allows to utilize Flink's processing APIs for querying / 
processing the state in the savepoint.

We should try to converge the efforts on this, and have a savepoint connector 
like this in Flink.
With this connector, the high-level benefits users should be able to achieve 
with it are:
1. Create savepoints using existing data from other systems (i.e. bootstrapping 
a Flink job's state with data in an external database).
2. Derive new state using existing state
3. Query state in savepoints, for example for debugging purposes
4. Migrate schema of state in savepoints offline, compared to the current more 
limited approach of online migration on state access.
5. Change max parallelism of jobs.

[1] https://github.com/king/bravo




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12048) ZooKeeperHADispatcherTest failed on Travis

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-12048:


 Summary: ZooKeeperHADispatcherTest failed on Travis
 Key: FLINK-12048
 URL: https://issues.apache.org/jira/browse/FLINK-12048
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination, Tests
Affects Versions: 1.9.0
Reporter: Chesnay Schepler


https://travis-ci.org/apache/flink/builds/512077301

{code}
01:14:56.351 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 9.671 s <<< FAILURE! - in 
org.apache.flink.runtime.dispatcher.ZooKeeperHADispatcherTest
01:14:56.364 [ERROR] 
testStandbyDispatcherJobExecution(org.apache.flink.runtime.dispatcher.ZooKeeperHADispatcherTest)
  Time elapsed: 1.209 s  <<< ERROR!
org.apache.flink.runtime.util.TestingFatalErrorHandler$TestingException: 
org.apache.flink.runtime.dispatcher.DispatcherException: Could not start the 
added job d51eeb908f360e44c0a2004e00a6afd2
at 
org.apache.flink.runtime.dispatcher.ZooKeeperHADispatcherTest.teardown(ZooKeeperHADispatcherTest.java:117)
Caused by: org.apache.flink.runtime.dispatcher.DispatcherException: Could not 
start the added job d51eeb908f360e44c0a2004e00a6afd2
Caused by: java.lang.IllegalStateException: Not running. Forgot to call start()?
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12049) ClassLoaderUtilsTest fails on Java 9

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-12049:


 Summary: ClassLoaderUtilsTest fails on Java 9
 Key: FLINK-12049
 URL: https://issues.apache.org/jira/browse/FLINK-12049
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Affects Versions: 1.9.0
Reporter: Chesnay Schepler


{code}
public void testWithAppClassLoader() {
String result = 
ClassLoaderUtil.getUserCodeClassLoaderInfo(ClassLoader.getSystemClassLoader());
assertTrue(result.toLowerCase().contains("system classloader"));
{code}

{{ClassLoader.getSystemClassLoader()}} no longer returns an URLClassLoader on 
Java 9, but {{ClassLoaderUtil.getUserCodeClassLoaderInfo}} relies on this to 
extract information about the ClassLoader.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12050) BlockingShutdownTest fails on Java 9

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-12050:


 Summary: BlockingShutdownTest fails on Java 9
 Key: FLINK-12050
 URL: https://issues.apache.org/jira/browse/FLINK-12050
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.9.0
Reporter: Chesnay Schepler


{code}
21:21:28.689 [ERROR] 
testProcessShutdownBlocking(org.apache.flink.runtime.util.BlockingShutdownTest) 
 Time elapsed: 0.961 s  <<< FAILURE!
java.lang.AssertionError: Cannot determine process ID
at 
org.apache.flink.runtime.util.BlockingShutdownTest.testProcessShutdownBlocking(BlockingShutdownTest.java:57)

21:21:28.689 [ERROR] 
testProcessExitsDespiteBlockingShutdownHook(org.apache.flink.runtime.util.BlockingShutdownTest)
  Time elapsed: 0.325 s  <<< FAILURE!
java.lang.AssertionError: Cannot determine process ID
at 
org.apache.flink.runtime.util.BlockingShutdownTest.testProcessExitsDespiteBlockingShutdownHook(BlockingShutdownTest.java:98)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-37: Rework of the Table API Type System (Part 1)

[Dawid Wysakowicz]

Another big +1 from my side. Thank you Timo for preparing the document!

I really look forward for this to have a standardized way of type
handling. This should solve loads of problems. I really like the
separation of logical type from its physical representation, I think we
should aim to introduce that and keep it separated.

Best,

Dawid

On 28/03/2019 08:51, Kurt Young wrote:
> Big +1 to this! I left some comments in google doc.
>
> Best,
> Kurt
>
>
> On Wed, Mar 27, 2019 at 11:32 PM Timo Walther  wrote:
>
>> Hi everyone,
>>
>> some of you might have already read FLIP-32 [1] where we've described an
>> approximate roadmap of how to handle the big Blink SQL contribution and
>> how we can make the Table & SQL API equally important to the existing
>> DataStream API.
>>
>> As mentioned there (Advance the API and Unblock New Features, Item 1),
>> the rework of the Table/SQL type system is a crucial step for unblocking
>> future contributions. In particular, Flink's current type system has
>> many shortcomings which make an integration with other systems (such as
>> Hive), DDL statements, and a unified API for Java/Scala difficult. We
>> propose a new type system that is closer to the SQL standard, integrates
>> better with other SQL vendors, and solves most of the type-related
>> issues we had in the past.
>>
>> The design document for FLIP-37 can be found here:
>>
>>
>> https://docs.google.com/document/d/1a9HUb6OaBIoj9IRfbILcMFPrOL7ALeZ3rVI66dvA2_U/edit?usp=sharing
>>
>> I'm looking forward to your feedback.
>>
>> Thanks,
>> Timo
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
>>
>>



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] FLIP-37: Rework of the Table API Type System (Part 1)

[Timo Walther]

Maybe to give some background about Dawid's latest email:

Kurt raised some good points regarding the conversion of data types at 
the boundaries of the API and SPI. After that, Dawid and I had a long 
discussion of how users can define those boundaries in a nicer way. The 
outcome of this discussion was similar to Blink's current distinction 
between InternalTypes and ExternalTypes. I updated the document with a 
improved structure of DataTypes (for users, API, SPI with conversion 
information) and LogicalTypes (used internally and close to standard SQL 
types).


Thanks for the feedback so far,
Timo

Am 28.03.19 um 11:18 schrieb Dawid Wysakowicz:

Another big +1 from my side. Thank you Timo for preparing the document!

I really look forward for this to have a standardized way of type
handling. This should solve loads of problems. I really like the
separation of logical type from its physical representation, I think we
should aim to introduce that and keep it separated.

Best,

Dawid

On 28/03/2019 08:51, Kurt Young wrote:

Big +1 to this! I left some comments in google doc.

Best,
Kurt


On Wed, Mar 27, 2019 at 11:32 PM Timo Walther  wrote:


Hi everyone,

some of you might have already read FLIP-32 [1] where we've described an
approximate roadmap of how to handle the big Blink SQL contribution and
how we can make the Table & SQL API equally important to the existing
DataStream API.

As mentioned there (Advance the API and Unblock New Features, Item 1),
the rework of the Table/SQL type system is a crucial step for unblocking
future contributions. In particular, Flink's current type system has
many shortcomings which make an integration with other systems (such as
Hive), DDL statements, and a unified API for Java/Scala difficult. We
propose a new type system that is closer to the SQL standard, integrates
better with other SQL vendors, and solves most of the type-related
issues we had in the past.

The design document for FLIP-37 can be found here:


https://docs.google.com/document/d/1a9HUb6OaBIoj9IRfbILcMFPrOL7ALeZ3rVI66dvA2_U/edit?usp=sharing

I'm looking forward to your feedback.

Thanks,
Timo

[1]

https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions

[jira] [Created] (FLINK-12051) TaskExecutorTest.testFilterOutDuplicateJobMasterRegistrations() failed locally.

[Kostas Kloudas (JIRA)]

Kostas Kloudas created FLINK-12051:
--

 Summary: 
TaskExecutorTest.testFilterOutDuplicateJobMasterRegistrations() failed locally.
 Key: FLINK-12051
 URL: https://issues.apache.org/jira/browse/FLINK-12051
 Project: Flink
  Issue Type: Bug
  Components: Tests
Affects Versions: 1.8.0
Reporter: Kostas Kloudas


The test failed locally with:

{code}

Wanted but not invoked:
 jobLeaderService.start(
 ,
 ,
 ,
 
 );
 -> at 
org.apache.flink.runtime.taskexecutor.TaskExecutorTest.testFilterOutDuplicateJobMasterRegistrations(TaskExecutorTest.java:1171)
 Actually, there were zero interactions with this mock.

Wanted but not invoked:
 jobLeaderService.start(
 ,
 ,
 ,
 
 );
 -> at 
org.apache.flink.runtime.taskexecutor.TaskExecutorTest.testFilterOutDuplicateJobMasterRegistrations(TaskExecutorTest.java:1171)
 Actually, there were zero interactions with this mock.

at 
org.apache.flink.runtime.taskexecutor.TaskExecutorTest.testFilterOutDuplicateJobMasterRegistrations(TaskExecutorTest.java:1171)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:498)
 at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
 at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
 at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
 at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
 at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
 at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
 at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
 at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
 at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
 at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.junit.runners.Suite.runChild(Suite.java:128)
 at org.junit.runners.Suite.runChild(Suite.java:27)
 at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
 at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
 at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
 at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
 at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
 at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
 at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
 at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
 at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
 at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
 at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12052) Redistributed unevenly sized result partitions

[ryantaocer (JIRA)]

ryantaocer created FLINK-12052:
--

 Summary: Redistributed unevenly sized result partitions
 Key: FLINK-12052
 URL: https://issues.apache.org/jira/browse/FLINK-12052
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: ryantaocer
Assignee: BoWang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12053) Calculation of parallelism based on metrics/signals

[ryantaocer (JIRA)]

ryantaocer created FLINK-12053:
--

 Summary: Calculation of parallelism based on metrics/signals
 Key: FLINK-12053
 URL: https://issues.apache.org/jira/browse/FLINK-12053
 Project: Flink
  Issue Type: Sub-task
Reporter: ryantaocer
Assignee: BoWang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12054) HBaseConnectorITCase fails on Java 9

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-12054:


 Summary: HBaseConnectorITCase fails on Java 9
 Key: FLINK-12054
 URL: https://issues.apache.org/jira/browse/FLINK-12054
 Project: Flink
  Issue Type: Sub-task
Reporter: Chesnay Schepler


An issue in hbase.

{code}
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 21.83 sec <<< 
FAILURE! - in org.apache.flink.addons.hbase.HBaseConnectorITCase
org.apache.flink.addons.hbase.HBaseConnectorITCase  Time elapsed: 21.829 sec  
<<< FAILURE!
java.lang.AssertionError: We should get a URLClassLoader
at 
org.apache.flink.addons.hbase.HBaseConnectorITCase.activateHBaseCluster(HBaseConnectorITCase.java:81)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12055) Remove legacy class ActorSystemLoader

[TisonKun (JIRA)]

TisonKun created FLINK-12055:


 Summary: Remove legacy class ActorSystemLoader
 Key: FLINK-12055
 URL: https://issues.apache.org/jira/browse/FLINK-12055
 Project: Flink
  Issue Type: Sub-task
  Components: Command Line Client
Affects Versions: 1.9.0
Reporter: TisonKun
Assignee: TisonKun
 Fix For: 1.9.0


Also its implementation {{DefaultActorSystemLoader}}.

Also remove unused test dependency of {{akka_testkit}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12056) Remove legacy methods in LeaderRetrievalUtils

[TisonKun (JIRA)]

TisonKun created FLINK-12056:


 Summary: Remove legacy methods in LeaderRetrievalUtils
 Key: FLINK-12056
 URL: https://issues.apache.org/jira/browse/FLINK-12056
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: TisonKun
Assignee: TisonKun
 Fix For: 1.9.0


Also some refactor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12057) Refactor MemoryLogger to accept termination future instead of ActorSystem

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-12057:


 Summary: Refactor MemoryLogger to accept termination future 
instead of ActorSystem
 Key: FLINK-12057
 URL: https://issues.apache.org/jira/browse/FLINK-12057
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.9.0


The {{MemoryLogger}} currently accepts an {{ActorSystem}} and uses it to 
determine whether the logging thread should be shut down.

For this purpose a termination future would be sufficient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12058) Cancel checkpoint operations belonging to a discarded/aborted checkpoint

[Till Rohrmann (JIRA)]

Till Rohrmann created FLINK-12058:
-

 Summary: Cancel checkpoint operations belonging to a 
discarded/aborted checkpoint
 Key: FLINK-12058
 URL: https://issues.apache.org/jira/browse/FLINK-12058
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Checkpointing
Affects Versions: 1.7.2, 1.8.0
Reporter: Till Rohrmann


In order to save CPU cycles and reduce disk and network I/O, we should try to 
cancel local checkpoint operations belonging to discarded aborted or subsumed 
checkpoints. For example, if a {{Task}} declines a checkpoint, the 
{{CheckpointCoordinator}} will discard this checkpoint. However, other 
checkpointing operations belonging to this checkpoint won't be necessarily 
notified and canceled.

The notification mechanism could piggy back on the existing 
{{CancelCheckpointMarker}} or be a separate signal sent to all participating 
{{Tasks}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12059) How to increase outputQueueLength/inputQueueLength

[WuXiZao (JIRA)]

WuXiZao created FLINK-12059:
---

 Summary: How to increase outputQueueLength/inputQueueLength
 Key: FLINK-12059
 URL: https://issues.apache.org/jira/browse/FLINK-12059
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Affects Versions: 1.6.3
Reporter: WuXiZao
 Attachments: 11.png

My job outputQueueLength too high always keep 48, how increase this buffer in 
order to decrease this value?




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12060) Unify change pom version scripts

[Ufuk Celebi (JIRA)]

Ufuk Celebi created FLINK-12060:
---

 Summary: Unify change pom version scripts
 Key: FLINK-12060
 URL: https://issues.apache.org/jira/browse/FLINK-12060
 Project: Flink
  Issue Type: Bug
  Components: Deployment / Scripts
Affects Versions: 1.8.0
Reporter: Ufuk Celebi


We have three places in `tools` that we use to update the pom versions when 
releasing:
1. https://github.com/apache/flink/blob/048367b/tools/change-version.sh#L31
2. 
https://github.com/apache/flink/blob/048367b/tools/releasing/create_release_branch.sh#L60
3. 
https://github.com/apache/flink/blob/048367b/tools/releasing/update_branch_version.sh#L52

The 1st option is buggy (it does not work with the new versioning of the shaded 
Hadoop build, e.g. {{2.4.1-1.9-SNAPSHOT}} will not be replaced). The 2nd and 
3rd work for pom files, but the 2nd one misses a change for the doc version 
that is present in the 3rd one.

I think we should unify these and call them where needed instead of duplicating 
this code in unexpected ways.

An initial quick fix could remove the 1st script and update the 2rd one to 
match the 3rd one.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] FLIP-37: Rework of the Table API Type System (Part 1)

[Rong Rong]

Thanks @Timo for starting this effort and preparing the document :-)

I took a pass and left some comments. I also very much like the idea of the
DataType and LogicalType separation.
As explained in the doc, we've also been looking into ways to improve the
type system so a huge +1 on our side.

One question I have is, since this touches many of the external systems
like Hive / Blink comparison, does it make sense to share this to a border
audience (such as user@) later to gather more feedbacks?

Looking forward to this change and would love to contribute in anyway I can!

Best,
Rong








On Thu, Mar 28, 2019 at 3:25 AM Timo Walther  wrote:

> Maybe to give some background about Dawid's latest email:
>
> Kurt raised some good points regarding the conversion of data types at
> the boundaries of the API and SPI. After that, Dawid and I had a long
> discussion of how users can define those boundaries in a nicer way. The
> outcome of this discussion was similar to Blink's current distinction
> between InternalTypes and ExternalTypes. I updated the document with a
> improved structure of DataTypes (for users, API, SPI with conversion
> information) and LogicalTypes (used internally and close to standard SQL
> types).
>
> Thanks for the feedback so far,
> Timo
>
> Am 28.03.19 um 11:18 schrieb Dawid Wysakowicz:
> > Another big +1 from my side. Thank you Timo for preparing the document!
> >
> > I really look forward for this to have a standardized way of type
> > handling. This should solve loads of problems. I really like the
> > separation of logical type from its physical representation, I think we
> > should aim to introduce that and keep it separated.
> >
> > Best,
> >
> > Dawid
> >
> > On 28/03/2019 08:51, Kurt Young wrote:
> >> Big +1 to this! I left some comments in google doc.
> >>
> >> Best,
> >> Kurt
> >>
> >>
> >> On Wed, Mar 27, 2019 at 11:32 PM Timo Walther 
> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> some of you might have already read FLIP-32 [1] where we've described
> an
> >>> approximate roadmap of how to handle the big Blink SQL contribution and
> >>> how we can make the Table & SQL API equally important to the existing
> >>> DataStream API.
> >>>
> >>> As mentioned there (Advance the API and Unblock New Features, Item 1),
> >>> the rework of the Table/SQL type system is a crucial step for
> unblocking
> >>> future contributions. In particular, Flink's current type system has
> >>> many shortcomings which make an integration with other systems (such as
> >>> Hive), DDL statements, and a unified API for Java/Scala difficult. We
> >>> propose a new type system that is closer to the SQL standard,
> integrates
> >>> better with other SQL vendors, and solves most of the type-related
> >>> issues we had in the past.
> >>>
> >>> The design document for FLIP-37 can be found here:
> >>>
> >>>
> >>>
> https://docs.google.com/document/d/1a9HUb6OaBIoj9IRfbILcMFPrOL7ALeZ3rVI66dvA2_U/edit?usp=sharing
> >>>
> >>> I'm looking forward to your feedback.
> >>>
> >>> Thanks,
> >>> Timo
> >>>
> >>> [1]
> >>>
> >>>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions
> >>>
> >>>
>
>

Bump up the shaded dependency version

[mayo zhang]

Hi, all 

Is there some way to bump up the version of shaded jars like 
flink-shaded-jackson?  Or is there someone can deal with this




Best,
Zhang

Re: [VOTE] Release 1.8.0, release candidate #4

[Richard Deurwaarder]

-1 (non-binding)

- Ran integration tests locally (1000+) of our flink job, all succeeded.
- Attempted to run job on hadoop, failed. It failed because we have a
firewall in place and we cannot set the rest port to a specific port/port
range.
Unless I am mistaken, it seems like FLINK-11081 broke the possibility of
setting a REST port when running on yarn (
https://github.com/apache/flink/commit/730eed71ef3f718d61f85d5e94b1060844ca56db#diff-487838863ab693af7008f04cb3359be3R102
)
Code-wise it seems rather straightforward to fix but I am unsure about the
reason why this is hard-coded to 0 and what the impact would be.

It would benefit us greatly if a fix for this could make it to 1.8.0.

Regards,

Richard

On Thu, Mar 28, 2019 at 9:54 AM Tzu-Li (Gordon) Tai 
wrote:

> +1 (binding)
>
> Functional checks:
>
> - Built Flink from source (`mvn clean verify`) locally, with success
> - Ran end-to-end tests locally for 5 times in a loop, no attempts failed
> (Hadoop 2.8.4, Scala 2.12)
> - Manually tested state schema evolution for POJO. Besides the tests that
> @Congxian already did, additionally tested evolution cases with POJO
> subclasses + non-registered POJOs.
> - Manually tested migration of Scala stateful jobs that use case classes /
> Scala collections as state types, performing the migration across Scala
> 2.11 to Scala 2.12.
> - Reviewed release announcement PR
>
> Misc / legal checks:
>
> - checked checksums and signatures
> - No binaries in source distribution
> - Staging area does not seem to have any missing artifacts
>
> Cheers,
> Gordon
>
> On Thu, Mar 28, 2019 at 4:52 PM Tzu-Li (Gordon) Tai 
> wrote:
>
> > @Shaoxuan
> >
> > The drop in the serializerAvro benchmark, as explained earlier in
> previous
> > voting threads of earlier RCs, was due to a slower job initialization
> phase
> > caused by slower deserialization of the AvroSerializer.
> > Piotr also pointed out that after the number of records was increased in
> > the serializer benchmarks, this drop was no longer observable before /
> > after the changes in mid February.
> > IMO, this is not critical as it does not affect the per-record
> performance
> > / throughput, and therefore should not block this release.
> >
> > On Thu, Mar 28, 2019 at 1:08 AM Aljoscha Krettek 
> > wrote:
> >
> >> By now, I'm reasonably sure that the test instabilities on the
> end-to-end
> >> test are only instabilities. I pushed changes to increase timeouts to
> make
> >> the tests more stable. As in any project, there will always be bugs but
> I
> >> think we could release this RC4 and be reasonably sure that it works
> well.
> >>
> >> Now, we only need to have the required number of PMC votes.
> >>
> >> On Wed, Mar 27, 2019, at 07:22, Congxian Qiu wrote:
> >> > +1 （non-binding)
> >> >
> >> > • checked signature and checksum  ok
> >> > • mvn clean package -DskipTests ok
> >> > • Run job on yarn ok
> >> > • Test state migration with POJO type (both heap and rocksdb) ok
> >> > • - 1.6 -> 1.8
> >> > • - 1.7 -> 1.8
> >> > • - 1.8 -> 1.8
> >> >
> >> >
> >> > Best, Congxian
> >> > On Mar 27, 2019, 10:26 +0800, vino yang ,
> wrote:
> >> > > +1 (non-binding)
> >> > >
> >> > > - checked JIRA release note
> >> > > - ran "mvn package -DskipTests"
> >> > > - checked signature and checksum
> >> > > - started a cluster locally and ran some examples in binary
> >> > > - checked web site announcement's PR
> >> > >
> >> > > Best,
> >> > > Vino
> >> > >
> >> > >
> >> > > Xiaowei Jiang  于2019年3月26日周二 下午8:20写道：
> >> > >
> >> > > > +1 (non-binding)
> >> > > >
> >> > > > - checked checksums and GPG files
> >> > > > - build from source successfully- run end-to-end precommit tests
> >> > > > successfully- run end-to-end nightly tests successfully
> >> > > > Xiaowei
> >> > > > On Tuesday, March 26, 2019, 8:09:19 PM GMT+8, Yu Li <
> >> car...@gmail.com>
> >> > > > wrote:
> >> > > >
> >> > > > +1 (non-binding)
> >> > > >
> >> > > > - Checked release notes: OK
> >> > > > - Checked sums and signatures: OK
> >> > > > - Source release
> >> > > > - contains no binaries: OK
> >> > > > - contains no 1.8-SNAPSHOT references: OK
> >> > > > - build from source: OK (8u101)
> >> > > > - mvn clean verify: OK (8u101)
> >> > > > - Binary release
> >> > > > - no examples appear to be missing
> >> > > > - started a cluster; WebUI reachable, example ran successfully
> >> > > > - end-to-end test (all but K8S and docker ones): OK (8u101)
> >> > > > - Repository appears to contain all expected artifacts
> >> > > >
> >> > > > Best Regards,
> >> > > > Yu
> >> > > >
> >> > > >
> >> > > > On Tue, 26 Mar 2019 at 14:28, Kurt Young 
> wrote:
> >> > > >
> >> > > > > +1 (non-binding)
> >> > > > >
> >> > > > > Checked items:
> >> > > > > - checked checksums and GPG files
> >> > > > > - verified that the source archives do not contains any binaries
> >> > > > > - checked that all POM files point to the same version
> >> > > > > - build from source successfully
> >> > > > >
> >> > > > > Best,
> >> > > > > Kurt
> >> > > > >
> >

[jira] [Created] (FLINK-12061) Add more window operator contract tests to table runtime

[Kurt Young (JIRA)]

Kurt Young created FLINK-12061:
--

 Summary: Add more window operator contract tests to table runtime
 Key: FLINK-12061
 URL: https://issues.apache.org/jira/browse/FLINK-12061
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Reporter: Kurt Young


[FLINK-11959|https://issues.apache.org/jira/browse/FLINK-11959] introduced a 
window operator for table runtime. But the tests are not enough comparing to 
DataStream's window operator. We can borrow more tests from it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12062) Introduce bundle operator to streaming table runtime

[Kurt Young (JIRA)]

Kurt Young created FLINK-12062:
--

 Summary: Introduce bundle operator to streaming table runtime
 Key: FLINK-12062
 URL: https://issues.apache.org/jira/browse/FLINK-12062
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Reporter: Kurt Young


Bundle operator will try to save incoming records in a key-value map. Once 
bundler triggers, the bundle function will be invoked. All buffered data will 
be passed in, and one can do some optimizations based on this. 

One useful scenario for bundle operator is "Group Aggregate". We can organize 
the bundle data with grouping key. Once bundle triggers, we can first pre 
aggregate all data belongs to same key in memory, then we only have to operate 
with state once for each key. This will save lots of cost and have better 
performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)