[jira] [Created] (FLINK-13072) RocksDBStateBachend is not thread safe and data loss silently

[lynn1.zhang (JIRA)]

lynn1.zhang created FLINK-13072:
---

 Summary: RocksDBStateBachend is not thread safe and data loss 
silently
 Key: FLINK-13072
 URL: https://issues.apache.org/jira/browse/FLINK-13072
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Affects Versions: 1.8.1, 1.8.0
Reporter: lynn1.zhang
 Attachments: flink-demo.zip

I create 2 mapstates in one operator, then create 2 threads in apply method, 
each thread operate each map state(the operator is same), the expect result is 
that 2 state have the same result but not. I upload the code, please help to 
try it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13071) QueryOperationConverter in Blink planner support add kinds of QueryOperations.

[Jing Zhang (JIRA)]

Jing Zhang created FLINK-13071:
--

 Summary: QueryOperationConverter in Blink planner support add 
kinds of QueryOperations.
 Key: FLINK-13071
 URL: https://issues.apache.org/jira/browse/FLINK-13071
 Project: Flink
  Issue Type: Task
  Components: Table SQL / API
Reporter: Jing Zhang
Assignee: Jing Zhang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Bowen Li]

I responded in the INFRA ticket [1] that I believe they are using a wrong
metric against Flink and the total build time is a completely different
thing than guaranteed build capacity.

My response:

"As mentioned above, since I started to pay attention to Flink's build
queue a few tens of days ago, I'm in Seattle and I saw no build was kicking
off in PST daytime in weekdays for Flink. Our teammates in China and Europe
have also reported similar observations. So we need to evaluate how the
large total build time came from - if 1) your number and 2) our
observations from three locations that cover pretty much a full day, are
all true, I **guess** one reason can be that - highly likely the extra
build time came from weekends when other Apache projects may be idle and
Flink just drains hard its congested queue.

Please be aware of that we're not complaining about the lack of resources
in general, I'm complaining about the lack of **stable, dedicated**
resources. An example for the latter one is, currently even if no build is
in Flink's queue and I submit a request to be the queue head in PST
morning, my build won't even start in 6-8+h. That is an absurd amount of
waiting time.

That's saying, if ASF INFRA decides to adopt a quota system and grants
Flink five DEDICATED servers that runs all the time only for Flink, that'll
be PERFECT and can totally solve our problem now.

Please be aware of that we're not complaining about the lack of resources
in general, I'm complaining about the lack of **stable, dedicated**
resources. An example for the latter one is, currently even if no build is
in Flink's queue and I submit a request to be the queue head in PST
morning, my build won't even start in 6-8+h. That is an absurd amount of
waiting time.


That's saying, if ASF INFRA decides to adopt a quota system and grants
Flink five DEDICATED servers that runs all the time only for Flink, that'll
be PERFECT and can totally solve our problem now.

I feel what's missing in the ASF INFRA's Travis resource pool is some level
of build capacity SLAs and certainty"


Again, I believe there are differences in nature of these two problems,
long build time v.s. lack of dedicated build resource. That's saying,
shortening build time may relieve the situation, and may not. I'm sightly
negative on disabling IT cases for PRs, due to the downside is that we are
at risk of any potential bugs in PR that UTs doesn't catch, and may cost a
lot more to fix and if it slows others down or even block others, but am
open to others opinions on it.

AFAICT from INFRA ticket[1], donating to ASF INFRA won't be feasible to
solve our problem since INFRA's pool is fully shared and they have no
control and finer insights over resource allocation to a specific Apache
project. As mentioned in [1], Apache Arrow is moving away from ASF INFRA
Travis pool (they are actually surprised Flink hasn't plan to do so). I
know that Spark is on its own build infra. If we all agree that funding our
own build infra, I'd be glad to help investigate any potential options
after releasing 1.9 since I'm super busy with 1.9 now.

[1] https://issues.apache.org/jira/browse/INFRA-18533



On Tue, Jul 2, 2019 at 4:46 AM Chesnay Schepler  wrote:

> As a short-term stopgap, since we can assume this issue to become much
> worse in the following days/weeks, we could disable IT cases in PRs and
> only run them on master.
>
> On 02/07/2019 12:03, Chesnay Schepler wrote:
> > People really have to stop thinking that just because something works
> > for us it is also a good solution.
> > Also, please remember that our builds run for 2h from start to finish,
> > and not the 14 _minutes_ it takes for zeppelin.
> > We are dealing with an entirely different scale here, both in terms of
> > build times and number of builds.
> >
> > In this very thread people have been complaining about long queue
> > times for their builds. Surprise, other Apache projects have been
> > suffering the very same thing due to us not controlling our build
> > times. While switching services (be it Jenkins, CircleCI or whatever)
> > will possibly work for us (and these options are actually attractive,
> > like CircleCI's proper support for build artifacts), it will also
> > result in us likely negatively affecting other projects in significant
> > ways.
> >
> > Sure, the Jenkins setup has a good user experience for us, at the cost
> > of blocking Jenkins workers for a _lot_ of time. Right now we have 25
> > PR's in our queue; that's possibly 50h we'd consume of Jenkins
> > resources, and the European contributors haven't even really started yet.
> >
> > FYI, the latest INFRA response from INFRA-18533:
> >
> > "Our rough metrics shows that Flink used over 5800 hours of build time
> > last month. That is equal to EIGHT servers running 24/7 for the ENTIRE
> > MONTH. EIGHT. nonstop.
> > When we discovered this last night, we discussed it some and are going
> > to tune down Flink to allow only five executors maximum. We 

[jira] [Created] (FLINK-13070) Remove TableImpl and use api.internal.TableImpl in blink

[Jingsong Lee (JIRA)]

Jingsong Lee created FLINK-13070:


 Summary: Remove TableImpl and use api.internal.TableImpl in blink
 Key: FLINK-13070
 URL: https://issues.apache.org/jira/browse/FLINK-13070
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Planner
Reporter: Jingsong Lee
Assignee: Jingsong Lee






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13069) HiveTableSink should implement OverwritableTableSink

[Rui Li (JIRA)]

Rui Li created FLINK-13069:
--

 Summary: HiveTableSink should implement OverwritableTableSink
 Key: FLINK-13069
 URL: https://issues.apache.org/jira/browse/FLINK-13069
 Project: Flink
  Issue Type: Sub-task
Reporter: Rui Li
Assignee: Rui Li






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13068) HiveTableSink should implement PartitionableTableSink

[Rui Li (JIRA)]

Rui Li created FLINK-13068:
--

 Summary: HiveTableSink should implement PartitionableTableSink
 Key: FLINK-13068
 URL: https://issues.apache.org/jira/browse/FLINK-13068
 Project: Flink
  Issue Type: Sub-task
Reporter: Rui Li
Assignee: Rui Li






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Vectorization Support in Flink

[Fan Liya]

Hi JingsongLee,

Thanks a lot for your feedback. It's nice discussing with you guys :-)

My comments below:

1. Sorry I did not clearly state the experimental environment. The
evaluations were performed on a cluster with 21 N42 nodes (64 core, 256 GB
memory, 10 Gb/s network card). I will add such information in our doc. The
dataset is clearly stated in our proposal: 1TB TPC-H.

2. We emphasize that this is only initial implementation, and the
performance results are initial results. That means our work is far from
perfect. So invite the Flink community to work with us, to make it perfect.

3. Can you please give more details about blink-planner avoiding virtual
function calls? I would like to know more about that.

4. I agree that BinaryRow and ColumnarRow are highly efficient, compared
with previous row data structures in Flink. However, my point is that, they
can be further improved. Let's talk about BinaryRow first: Each field with
fixed width takes 8 bytes, right? That means 50% overhead for an int, 75%
overhead for a short/char, 87.5% overhead for a byte, and 98% overhead for
a boolean. The memory layout of BinaryRow is far from being perfectly
compact.

5. Then let's talk about ColumnarRow. It is closer to the perfect compact
memory layout. But, it is not perfect either. The limitations of
ColumnarRow include: 1) It is based on heap space, the problem with this
can be found in [1]. 2) Its memory layout can be further improved, e.g. by
using a bit for a nullablility. 3) Lack of operations (e.g. realloc).

6. Calculation push down sounds like a great idea. What is the relation to
vectorization?

7. We agree that row/batch conversion is a performance killer. With the
current row-based implementation, such conversion is unavoidable, for input
formats like orc and parquet, right? Vectorization eliminates such
conversions completely.

8. SIMD can be applied to a wide range of operations, not just filter &
calc. For example, in hash join & hash agg, SIMD can be used to accelerate
the calculation of hash codes. In addition, if we use a row-based memory
layout, additional data copy is required, and that is why we use
vectorization, right?

9. Whole stage code-gen seems like an orthogonal technique to
vectorization. It is feasible to achieve the benefits of both whole-stage
code-gen and vectorization.

10. Memory management is an important problem. Fortunately, Flink provides
interfaces for off-heap memory management. So Arrow memory buffer can be
integrated with little effort.

As indicated in our proposal, if the users do not like vectorization, they
can simply turn it off with a single flag. So at the very least,
vectorization will not bring harm to Flink. It just enrich the SQL engine
for Flink, making it perform well for an even wider range of scenarios.
Just think about it.

Best,
Liya Fan

[1]  Off-heap Memory in Apache Flink and the curious JIT compiler.
https://flink.apache.org/news/2015/09/16/off-heap-memory.html


On Tue, Jul 2, 2019 at 8:29 PM JingsongLee 
wrote:

> Thank liya for bringing this discussion.
> As the changes are very big and has a great impact on architecture.
>  I think we should be more clear about the benchmark. We need to
>  be cautious about testing to ensure that we really benefit from it.
>
> About the benchmark:
> 1.Can you make the test report clearer? For example, environment
> and data scale.
> 2.Can test scenarios be richer? For example, benchmark about
> spill scenarios. Benchmark about TPCDS.
> 3.Can we have more detailed test conclusions? About what kind of
> case will be quicker. At present, the calculation of blink-planner is
> not perfect. For example, it can avoid the overhead of virtual
> function calls. Aggregate algorithm needs to be improved. Can
> you can make further analysis with your benchmark.
>
> > More compact memory layout
> I think BinaryRow and ColumnarRow already have an efficient
> and compact memory layout.
>
> Just like you mentioned in doc. Blink's ColumnarRow now has
> vector computing features. And we can also push down a lot of
> calculations into the specific source, which can be more native
> to support the calculation near the source. I don't think complete
> vector calculation is that necessary. Because of the following
> reasons, the latter calculation is difficult to obtain benefits through
> Vector calculation:
> 1. Maybe the cost of conversion between VectorBatch and Row will
> be the performance killer. I think maybe we should do some
> performance test to it. If there are join/shuffle nodes, there will be
>  vector-to-row and row-to-vector overhead? These two operators
> are often the key to job performance.
> 2. Operators like sort, aggregation, their vectorized computational
> versions maybe need more benchmarks. I have no idea about it.
> 3. Now Java SIMD can only improve a limited number of vector
> computation like filter and calc, but often the bottleneck of batch
> jobs is not there, more on Join and Shuffler. Complete Java 

[jira] [Created] (FLINK-13067) Fix broken links to contributing docs

[Yun Tang (JIRA)]

Yun Tang created FLINK-13067:


 Summary: Fix broken links to contributing docs
 Key: FLINK-13067
 URL: https://issues.apache.org/jira/browse/FLINK-13067
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: Yun Tang
Assignee: Yun Tang


As contributing links change on [https://github.com/apache/flink-web], all 
links to contributing related docs have become broken. We need to fix these 
broken links.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] Apache Flink 1.8.1 released

[Jark Wu]

Thanks for being the release manager and the great job!

Cheers,
Jark

On Wed, 3 Jul 2019 at 10:16, Dian Fu  wrote:

> Awesome! Thanks a lot for being the release manager. Great job! @Jincheng
>
> Regards,
> Dian
>
> 在 2019年7月3日，上午10:08，jincheng sun  写道：
>
> I've also tweeted about it from my twitter:
> https://twitter.com/sunjincheng121/status/1146236834344648704
> later would be tweeted it from @ApacheFlink!
>
> Best, Jincheng
>
> Hequn Cheng  于2019年7月3日周三 上午9:48写道：
>
>> Thanks for being the release manager and the great work Jincheng!
>> Also thanks to Gorden and the community making this release possible!
>>
>> Best, Hequn
>>
>> On Wed, Jul 3, 2019 at 9:40 AM jincheng sun 
>> wrote:
>>
>>> Hi,
>>>
>>> The Apache Flink community is very happy to announce the release of
>>> Apache Flink 1.8.1, which is the first bugfix release for the Apache Flink
>>> 1.8 series.
>>>
>>> Apache Flink® is an open-source stream processing framework for
>>> distributed, high-performing, always-available, and accurate data streaming
>>> applications.
>>>
>>> The release is available for download at:
>>> https://flink.apache.org/downloads.html
>>>
>>> Please check out the release blog post for an overview of the
>>> improvements for this bugfix release:
>>> https://flink.apache.org/news/2019/07/02/release-1.8.1.html
>>>
>>> The full release notes are available in Jira:
>>>
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345164
>>>
>>> We would like to thank all contributors of the Apache Flink community
>>> who made this release possible!
>>>
>>> Great thanks to @Tzu-Li (Gordon) Tai  's offline
>>> kind help!
>>>
>>> Regards,
>>> Jincheng
>>>
>>
>

Re: [ANNOUNCE] Apache Flink 1.8.1 released

[Kurt Young]

Thanks for being the release manager and great job! @Jincheng

Best,
Kurt


On Wed, Jul 3, 2019 at 10:19 AM Tzu-Li (Gordon) Tai 
wrote:

> Thanks for being the release manager @jincheng sun
>  :)
>
> On Wed, Jul 3, 2019 at 10:16 AM Dian Fu  wrote:
>
>> Awesome! Thanks a lot for being the release manager. Great job! @Jincheng
>>
>> Regards,
>> Dian
>>
>> 在 2019年7月3日，上午10:08，jincheng sun  写道：
>>
>> I've also tweeted about it from my twitter:
>> https://twitter.com/sunjincheng121/status/1146236834344648704
>> later would be tweeted it from @ApacheFlink!
>>
>> Best, Jincheng
>>
>> Hequn Cheng  于2019年7月3日周三 上午9:48写道：
>>
>>> Thanks for being the release manager and the great work Jincheng!
>>> Also thanks to Gorden and the community making this release possible!
>>>
>>> Best, Hequn
>>>
>>> On Wed, Jul 3, 2019 at 9:40 AM jincheng sun 
>>> wrote:
>>>
 Hi,

 The Apache Flink community is very happy to announce the release of
 Apache Flink 1.8.1, which is the first bugfix release for the Apache Flink
 1.8 series.

 Apache Flink® is an open-source stream processing framework for
 distributed, high-performing, always-available, and accurate data streaming
 applications.

 The release is available for download at:
 https://flink.apache.org/downloads.html

 Please check out the release blog post for an overview of the
 improvements for this bugfix release:
 https://flink.apache.org/news/2019/07/02/release-1.8.1.html

 The full release notes are available in Jira:

 https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345164

 We would like to thank all contributors of the Apache Flink community
 who made this release possible!

 Great thanks to @Tzu-Li (Gordon) Tai  's offline
 kind help!

 Regards,
 Jincheng

>>>
>>

Re: [ANNOUNCE] Apache Flink 1.8.1 released

[Tzu-Li (Gordon) Tai]

Thanks for being the release manager @jincheng sun
 :)

On Wed, Jul 3, 2019 at 10:16 AM Dian Fu  wrote:

> Awesome! Thanks a lot for being the release manager. Great job! @Jincheng
>
> Regards,
> Dian
>
> 在 2019年7月3日，上午10:08，jincheng sun  写道：
>
> I've also tweeted about it from my twitter:
> https://twitter.com/sunjincheng121/status/1146236834344648704
> later would be tweeted it from @ApacheFlink!
>
> Best, Jincheng
>
> Hequn Cheng  于2019年7月3日周三 上午9:48写道：
>
>> Thanks for being the release manager and the great work Jincheng!
>> Also thanks to Gorden and the community making this release possible!
>>
>> Best, Hequn
>>
>> On Wed, Jul 3, 2019 at 9:40 AM jincheng sun 
>> wrote:
>>
>>> Hi,
>>>
>>> The Apache Flink community is very happy to announce the release of
>>> Apache Flink 1.8.1, which is the first bugfix release for the Apache Flink
>>> 1.8 series.
>>>
>>> Apache Flink® is an open-source stream processing framework for
>>> distributed, high-performing, always-available, and accurate data streaming
>>> applications.
>>>
>>> The release is available for download at:
>>> https://flink.apache.org/downloads.html
>>>
>>> Please check out the release blog post for an overview of the
>>> improvements for this bugfix release:
>>> https://flink.apache.org/news/2019/07/02/release-1.8.1.html
>>>
>>> The full release notes are available in Jira:
>>>
>>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345164
>>>
>>> We would like to thank all contributors of the Apache Flink community
>>> who made this release possible!
>>>
>>> Great thanks to @Tzu-Li (Gordon) Tai  's offline
>>> kind help!
>>>
>>> Regards,
>>> Jincheng
>>>
>>
>

Re: [ANNOUNCE] Apache Flink 1.8.1 released

[Dian Fu]

Awesome! Thanks a lot for being the release manager. Great job! @Jincheng

Regards,
Dian

> 在 2019年7月3日，上午10:08，jincheng sun  写道：
> 
> I've also tweeted about it from my twitter: 
> https://twitter.com/sunjincheng121/status/1146236834344648704 
>  
> later would be tweeted it from @ApacheFlink!
> 
> Best, Jincheng
> 
> Hequn Cheng mailto:chenghe...@gmail.com>> 于2019年7月3日周三 
> 上午9:48写道：
> Thanks for being the release manager and the great work Jincheng!
> Also thanks to Gorden and the community making this release possible!
> 
> Best, Hequn
> 
> On Wed, Jul 3, 2019 at 9:40 AM jincheng sun  > wrote:
> Hi,
> 
> The Apache Flink community is very happy to announce the release of Apache 
> Flink 1.8.1, which is the first bugfix release for the Apache Flink 1.8 
> series. 
> 
> Apache Flink® is an open-source stream processing framework for distributed, 
> high-performing, always-available, and accurate data streaming applications. 
> 
> The release is available for download at: 
> https://flink.apache.org/downloads.html  
> 
> 
> Please check out the release blog post for an overview of the 
> improvements for this bugfix release: 
> https://flink.apache.org/news/2019/07/02/release-1.8.1.html 
> 
> 
> The full release notes are available in Jira:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345164
>  
> 
> 
> We would like to thank all contributors of the Apache Flink community who 
> made this release possible! 
> 
> Great thanks to @Tzu-Li (Gordon) Tai  's offline 
> kind help!
> 
> Regards,
> Jincheng

Re: [ANNOUNCE] Apache Flink 1.8.1 released

[jincheng sun]

I've also tweeted about it from my twitter:
https://twitter.com/sunjincheng121/status/1146236834344648704
later would be tweeted it from @ApacheFlink!

Best, Jincheng

Hequn Cheng  于2019年7月3日周三 上午9:48写道：

> Thanks for being the release manager and the great work Jincheng!
> Also thanks to Gorden and the community making this release possible!
>
> Best, Hequn
>
> On Wed, Jul 3, 2019 at 9:40 AM jincheng sun 
> wrote:
>
>> Hi,
>>
>> The Apache Flink community is very happy to announce the release of
>> Apache Flink 1.8.1, which is the first bugfix release for the Apache Flink
>> 1.8 series.
>>
>> Apache Flink® is an open-source stream processing framework for
>> distributed, high-performing, always-available, and accurate data streaming
>> applications.
>>
>> The release is available for download at:
>> https://flink.apache.org/downloads.html
>>
>> Please check out the release blog post for an overview of the
>> improvements for this bugfix release:
>> https://flink.apache.org/news/2019/07/02/release-1.8.1.html
>>
>> The full release notes are available in Jira:
>>
>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345164
>>
>> We would like to thank all contributors of the Apache Flink community who
>> made this release possible!
>>
>> Great thanks to @Tzu-Li (Gordon) Tai  's offline
>> kind help!
>>
>> Regards,
>> Jincheng
>>
>

Re: [ANNOUNCE] Apache Flink 1.8.1 released

[Hequn Cheng]

Thanks for being the release manager and the great work Jincheng!
Also thanks to Gorden and the community making this release possible!

Best, Hequn

On Wed, Jul 3, 2019 at 9:40 AM jincheng sun 
wrote:

> Hi,
>
> The Apache Flink community is very happy to announce the release of Apache
> Flink 1.8.1, which is the first bugfix release for the Apache Flink 1.8
> series.
>
> Apache Flink® is an open-source stream processing framework for
> distributed, high-performing, always-available, and accurate data streaming
> applications.
>
> The release is available for download at:
> https://flink.apache.org/downloads.html
>
> Please check out the release blog post for an overview of the
> improvements for this bugfix release:
> https://flink.apache.org/news/2019/07/02/release-1.8.1.html
>
> The full release notes are available in Jira:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345164
>
> We would like to thank all contributors of the Apache Flink community who
> made this release possible!
>
> Great thanks to @Tzu-Li (Gordon) Tai  's offline
> kind help!
>
> Regards,
> Jincheng
>

[ANNOUNCE] Apache Flink 1.8.1 released

[jincheng sun]

Hi,

The Apache Flink community is very happy to announce the release of Apache
Flink 1.8.1, which is the first bugfix release for the Apache Flink 1.8
series.

Apache Flink® is an open-source stream processing framework for
distributed, high-performing, always-available, and accurate data streaming
applications.

The release is available for download at:
https://flink.apache.org/downloads.html

Please check out the release blog post for an overview of the
improvements for this bugfix release:
https://flink.apache.org/news/2019/07/02/release-1.8.1.html

The full release notes are available in Jira:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12345164

We would like to thank all contributors of the Apache Flink community who
made this release possible!

Great thanks to @Tzu-Li (Gordon) Tai  's offline kind
help!

Regards,
Jincheng

[jira] [Created] (FLINK-13066) append hive-site.xml to path defined in 'hive-conf-dir'

[Bowen Li (JIRA)]

Bowen Li created FLINK-13066:


 Summary: append hive-site.xml to path defined in 'hive-conf-dir'
 Key: FLINK-13066
 URL: https://issues.apache.org/jira/browse/FLINK-13066
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: Bowen Li
Assignee: Bowen Li
 Fix For: 1.9.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13065) Document example snippet correction using KeySelector

[Mans Singh (JIRA)]

Mans Singh created FLINK-13065:
--

 Summary: Document example snippet correction using KeySelector
 Key: FLINK-13065
 URL: https://issues.apache.org/jira/browse/FLINK-13065
 Project: Flink
  Issue Type: Improvement
  Components: Documentation
Reporter: Mans Singh
Assignee: Mans Singh


The broadcast state 
[example|[https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/state/broadcast_state.html#provided-apis]]
 states:

 
{noformat}
Starting from the stream of Items, we just need to key it by Color, as we want 
pairs of the same color. This will make sure that elements of the same color 
end up on the same physical machine.

// key the shapes by color
KeyedStream colorPartitionedStream = shapeStream
.keyBy(new KeySelector(){...});{noformat}
 

How it uses shape stream and use KeySelector but should use 
KeySelector to create KeyedStream.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13064) Add Python Table API to Table Api Walkthrough

[Seth Wiesman (JIRA)]

Seth Wiesman created FLINK-13064:


 Summary: Add Python Table API to Table Api Walkthrough
 Key: FLINK-13064
 URL: https://issues.apache.org/jira/browse/FLINK-13064
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Seth Wiesman






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13063) AsyncWaitOperator can loose data during checkpointing

[Piotr Nowojski (JIRA)]

Piotr Nowojski created FLINK-13063:
--

 Summary: AsyncWaitOperator can loose data during checkpointing
 Key: FLINK-13063
 URL: https://issues.apache.org/jira/browse/FLINK-13063
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Affects Versions: 1.8.1, 1.7.2, 1.6.4, 1.9.0
Reporter: Piotr Nowojski
Assignee: Piotr Nowojski


For the following setup of chained operators:
{noformat}
SourceOperator -> FlatMap -> AsyncOperator{noformat}
Lets assume that input buffer of {{AsyncOperator}} is full. We start processing 
a record from the {{SourceOperator}}, we pass it to the {{FlatMap}}, which fan 
it out (multiplies it 10 times). First multiplied record reaches 
{{AsyncOperator}} and is special treated (stored in 
{{AsyncWaitOperator#pendingStreamElementQueueEntry}} ) and then 
{{AsyncWaitOperator}} waits (and releases) on the checkpoint lock (in 
{{AsyncWaitOperator#addAsyncBufferEntry}} . If a checkpoint is triggered now, 
both {{SourceOperator}} and {{FlatMap}} will be checkpointed assumed that all 
of those 10 multiplied records were processed, which is not true. Only the 
first one is checkpointed by the {{AsyncWatiOperator}}. Remaining 9 are not. So 
if we ever restore state from this checkpoint, we have lost those 9 records.

CC [~aljoscha] [~StephanEwen] [~srichter] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13062) Set ScheduleMode based on boundedness of streaming Pipeline

[Aljoscha Krettek (JIRA)]

Aljoscha Krettek created FLINK-13062:


 Summary: Set ScheduleMode based on boundedness of streaming 
Pipeline
 Key: FLINK-13062
 URL: https://issues.apache.org/jira/browse/FLINK-13062
 Project: Flink
  Issue Type: Improvement
  Components: API / DataStream
Reporter: Aljoscha Krettek
Assignee: Aljoscha Krettek


The new Blink-based Table Runner needs "streaming pipelines" to be executed 
with {{ScheduleMode.LAZY_FROM_SOURCES}} if all sources are bounded. The current 
Blink code base uses a global flag for this and configures the 
{{StreamGraphGenerator}} accordingly.

We propose to add an {{isBounded()}} property to {{Transformation}} (formerly 
known as {{StreamTransformation}}). The property would only be explicitly 
settable on sources, other transformations inherit the property from their 
inputs. The {{StreamGraphGenerator}} must use 
{{ScheduleMode.LAZY_FROM_SOURCES}} if all sources are bounded, otherwise, it 
should use {{ScheduleMode.EAGER}}, as is the currently existing behaviour.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13061) Support set hive table Ser/de properites in HiveCatalog

[zjuwangg (JIRA)]

zjuwangg created FLINK-13061:


 Summary: Support set hive table Ser/de properites in HiveCatalog
 Key: FLINK-13061
 URL: https://issues.apache.org/jira/browse/FLINK-13061
 Project: Flink
  Issue Type: Sub-task
Reporter: zjuwangg






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13060) FailoverStrategies should respect restart constraints

[Chesnay Schepler (JIRA)]

Chesnay Schepler created FLINK-13060:


 Summary: FailoverStrategies should respect restart constraints
 Key: FLINK-13060
 URL: https://issues.apache.org/jira/browse/FLINK-13060
 Project: Flink
  Issue Type: Improvement
  Components: Runtime / Coordination
Reporter: Chesnay Schepler
Assignee: Chesnay Schepler
 Fix For: 1.9.0


RestartStrategies can define their own restrictions for whether job can be 
restarted or not. For example, they could count the number of total failures or 
observe failure rates.

FailoverStrategies are used for partial restarts of jobs, and currently largely 
bypass the restrictions defined by the restart strategies.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Vectorization Support in Flink

[JingsongLee]

Thank liya for bringing this discussion.
As the changes are very big and has a great impact on architecture.
 I think we should be more clear about the benchmark. We need to
 be cautious about testing to ensure that we really benefit from it.

About the benchmark:
1.Can you make the test report clearer? For example, environment
and data scale.
2.Can test scenarios be richer? For example, benchmark about
spill scenarios. Benchmark about TPCDS.
3.Can we have more detailed test conclusions? About what kind of 
case will be quicker. At present, the calculation of blink-planner is 
not perfect. For example, it can avoid the overhead of virtual 
function calls. Aggregate algorithm needs to be improved. Can 
you can make further analysis with your benchmark.

> More compact memory layout
I think BinaryRow and ColumnarRow already have an efficient 
and compact memory layout.

Just like you mentioned in doc. Blink's ColumnarRow now has 
vector computing features. And we can also push down a lot of 
calculations into the specific source, which can be more native 
to support the calculation near the source. I don't think complete 
vector calculation is that necessary. Because of the following 
reasons, the latter calculation is difficult to obtain benefits through 
Vector calculation:
1. Maybe the cost of conversion between VectorBatch and Row will 
be the performance killer. I think maybe we should do some 
performance test to it. If there are join/shuffle nodes, there will be
 vector-to-row and row-to-vector overhead? These two operators 
are often the key to job performance.
2. Operators like sort, aggregation, their vectorized computational 
versions maybe need more benchmarks. I have no idea about it.
3. Now Java SIMD can only improve a limited number of vector 
computation like filter and calc, but often the bottleneck of batch 
jobs is not there, more on Join and Shuffler. Complete Java vector 
computing looks like a long way off. If we vectorize through JNI, 
the cost of JNI can not be ignored. And SIMD algorithm is not 
necessarily faster, it brings a lot of additional data copies.
4. If we move forward with CodGenerator(Like Spark 
WholeStageCodeGen), can we achieve better results without 
vector computation? The JavaCompiler/JVM may optimize the 
code to SIMD.

Other thing is that the vector version of operators maybe need 
consider the problem of memory management?

Best, JingsongLee


--
From:Fan Liya 
Send Time:2019年7月2日(星期二) 16:31
To:dev ; Ji Liu 
Subject:Re: [DISCUSS] Vectorization Support in Flink

@Ji Liu, thanks a lot for your feedback.
This work must be performed in a progressive manner, so as not to break
existing code.

Best,
Liya Fan

On Tue, Jul 2, 2019 at 3:57 PM Ji Liu  wrote:

> Hi Liya,
> Thanks for opening this discuss.
> +1 for this, vectorization makes sense for Flink especially for batch work
> loads, I think Flink should look into supporting it progressively.
>
> Thanks,
> Ji Liu
>
>
> --
> From:Jeff Zhang 
> Send Time:2019年7月2日(星期二) 15:50
> To:dev 
> Subject:Re: [DISCUSS] Vectorization Support in Flink
>
> Hi Liya,
>
> Displaying image is not supported in apache mail list, you need to put it
> elsewhere and post link in mail list.
>
>
>
> Fan Liya  于2019年7月2日周二 下午3:40写道：
>
> > Performance chart. FYI.
> >
> > Best,
> > Liya Fan
> > [image: image.png]
> >
> > On Tue, Jul 2, 2019 at 3:37 PM Fan Liya  wrote:
> >
> >> Hi all,
> >>
> >> We have opened an issue about vectorization in Flink (FLINK-13053
> >> ). Would you please
> >> give your valuable feedback? Thank you in advance.
> >>
> >> Vectorization is a popular technique in SQL engines today. Compared with
> >> traditional row-based approach, it has some distinct advantages, for
> >> example:
> >>
> >>
> >>
> >> 1)  Better use of CPU resources (e.g. SIMD)
> >>
> >> 2)  More compact memory layout
> >>
> >> 3)  More friendly to compressed data format.
> >>
> >>
> >>
> >> Currently, Flink is based on a row-based SQL engine for both stream and
> >> batch workloads. To enjoy the above benefits, we want to bring
> >> vectorization to Flink. This involves substantial changes to the
> existing
> >> code base. Therefore, we give a plan to carry out such changes in small,
> >> incremental steps, in order not to affect existing features. We want to
> >> apply it to batch workload first. The details can be found in our
> proposal.
> >>
> >>
> >>
> >> For the past months, we have developed an initial implementation of the
> >> above ideas. Initial performance evaluations on TPC-H benchmarks show
> that
> >> substantial performance improvements can be obtained by vectorization
> (see
> >> the figure below). More details can be found in our proposal.
> >>
> >>
> >>
> >> [image:
> >>
> 

[jira] [Created] (FLINK-13059) Cassandra Connector leaks Semaphore on Exception; hangs on close

[Mads Chr. Olesen (JIRA)]

Mads Chr. Olesen created FLINK-13059:


 Summary: Cassandra Connector leaks Semaphore on Exception; hangs 
on close
 Key: FLINK-13059
 URL: https://issues.apache.org/jira/browse/FLINK-13059
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Cassandra
Affects Versions: 1.8.0
Reporter: Mads Chr. Olesen


In CassandraSinkBase the following code is present (comments are mine):

 
{code:java}
public void invoke(IN value) throws Exception {
   checkAsyncErrors();
   tryAcquire();
   //Semaphore held here

   final ListenableFuture result = send(value);

   Futures.addCallback(result, callback); //Callback releases semaphore
}{code}
Any Exception happening inside send(value) will result in the semaphore not 
being released. Such exceptions are possible, e.g.
{code:java}
com.datastax.driver.core.exceptions.InvalidQueryException: Some partition key 
parts are missing: hest
at 
com.datastax.driver.core.exceptions.InvalidQueryException.copy(InvalidQueryException.java:50)
at 
com.datastax.driver.core.DriverThrowables.propagateCause(DriverThrowables.java:37)
at com.datastax.driver.core.AbstractSession.prepare(AbstractSession.java:98)
at com.datastax.driver.mapping.Mapper.getPreparedQuery(Mapper.java:118)
at com.datastax.driver.mapping.Mapper.saveQuery(Mapper.java:201)
at com.datastax.driver.mapping.Mapper.saveQuery(Mapper.java:163)
at 
org.apache.flink.streaming.connectors.cassandra.CassandraPojoSink.send(CassandraPojoSink.java:128)
at 
org.apache.flink.streaming.connectors.cassandra.CassandraSinkBase.invoke(CassandraSinkBase.java:131)
at 
org.apache.flink.streaming.api.functions.sink.SinkFunction.invoke(SinkFunction.java:52)
{code}
The result of the semaphore not being released will be that when the exception 
bubbles out and causes the job to close, CassandraSinkBase.flush() will 
eventually be called. Flush will be deadlocked trying to acquire 
config.getMaxConcurrentRequests() from the semaphore, which has 1 less than 
that available.

The Flink job will thus be half-way closed, but marked as "RUNNING". 
Checkpointing will however fail with
{noformat}
INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Discarding 
checkpoint 201325 of job XXX. 
org.apache.flink.runtime.checkpoint.decline.CheckpointDeclineTaskNotReadyException:
 Task Source: XXX (3/4) was not running {noformat}
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Chesnay Schepler]

As a short-term stopgap, since we can assume this issue to become much 
worse in the following days/weeks, we could disable IT cases in PRs and 
only run them on master.


On 02/07/2019 12:03, Chesnay Schepler wrote:
People really have to stop thinking that just because something works 
for us it is also a good solution.
Also, please remember that our builds run for 2h from start to finish, 
and not the 14 _minutes_ it takes for zeppelin.
We are dealing with an entirely different scale here, both in terms of 
build times and number of builds.


In this very thread people have been complaining about long queue 
times for their builds. Surprise, other Apache projects have been 
suffering the very same thing due to us not controlling our build 
times. While switching services (be it Jenkins, CircleCI or whatever) 
will possibly work for us (and these options are actually attractive, 
like CircleCI's proper support for build artifacts), it will also 
result in us likely negatively affecting other projects in significant 
ways.


Sure, the Jenkins setup has a good user experience for us, at the cost 
of blocking Jenkins workers for a _lot_ of time. Right now we have 25 
PR's in our queue; that's possibly 50h we'd consume of Jenkins 
resources, and the European contributors haven't even really started yet.


FYI, the latest INFRA response from INFRA-18533:

"Our rough metrics shows that Flink used over 5800 hours of build time 
last month. That is equal to EIGHT servers running 24/7 for the ENTIRE 
MONTH. EIGHT. nonstop.
When we discovered this last night, we discussed it some and are going 
to tune down Flink to allow only five executors maximum. We cannot 
allow Flink to consume so much of a Foundation shared resource."


So yes, we either
a) have to heavily reduce our CI usage or
b) fund our own, either maintaining it ourselves or donating to Apache.

On 02/07/2019 05:11, Bowen Li wrote:
By looking at the git history of the Jenkins script, its core part 
was finished in March 2017 (and only two minor update in 2017/2018), 
so it's been running for over two years now and feels like Zepplin 
community has been quite happy with it. @Jeff Zhang 
 can you share your insights and user 
experience with the Jenkins+Travis approach?


Things like:

- has the approach completely solved the resource capacity problem 
for Zepplin community? is Zepplin community happy with the result?

- is the whole configuration chain stable (e.g. uptime) enough?
- how often do you need to maintain the Jenkins infra? how many 
people are usually involved in maintenance and bug-fixes?


The downside of this approach seems mostly to be on the maintenance 
to me - maintain the script and Jenkins infra.


** Having Our Own Travis-CI.com Account **

Another alternative I've been thinking of is to have our own 
travis-ci.com  account with paid dedicated 
resources. Note travis-ci.org  is the free 
version and travis-ci.com  is the commercial 
version. We currently use a shared resource pool managed by ASK INFRA 
team on travis-ci.org , but we have no control 
over it - we can't see how it's configured, how much resources are 
available, how resources are allocated among Apache projects, etc. 
The nice thing about having an account on travis-ci.com 
 are:


- relatively low cost with much better resource guarantee than what 
we currently have [1]: $249/month with 5 dedicated concurrency, 
$489/month with 10 concurrency

- low maintenance work compared to using Jenkins
- (potentially) no migration cost according to Travis's doc [2] 
(pending verification)
- full control over the build capacity/configuration compared to 
using ASF INFRA's pool


I'd be surprised if we as such a vibrant community cannot find and 
fund $249*12=$2988 a year in exchange for a much better developer 
experience and much higher productivity.


[1] https://travis-ci.com/plans
[2] 
https://docs.travis-ci.com/user/migrate/open-source-repository-migration


On Sat, Jun 29, 2019 at 8:39 AM Chesnay Schepler > wrote:


So yes, the Jenkins job keeps pulling the state from Travis until it
finishes.

Note sure I'm comfortable with the idea of using Jenkins workers
just to
idle for a several hours.

On 29/06/2019 14:56, Jeff Zhang wrote:
> Here's what zeppelin community did, we make a python script to
check the
> build status of pull request.
> Here's script:
> https://github.com/apache/zeppelin/blob/master/travis_check.py
>
> And this is the script we used in Jenkins build job.
>
> if [ -f "travis_check.py" ]; then
>git log -n 1
>STATUS=$(curl -s $BUILD_URL | grep -e "GitHub pull
request.*from.*" | sed
> 's/.*GitHub pull request  href=\"\(https[^"]*\).*from[^"]*.\(https[^"]*\).*/\1 \2/g')
>AUTHOR=$(echo $STATUS | sed 

Re: [DISCUSS] solve unstable build capacity problem on TravisCI

[Chesnay Schepler]

People really have to stop thinking that just because something works 
for us it is also a good solution.
Also, please remember that our builds run for 2h from start to finish, 
and not the 14 _minutes_ it takes for zeppelin.
We are dealing with an entirely different scale here, both in terms of 
build times and number of builds.


In this very thread people have been complaining about long queue times 
for their builds. Surprise, other Apache projects have been suffering 
the very same thing due to us not controlling our build times. While 
switching services (be it Jenkins, CircleCI or whatever) will possibly 
work for us (and these options are actually attractive, like CircleCI's 
proper support for build artifacts), it will also result in us likely 
negatively affecting other projects in significant ways.


Sure, the Jenkins setup has a good user experience for us, at the cost 
of blocking Jenkins workers for a _lot_ of time. Right now we have 25 
PR's in our queue; that's possibly 50h we'd consume of Jenkins 
resources, and the European contributors haven't even really started yet.


FYI, the latest INFRA response from INFRA-18533:

"Our rough metrics shows that Flink used over 5800 hours of build time 
last month. That is equal to EIGHT servers running 24/7 for the ENTIRE 
MONTH. EIGHT. nonstop.
When we discovered this last night, we discussed it some and are going 
to tune down Flink to allow only five executors maximum. We cannot allow 
Flink to consume so much of a Foundation shared resource."


So yes, we either
a) have to heavily reduce our CI usage or
b) fund our own, either maintaining it ourselves or donating to Apache.

On 02/07/2019 05:11, Bowen Li wrote:
By looking at the git history of the Jenkins script, its core part was 
finished in March 2017 (and only two minor update in 2017/2018), so 
it's been running for over two years now and feels like Zepplin 
community has been quite happy with it. @Jeff Zhang 
 can you share your insights and user 
experience with the Jenkins+Travis approach?


Things like:

- has the approach completely solved the resource capacity problem for 
Zepplin community? is Zepplin community happy with the result?

- is the whole configuration chain stable (e.g. uptime) enough?
- how often do you need to maintain the Jenkins infra? how many people 
are usually involved in maintenance and bug-fixes?


The downside of this approach seems mostly to be on the maintenance to 
me - maintain the script and Jenkins infra.


** Having Our Own Travis-CI.com Account **

Another alternative I've been thinking of is to have our own 
travis-ci.com  account with paid dedicated 
resources. Note travis-ci.org  is the free 
version and travis-ci.com  is the commercial 
version. We currently use a shared resource pool managed by ASK INFRA 
team on travis-ci.org , but we have no control 
over it - we can't see how it's configured, how much resources are 
available, how resources are allocated among Apache projects, etc. The 
nice thing about having an account on travis-ci.com 
 are:


- relatively low cost with much better resource guarantee than what we 
currently have [1]: $249/month with 5 dedicated concurrency, 
$489/month with 10 concurrency

- low maintenance work compared to using Jenkins
- (potentially) no migration cost according to Travis's doc [2] 
(pending verification)
- full control over the build capacity/configuration compared to using 
ASF INFRA's pool


I'd be surprised if we as such a vibrant community cannot find and 
fund $249*12=$2988 a year in exchange for a much better developer 
experience and much higher productivity.


[1] https://travis-ci.com/plans
[2] 
https://docs.travis-ci.com/user/migrate/open-source-repository-migration


On Sat, Jun 29, 2019 at 8:39 AM Chesnay Schepler > wrote:


So yes, the Jenkins job keeps pulling the state from Travis until it
finishes.

Note sure I'm comfortable with the idea of using Jenkins workers
just to
idle for a several hours.

On 29/06/2019 14:56, Jeff Zhang wrote:
> Here's what zeppelin community did, we make a python script to
check the
> build status of pull request.
> Here's script:
> https://github.com/apache/zeppelin/blob/master/travis_check.py
>
> And this is the script we used in Jenkins build job.
>
> if [ -f "travis_check.py" ]; then
>git log -n 1
>STATUS=$(curl -s $BUILD_URL | grep -e "GitHub pull
request.*from.*" | sed
> 's/.*GitHub pull request  href=\"\(https[^"]*\).*from[^"]*.\(https[^"]*\).*/\1 \2/g')
>AUTHOR=$(echo $STATUS | sed 's/.*[/]\(.*\)$/\1/g')
>PR=$(echo $STATUS | awk '{print $1}' | sed 's/.*[/]\(.*\)$/\1/g')
>#COMMIT=$(git log -n 1 | grep "^Merge:" | awk '{print $3}')
>#if [ -z $COMMIT ]; then
>#  COMMIT=$(curl -s

[jira] [Created] (FLINK-13058) Avoid memory copy for the trimming operations of BinaryString

[Liya Fan (JIRA)]

Liya Fan created FLINK-13058:


 Summary: Avoid memory copy for the trimming operations of 
BinaryString
 Key: FLINK-13058
 URL: https://issues.apache.org/jira/browse/FLINK-13058
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Reporter: Liya Fan
Assignee: Liya Fan


For trimming operations of BinaryString (trim, trimLeft, trimRight), if the 
trimmed string is identical to the original string. The memory copy can be 
avoided by directly returning the original string. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13057) Correct comments in ListState class

[Hequn Cheng (JIRA)]

Hequn Cheng created FLINK-13057:
---

 Summary: Correct comments in ListState class
 Key: FLINK-13057
 URL: https://issues.apache.org/jira/browse/FLINK-13057
 Project: Flink
  Issue Type: Bug
  Components: Runtime / State Backends
Reporter: Hequn Cheng
Assignee: Hequn Cheng


ListState can be a keyed state or an operator state, but the comment in 
ListState said it can only be a keyed state：
{code:java}
The state is only accessible by functions applied on a {@code KeyedStream}. 
{code}
We can change the comment from
{code:java}
* The state is only accessible by functions applied on a {@code 
KeyedStream}. The key is
* automatically supplied by the system, so the function always sees the value 
mapped to the
* key of the current element. That way, the system can handle stream and state 
partitioning
* consistently together.{code}
to
{code:java}
* The state can be a keyed state or an operator state. When it is a keyed 
state, it is only
* accessible by functions applied on a {@code KeyedStream}. The key is 
automatically supplied by
* the system, so the function always sees the value mapped to the key of the 
current element.
* That way, the system can handle stream and state partitioning consistently 
together.{code}
Appreciate any suggestions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13056) Optimize region failover performance on calculating vertices to restart

[Zhu Zhu (JIRA)]

Zhu Zhu created FLINK-13056:
---

 Summary: Optimize region failover performance on calculating 
vertices to restart
 Key: FLINK-13056
 URL: https://issues.apache.org/jira/browse/FLINK-13056
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Zhu Zhu
Assignee: Zhu Zhu


Currently some region boundary structures are calculated each time of a region 
failover. This calculation can be heavy as its complexity goes up with 
execution edge count.

We tested it in a sample case with 8000 vertices and 16,000,000 edges. It takes 
~2.0s to calculate vertices to restart.

(more details in 
[https://docs.google.com/document/d/197Ou-01h2obvxq8viKqg4FnOnsykOEKxk3r5WrVBPuA/edit?usp=sharing)]

That's why we'd propose to cache the region boundary structures to improve the 
region failover performance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13055) Leverage JM side partition state to improve region failover experience

[Zhu Zhu (JIRA)]

Zhu Zhu created FLINK-13055:
---

 Summary: Leverage JM side partition state to improve region 
failover experience
 Key: FLINK-13055
 URL: https://issues.apache.org/jira/browse/FLINK-13055
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Affects Versions: 1.8.1
Reporter: Zhu Zhu
Assignee: Zhu Zhu


In current region failover process, most of the input result partition states 
are unknown. Even though the failure cause is a PartitionException, only one 
unhealthy partition can be identified.

The may lead to multiple unsuccessful failovers before all the unhealthy but 
needed partitions are identified and their producers are involved in the 
failover as well.

Using JM side tracked partition states to help the region failover to identify 
unhealthy(missing) partitions earlier can help with this case.

The basic idea is to build RestartPipelinedRegionStrategy with a 
ResultPartitionAvailabilityChecker which can query the JM side tracked 
partition states.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: RE: [DISCUSS] Improve Queryable State and introduce a QueryServerProxy component

[vino yang]

Hi all,

In the past, I have tried to further refine the design of this topic thread
and wrote a design document to give more detailed design images and text
description, so that it is more conducive to discussion.[1]

Note: The document is not yet completed, for example, the "Implementation"
section is missing. Therefore, it is still in an open discussion state. I
will improve the rest while listening to the opinions of the community.

Welcome and appreciate more discussions and feedback.

Best,
Vino

[1]:
https://docs.google.com/document/d/181qYVIiHQGrc3hCj3QBn1iEHF4bUztdw4XO8VSaf_uI/edit?usp=sharing


yanghua1127  于2019年6月7日周五 下午11:32写道：

> Hi Georgi,
>
> Thanks for your feedback. And glad to hear you are using queryable state.
>
> I agree that implementation of option 1 is easier than others. However,
> when we design the new architecture we need to consider more aspects .e.g.
> scalability. So it seems option 3 is more suitable. Actually, some
> committers such as Stefan, Gordon and Aljoscha have given me feedback and
> direction.
>
> Currently, I am writing the design document. If it is ready to be
> presented. I will copy to this thread and we can discuss further details.
>
> 
> Best,
> Vino
>
>
> On 2019-06-07 19:03 , Georgi Stoyanov  Wrote:
>
> Hi Vino,
>
>
>
> I was investigating the current architecture and AFAIK the first proposal
> will be a lot easier to implement, cause currently JM has the information
> about the states (where, which etc thanks to KvStateLocationRegistry.
> Correct me if I’m wrong)
>
> We are using the feature and it’s indeed not very cool to iterate trough
> ports, check which TM is the responsible one etc etc.
>
>
>
> It will be very useful if someone from the committers joins the topic and
> give us some insights what’s going to happen with that feature.
>
>
>
>
>
> Kind Regards,
>
> Georgi
>
>
>
>
>
>
>
> *From:* vino yang 
> *Sent:* Thursday, April 25, 2019 5:18 PM
> *To:* dev ; user 
> *Cc:* Stefan Richter ; Aljoscha Krettek <
> aljos...@apache.org>; kklou...@gmail.com
> *Subject:* [DISCUSS] Improve Queryable State and introduce a
> QueryServerProxy component
>
>
>
> Hi all,
>
>
>
> I want to share my thought with you about improving the queryable state
> and introducing a QueryServerProxy component.
>
>
>
> I think the current queryable state's client is hard to use. Because it
> needs users to know the TaskManager's address and proxy's port. Actually,
> some business users who do not have good knowledge about the Flink's inner
> or runtime in production. However, sometimes they need to query the values
> of states.
>
>
>
> IMO, the reason caused this problem is because of the queryable state's
> architecture. Currently, the queryable state clients interact with
> query state client proxy components which host on each TaskManager. This
> design is difficult to encapsulate the point of change and exposes too much
> detail to the user.
>
>
>
> My personal idea is that we could introduce a really queryable state
> server, named e.g. *QueryStateProxyServer* which would delegate all the
> query state request and query the local registry then redirect the request
> to the specific *QueryStateClientProxy*(runs on each TaskManager). The
> server is the users really want to care about. And it would make the users
> ignorant to the TaskManagers' address and proxies' port. The current
> *QueryStateClientProxy* would become *QueryStateProxyClient*.
>
>
>
> Generally speaking, the roles of the QueryStateProxyServer list below:
>
>
>
>- works as all the query client's proxy to receive all the request and
>send response;
>- a router to redirect the real query requests to the specific proxy
>client;
>- maintain route table registry (state <-> TaskManager,
>TaskManager<->proxy client address)
>- more fine-granted control, such as cache result, ACL, TTL, SLA(rate
>limit) and so on
>
> About the implementation, there are three opts:
>
>
>
> opt 1:
>
>
>
> Let the JobManager acts as the query proxy server.
>
> ·  pros: reuse the exists JM, do not need to introduce a new process can
> reduce the complexity;
>
> ·  cons: would make JM heavy burdens, depends on the query frequency, may
> impact on the stability
>
>
>
> [image: Screen Shot 2019-04-25 at 5.12.07 PM.png]
>
>
>
> opt 2:
>
>
>
> Introduce a new component  which runs as a single process and acts as the
> query proxy server:
>
>
>
> ·  pros: reduce the burdens and make the JM more stability
>
> ·  cons: introduced a new component will make the implementation more
> complexity
>
> [image: Screen Shot 2019-04-25 at 5.14.05 PM.png]
>
>
>
> opt 3 (suggestion comes from Stefan Richter):
>
>
>
> Combining the two opts, the query server could run as a single entry
> point(process) and integrate with JobManager.
>
>
>
> If we keep it well encapsulated, the only difference would be how we
> register new TMs with the query server in the different scenarios, in JM we
> might have this information 

[jira] [Created] (FLINK-13054) [Java] Support compact fixed-width vectors

[Ji Liu (JIRA)]

Ji Liu created FLINK-13054:
--

 Summary: [Java] Support compact fixed-width vectors
 Key: FLINK-13054
 URL: https://issues.apache.org/jira/browse/FLINK-13054
 Project: Flink
  Issue Type: New Feature
Reporter: Ji Liu
Assignee: Ji Liu


In shuffle stage of some applications, FixedWitdhVectors may have very little 
non-null data.
In this case, directly serialize vectors is not a good choice, generally we can 
compact the vector make it only holding non-null value and create a BitVector 
to trace the indices for non-null values so that it could be deserialized 
properly.





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Vectorization Support in Flink

[Fan Liya]

@Ji Liu, thanks a lot for your feedback.
This work must be performed in a progressive manner, so as not to break
existing code.

Best,
Liya Fan

On Tue, Jul 2, 2019 at 3:57 PM Ji Liu  wrote:

> Hi Liya,
> Thanks for opening this discuss.
> +1 for this, vectorization makes sense for Flink especially for batch work
> loads, I think Flink should look into supporting it progressively.
>
> Thanks,
> Ji Liu
>
>
> --
> From:Jeff Zhang 
> Send Time:2019年7月2日(星期二) 15:50
> To:dev 
> Subject:Re: [DISCUSS] Vectorization Support in Flink
>
> Hi Liya,
>
> Displaying image is not supported in apache mail list, you need to put it
> elsewhere and post link in mail list.
>
>
>
> Fan Liya  于2019年7月2日周二 下午3:40写道：
>
> > Performance chart. FYI.
> >
> > Best,
> > Liya Fan
> > [image: image.png]
> >
> > On Tue, Jul 2, 2019 at 3:37 PM Fan Liya  wrote:
> >
> >> Hi all,
> >>
> >> We have opened an issue about vectorization in Flink (FLINK-13053
> >> ). Would you please
> >> give your valuable feedback? Thank you in advance.
> >>
> >> Vectorization is a popular technique in SQL engines today. Compared with
> >> traditional row-based approach, it has some distinct advantages, for
> >> example:
> >>
> >>
> >>
> >> 1)  Better use of CPU resources (e.g. SIMD)
> >>
> >> 2)  More compact memory layout
> >>
> >> 3)  More friendly to compressed data format.
> >>
> >>
> >>
> >> Currently, Flink is based on a row-based SQL engine for both stream and
> >> batch workloads. To enjoy the above benefits, we want to bring
> >> vectorization to Flink. This involves substantial changes to the
> existing
> >> code base. Therefore, we give a plan to carry out such changes in small,
> >> incremental steps, in order not to affect existing features. We want to
> >> apply it to batch workload first. The details can be found in our
> proposal.
> >>
> >>
> >>
> >> For the past months, we have developed an initial implementation of the
> >> above ideas. Initial performance evaluations on TPC-H benchmarks show
> that
> >> substantial performance improvements can be obtained by vectorization
> (see
> >> the figure below). More details can be found in our proposal.
> >>
> >>
> >>
> >> [image:
> >>
> https://lh5.googleusercontent.com/hjXkXGImWOjaiB8zF0SKIMoItY6VCBm-BmJWWEXRo0ZPHdwLgKzCmIoNKef1YPCaAA7NXN6RvO-nwBBXBee52KeAtBjyIvh_NcAuChvW3BEtQuZGL5GPddqxL_iMV7HvEVCC6k-m
> ]
> >>
> >>
> >>
> >> Special thanks to @Kurt Young’s team for all the kind help.
> >>
> >> Special thanks to @Piotr Nowojski for all the valuable feedback and help
> >> suggestions.
> >>
> >>
> >> Best,
> >>
> >> Liya Fan
> >>
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [DISCUSS] Vectorization Support in Flink

[Fan Liya]

@Jeff Zhang, thanks for your kind reminder.
The image can be found from our proposal

or the JIRA .

Best,
Liya Fan

On Tue, Jul 2, 2019 at 3:49 PM Jeff Zhang  wrote:

> Hi Liya,
>
> Displaying image is not supported in apache mail list, you need to put it
> elsewhere and post link in mail list.
>
>
>
> Fan Liya  于2019年7月2日周二 下午3:40写道：
>
> > Performance chart. FYI.
> >
> > Best,
> > Liya Fan
> > [image: image.png]
> >
> > On Tue, Jul 2, 2019 at 3:37 PM Fan Liya  wrote:
> >
> >> Hi all,
> >>
> >> We have opened an issue about vectorization in Flink (FLINK-13053
> >> ). Would you please
> >> give your valuable feedback? Thank you in advance.
> >>
> >> Vectorization is a popular technique in SQL engines today. Compared with
> >> traditional row-based approach, it has some distinct advantages, for
> >> example:
> >>
> >>
> >>
> >> 1)  Better use of CPU resources (e.g. SIMD)
> >>
> >> 2)  More compact memory layout
> >>
> >> 3)  More friendly to compressed data format.
> >>
> >>
> >>
> >> Currently, Flink is based on a row-based SQL engine for both stream and
> >> batch workloads. To enjoy the above benefits, we want to bring
> >> vectorization to Flink. This involves substantial changes to the
> existing
> >> code base. Therefore, we give a plan to carry out such changes in small,
> >> incremental steps, in order not to affect existing features. We want to
> >> apply it to batch workload first. The details can be found in our
> proposal.
> >>
> >>
> >>
> >> For the past months, we have developed an initial implementation of the
> >> above ideas. Initial performance evaluations on TPC-H benchmarks show
> that
> >> substantial performance improvements can be obtained by vectorization
> (see
> >> the figure below). More details can be found in our proposal.
> >>
> >>
> >>
> >> [image:
> >>
> https://lh5.googleusercontent.com/hjXkXGImWOjaiB8zF0SKIMoItY6VCBm-BmJWWEXRo0ZPHdwLgKzCmIoNKef1YPCaAA7NXN6RvO-nwBBXBee52KeAtBjyIvh_NcAuChvW3BEtQuZGL5GPddqxL_iMV7HvEVCC6k-m
> ]
> >>
> >>
> >>
> >> Special thanks to @Kurt Young’s team for all the kind help.
> >>
> >> Special thanks to @Piotr Nowojski for all the valuable feedback and help
> >> suggestions.
> >>
> >>
> >> Best,
> >>
> >> Liya Fan
> >>
> >
>
> --
> Best Regards
>
> Jeff Zhang
>

Re: [DISCUSS] Vectorization Support in Flink

[Ji Liu]

Hi Liya,
Thanks for opening this discuss.
+1 for this, vectorization makes sense for Flink especially for batch work 
loads, I think Flink should look into supporting it progressively.

Thanks,
Ji Liu


--
From:Jeff Zhang 
Send Time:2019年7月2日(星期二) 15:50
To:dev 
Subject:Re: [DISCUSS] Vectorization Support in Flink

Hi Liya,

Displaying image is not supported in apache mail list, you need to put it
elsewhere and post link in mail list.



Fan Liya  于2019年7月2日周二 下午3:40写道：

> Performance chart. FYI.
>
> Best,
> Liya Fan
> [image: image.png]
>
> On Tue, Jul 2, 2019 at 3:37 PM Fan Liya  wrote:
>
>> Hi all,
>>
>> We have opened an issue about vectorization in Flink (FLINK-13053
>> ). Would you please
>> give your valuable feedback? Thank you in advance.
>>
>> Vectorization is a popular technique in SQL engines today. Compared with
>> traditional row-based approach, it has some distinct advantages, for
>> example:
>>
>>
>>
>> 1)  Better use of CPU resources (e.g. SIMD)
>>
>> 2)  More compact memory layout
>>
>> 3)  More friendly to compressed data format.
>>
>>
>>
>> Currently, Flink is based on a row-based SQL engine for both stream and
>> batch workloads. To enjoy the above benefits, we want to bring
>> vectorization to Flink. This involves substantial changes to the existing
>> code base. Therefore, we give a plan to carry out such changes in small,
>> incremental steps, in order not to affect existing features. We want to
>> apply it to batch workload first. The details can be found in our proposal.
>>
>>
>>
>> For the past months, we have developed an initial implementation of the
>> above ideas. Initial performance evaluations on TPC-H benchmarks show that
>> substantial performance improvements can be obtained by vectorization (see
>> the figure below). More details can be found in our proposal.
>>
>>
>>
>> [image:
>> https://lh5.googleusercontent.com/hjXkXGImWOjaiB8zF0SKIMoItY6VCBm-BmJWWEXRo0ZPHdwLgKzCmIoNKef1YPCaAA7NXN6RvO-nwBBXBee52KeAtBjyIvh_NcAuChvW3BEtQuZGL5GPddqxL_iMV7HvEVCC6k-m]
>>
>>
>>
>> Special thanks to @Kurt Young’s team for all the kind help.
>>
>> Special thanks to @Piotr Nowojski for all the valuable feedback and help
>> suggestions.
>>
>>
>> Best,
>>
>> Liya Fan
>>
>

-- 
Best Regards

Jeff Zhang

Re: [DISCUSS] Vectorization Support in Flink

[Jeff Zhang]

Hi Liya,

Displaying image is not supported in apache mail list, you need to put it
elsewhere and post link in mail list.



Fan Liya  于2019年7月2日周二 下午3:40写道：

> Performance chart. FYI.
>
> Best,
> Liya Fan
> [image: image.png]
>
> On Tue, Jul 2, 2019 at 3:37 PM Fan Liya  wrote:
>
>> Hi all,
>>
>> We have opened an issue about vectorization in Flink (FLINK-13053
>> ). Would you please
>> give your valuable feedback? Thank you in advance.
>>
>> Vectorization is a popular technique in SQL engines today. Compared with
>> traditional row-based approach, it has some distinct advantages, for
>> example:
>>
>>
>>
>> 1)  Better use of CPU resources (e.g. SIMD)
>>
>> 2)  More compact memory layout
>>
>> 3)  More friendly to compressed data format.
>>
>>
>>
>> Currently, Flink is based on a row-based SQL engine for both stream and
>> batch workloads. To enjoy the above benefits, we want to bring
>> vectorization to Flink. This involves substantial changes to the existing
>> code base. Therefore, we give a plan to carry out such changes in small,
>> incremental steps, in order not to affect existing features. We want to
>> apply it to batch workload first. The details can be found in our proposal.
>>
>>
>>
>> For the past months, we have developed an initial implementation of the
>> above ideas. Initial performance evaluations on TPC-H benchmarks show that
>> substantial performance improvements can be obtained by vectorization (see
>> the figure below). More details can be found in our proposal.
>>
>>
>>
>> [image:
>> https://lh5.googleusercontent.com/hjXkXGImWOjaiB8zF0SKIMoItY6VCBm-BmJWWEXRo0ZPHdwLgKzCmIoNKef1YPCaAA7NXN6RvO-nwBBXBee52KeAtBjyIvh_NcAuChvW3BEtQuZGL5GPddqxL_iMV7HvEVCC6k-m]
>>
>>
>>
>> Special thanks to @Kurt Young’s team for all the kind help.
>>
>> Special thanks to @Piotr Nowojski for all the valuable feedback and help
>> suggestions.
>>
>>
>> Best,
>>
>> Liya Fan
>>
>

-- 
Best Regards

Jeff Zhang

Re: [DISCUSS] Vectorization Support in Flink

[Fan Liya]

Performance chart. FYI.

Best,
Liya Fan
[image: image.png]

On Tue, Jul 2, 2019 at 3:37 PM Fan Liya  wrote:

> Hi all,
>
> We have opened an issue about vectorization in Flink (FLINK-13053
> ). Would you please
> give your valuable feedback? Thank you in advance.
>
> Vectorization is a popular technique in SQL engines today. Compared with
> traditional row-based approach, it has some distinct advantages, for
> example:
>
>
>
> 1)  Better use of CPU resources (e.g. SIMD)
>
> 2)  More compact memory layout
>
> 3)  More friendly to compressed data format.
>
>
>
> Currently, Flink is based on a row-based SQL engine for both stream and
> batch workloads. To enjoy the above benefits, we want to bring
> vectorization to Flink. This involves substantial changes to the existing
> code base. Therefore, we give a plan to carry out such changes in small,
> incremental steps, in order not to affect existing features. We want to
> apply it to batch workload first. The details can be found in our proposal.
>
>
>
> For the past months, we have developed an initial implementation of the
> above ideas. Initial performance evaluations on TPC-H benchmarks show that
> substantial performance improvements can be obtained by vectorization (see
> the figure below). More details can be found in our proposal.
>
>
>
> [image:
> https://lh5.googleusercontent.com/hjXkXGImWOjaiB8zF0SKIMoItY6VCBm-BmJWWEXRo0ZPHdwLgKzCmIoNKef1YPCaAA7NXN6RvO-nwBBXBee52KeAtBjyIvh_NcAuChvW3BEtQuZGL5GPddqxL_iMV7HvEVCC6k-m]
>
>
>
> Special thanks to @Kurt Young’s team for all the kind help.
>
> Special thanks to @Piotr Nowojski for all the valuable feedback and help
> suggestions.
>
>
> Best,
>
> Liya Fan
>

[DISCUSS] Vectorization Support in Flink

[Fan Liya]

Hi all,

We have opened an issue about vectorization in Flink (FLINK-13053
). Would you please give
your valuable feedback? Thank you in advance.

Vectorization is a popular technique in SQL engines today. Compared with
traditional row-based approach, it has some distinct advantages, for
example:



1)  Better use of CPU resources (e.g. SIMD)

2)  More compact memory layout

3)  More friendly to compressed data format.



Currently, Flink is based on a row-based SQL engine for both stream and
batch workloads. To enjoy the above benefits, we want to bring
vectorization to Flink. This involves substantial changes to the existing
code base. Therefore, we give a plan to carry out such changes in small,
incremental steps, in order not to affect existing features. We want to
apply it to batch workload first. The details can be found in our proposal.



For the past months, we have developed an initial implementation of the
above ideas. Initial performance evaluations on TPC-H benchmarks show that
substantial performance improvements can be obtained by vectorization (see
the figure below). More details can be found in our proposal.



[image:
https://lh5.googleusercontent.com/hjXkXGImWOjaiB8zF0SKIMoItY6VCBm-BmJWWEXRo0ZPHdwLgKzCmIoNKef1YPCaAA7NXN6RvO-nwBBXBee52KeAtBjyIvh_NcAuChvW3BEtQuZGL5GPddqxL_iMV7HvEVCC6k-m]



Special thanks to @Kurt Young’s team for all the kind help.

Special thanks to @Piotr Nowojski for all the valuable feedback and help
suggestions.


Best,

Liya Fan

[jira] [Created] (FLINK-13053) Vectorization Support in Flink

[Liya Fan (JIRA)]

Liya Fan created FLINK-13053:


 Summary: Vectorization Support in Flink
 Key: FLINK-13053
 URL: https://issues.apache.org/jira/browse/FLINK-13053
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Runtime
Reporter: Liya Fan
Assignee: Liya Fan
 Attachments: image-2019-07-02-15-26-39-550.png

Vectorization is a popular technique in SQL engines today. Compared with 
traditional row-based approach, it has some distinct advantages, for example:

 
 * Better use of CPU resources (e.g. SIMD)
 * More compact memory layout
 * More friendly to compressed data format.

 

Currently, Flink is based on a row-based SQL engine for both stream and batch 
workloads. To enjoy the above benefits, we want to bring vectorization to 
Flink. This involves substantial changes to the existing code base. Therefore, 
we give a plan to carry out such changes in small, incremental steps, in order 
not to affect existing features. We want to apply it to batch workload first. 
The details can be found in our proposal.

 

For the past months, we have developed an initial implementation of the above 
ideas. Initial performance evaluations on TPC-H benchmarks show that 
substantial performance improvements can be obtained by vectorization (see the 
figure below). More details can be found in our proposal.

  !image-2019-07-02-15-26-39-550.png!

Special thanks to @Kurt Young’s team for all the kind help.

Special thanks to @Piotr Nowojski for all the valuable feedback and help 
suggestions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13052) Supporting multi-topic when using kafkaTableSourceSinkFactoryBase.createStreamTableSource

[chaiyongqiang (JIRA)]

chaiyongqiang created FLINK-13052:
-

 Summary: Supporting multi-topic when using 
kafkaTableSourceSinkFactoryBase.createStreamTableSource
 Key: FLINK-13052
 URL: https://issues.apache.org/jira/browse/FLINK-13052
 Project: Flink
  Issue Type: Improvement
Affects Versions: 1.8.0
Reporter: chaiyongqiang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-13051) Drop the non-selectable two-input StreamTask and Processor

[Haibo Sun (JIRA)]

Haibo Sun created FLINK-13051:
-

 Summary: Drop the non-selectable two-input StreamTask and Processor
 Key: FLINK-13051
 URL: https://issues.apache.org/jira/browse/FLINK-13051
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Task
Reporter: Haibo Sun
Assignee: Haibo Sun


After `StreamTwoInputSelectableProcessor` supports `CheckpointBarrierHandler`, 
we should  drop the non-selectable  two-input StreamTask and Processor.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)