[jira] [Created] (FLINK-12553) Fix a bug in SqlDateTimeUtils#parseToTimeMillis

[Liya Fan (JIRA)]

Liya Fan created FLINK-12553:


 Summary: Fix a bug in SqlDateTimeUtils#parseToTimeMillis
 Key: FLINK-12553
 URL: https://issues.apache.org/jira/browse/FLINK-12553
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Runtime
Reporter: Liya Fan
Assignee: Liya Fan


If parameter "1999-12-31 12:34:56.123" is used, it should return 123. But it 
returns 230 now.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12554) Create TableFactory for Hive connector

[Rui Li (JIRA)]

Rui Li created FLINK-12554:
--

 Summary: Create TableFactory for Hive connector
 Key: FLINK-12554
 URL: https://issues.apache.org/jira/browse/FLINK-12554
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive, Table SQL / Ecosystem
Reporter: Rui Li


Need a TableFactory to access Hive tables.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

questions about yarn container（taskmanager）memory allocation

[XiangWei Huang]

Hi all,
Currently i am running my flink application in yarn session mode and
using below commnad :
*bin/yarn-session.sh -d -s 3 -jm 1024 -tm 4096*
when taskmanager complete to started,i found the container launching
command is :
* bin/java -Xms2765m -Xmx2765m -XX:MaxDirectMemorySize=1331m ..*
from flink source code,i'v learned how direct memory size and heap memory
size calculated :
container total memory: 4096M
network memory fraction: 0.1
cut off memory fraction: 0.25
networkMemory = 4096*0.1 = 409.6
cut off memory = (4096 - 409.6) * 0.25 = 921.6
*directMemorySize* = 409.6 + 921.6 = 1331.2
*heapsize *= 4096 - 1331.2 = 2764.8

Below is the environment i'm using to run yarn and flink:
*  jdk version: 1.8*
* flink version: 1.6.1*
*OS: centos7*

there are two questions about memory allocation i want to ask:
1. Since the jdk version i an using is 1.8,is it necessary to consider
the *metaspace
memory *into calculation.According to the current way to calculate memory
size（without metaspace memory） may cause
container running beyond physical memory limit.
2. Is the native memory that rocksdb using part of direct memory（limit by
jvm parameter  *MaxDirectMemorySize*）,if not how do i control the size it
used.

Re: [DISCUSS] Clean up and reorganize the JIRA components

[Piotr Nowojski]

Hi,

> Concrete operator implementations will then go into the "API / DataStream"?
> (or "API / DataSet" or Table)
> Afaik, there were some ideas to share operator implementations between
> DataStream and Table

Yes & yes. I think for now we could keep the concrete operators implementations 
under API / DataStream and we can split them out once we have true use case for 
that. Unless this is confusing for someone, in that case we could split it now 
to API / DataStream Operators.

>> 2. I think we should add additional component for benchmarks and
>> benchmarking infrastructure. While this is more complicated topic (because
>> of the setup and how is it running), it should be on the same level as
>> correctness tests.
>> 
> 
> I'm not sure if it is a good idea to add a "Benchmarks" component into the
> Flink JIRA. Afaik, the benchmarks are managed from here?
> https://github.com/dataArtisans/flink/tree/benchmark-request 
> 

Not all of them, some of them are in apache/flink. And it might be a subject to 
change in the future. Ideally we should have benchmarking code in the same 
repository, if not for some licensing issues. Also if we ever implement full 
cluster benchmarks (not using JMH), they could also reside in the Flink 
repository. 

Regardless of that, does it matter where the benchmarks are? In my opinion the 
only thing that matters is that benchmarks are just another for of 
tests/verification, we have unit tests, integrations tests, end to end tests 
and also various level benchmarks. Why should those things be treated 
differently?

> Doesn't it make sense to track issues with GH issues there?
> Or asking more broadly, what types of issues would you see in that
> component?

Same kind of issues as for any other type of tests. For example:
- release blocker Jira issue that benchmarks are broken and are not testing 
anything (from time to time we have to fix something in the benchmarking setup 
and also it happened couple of times, that benchmarks have discovered some 
release blocker regressions in the Flink)
- Jira issue to fix some benchmark
- Jira issue to implement a missing benchmark
- …

Piotrek

> On 17 May 2019, at 14:41, Robert Metzger  wrote:
> 
> Hi,
> 
> 1. Renaming “Runtime / Operators” to “Runtime / Task” or something like
>> “Runtime / Processing”. “Runtime / Operators” was confusing me, since it
>> sounded like it covers concrete implementations of the operators, like
>> “WindowOperator” or various join implementations.
>> 
> 
> I'm fine with this renaming.
> Concrete operator implementations will then go into the "API / DataStream"?
> (or "API / DataSet" or Table)
> Afaik, there were some ideas to share operator implementations between
> DataStream and Table. If that's the case, we would have to find a good
> components for that as well.
> 
> 
>> 
>> 2. I think we should add additional component for benchmarks and
>> benchmarking infrastructure. While this is more complicated topic (because
>> of the setup and how is it running), it should be on the same level as
>> correctness tests.
>> 
> 
> I'm not sure if it is a good idea to add a "Benchmarks" component into the
> Flink JIRA. Afaik, the benchmarks are managed from here?
> https://github.com/dataArtisans/flink/tree/benchmark-request
> Doesn't it make sense to track issues with GH issues there?
> Or asking more broadly, what types of issues would you see in that
> component?
> 
> 
>> 
>> Piotrek
>> 
>>> On 20 Feb 2019, at 10:53, Robert Metzger  wrote:
>>> 
>>> Thanks a lot Timo!
>>> 
>>> I will start a vote Chesnay!
>>> 
>>> On Wed, Feb 20, 2019 at 10:11 AM Timo Walther 
>> wrote:
>>> 
 +1 for the vote. Btw I can help cleaning up the "Table API & SQL"
 component. It seems to be the biggest with 1229 Issues.
 
 Thanks,
 Timo
 
 Am 20.02.19 um 10:09 schrieb Chesnay Schepler:
> I would prefer if you'd start a vote with a new cleaned up proposal.
> 
> On 18.02.2019 15:23, Robert Metzger wrote:
>> I added "Runtime / Configuration" to the proposal:
>> 
 
>> https://cwiki.apache.org/confluence/display/FLINK/Proposal+for+new+JIRA+Components
>> 
>> 
>> Since this discussion has been open for 10 days, I assume we have
>> reached
>> consensus here. I will soon start renaming components.
>> 
>> On Wed, Feb 13, 2019 at 10:51 AM Chesnay Schepler >> 
>> wrote:
>> 
>>> The only parent I can think of is "Infrastructure", but I don't quite
>>> like it :/
>>> 
>>> +1 for "Runtime / Configuration"; this is too general to be placed in
>>> coordination imo.
>>> 
>>> On 12.02.2019 18:25, Robert Metzger wrote:
 Thanks a lot for your feedback Chesnay!
 
 re build/travis/release: Do you have a good idea for a common
 parent for
 "Build System", "Travis" and "Release System"?
 
 re legacy: Okay, I see your point. I will k

[jira] [Created] (FLINK-12555) Introduce an incapsulated metric group layout for shuffle API and deprecate old one

[Andrey Zagrebin (JIRA)]

Andrey Zagrebin created FLINK-12555:
---

 Summary: Introduce an incapsulated metric group layout for shuffle 
API and deprecate old one
 Key: FLINK-12555
 URL: https://issues.apache.org/jira/browse/FLINK-12555
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: Andrey Zagrebin
Assignee: Andrey Zagrebin
 Fix For: 1.9.0


At the moment, partition/gate create methods in NetworkEnvironment have a lot 
of metrics arguments to maintain original layout for metric groups. This 
approach is not quite incapsulated and clean for shuffle API. We can have just 
one parent group for shuffle metrics. The old layout can be still maintained in 
parallel and deprecated. At the moment we can do it with a couple of casts (if 
shuffle implementation is NetworkEnvironment) and adding an additional legacy 
metric registration which can be removed later.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12556) Extend some end-to-end tests to run with custom (input) File System implementation

[Alex (JIRA)]

Alex created FLINK-12556:


 Summary: Extend some end-to-end tests to run with custom (input) 
File System implementation
 Key: FLINK-12556
 URL: https://issues.apache.org/jira/browse/FLINK-12556
 Project: Flink
  Issue Type: Sub-task
  Components: Tests
Reporter: Alex
Assignee: Alex






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12557) Unify create table DDL with clause and connector descriptor keys

[Danny Chan (JIRA)]

Danny Chan created FLINK-12557:
--

 Summary: Unify create table DDL with clause and connector 
descriptor keys
 Key: FLINK-12557
 URL: https://issues.apache.org/jira/browse/FLINK-12557
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Affects Versions: 1.8.0
Reporter: Danny Chan
Assignee: Danny Chan
 Fix For: 1.9.0


The *with* option in table DDL defines the properties needed for specific 
connector to create TableSource/Sink. The properties structure for SqlClient 
config YAML is defined in [Improvements to the Unified SQL Connector 
API|https://docs.google.com/document/d/1Yaxp1UJUFW-peGLt8EIidwKIZEWrrA-pznWLuvaH39Y/edit#heading=h.41fd6rs7b3cf],
 in this design, the properties can be categorized into 4 parts:

 
 # Top level properties: name, type(source, sink), update-mode ...
 # Connector specific properties: connector.type, connector.path ...
 # Format properties: format.type, format.fields.1.name ...
 # Table schema properties: (can be omitted for DDL)

 

This properties structure is reasonable for YAML, but they are not that concise 
enough for developers.  So there also defines a tool class named 
[DescriptorProperties|https://github.com/apache/flink/blob/b3604f7bee7456b8533e9ea222a833a2624e36c2/flink-table/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java#L67]
 to reconstruct the data structure(like TableSchema) from the flat k-v strings.

 

So in order to reduce complexity and keep the KV consistency for DDL with 
properties and TableFactory properties, i proposed to simplify the DDL with 
properties keys as following (corresponding to above 4 categories):

 
 # Top level properties: keep same as that in the YAML e.g. type, update-mode
 # Connector specific properties: start with prefix named the connector type 
e.g. for kafka connector, the properties are defined as kafka.k1 = v1, kafka.k2 
= v2
 # Format properties: format.type simplified to format and the others keep the 
same e.g. format = 'json', format.line-delimiter = "\n"
 # Table schema properties: omitted.

Here is a demo of creat table DDL:
{code:java}
CREATE TABLE Kafka10SourceTable (
  intField INTEGER,
  stringField VARCHAR(128) COMMENT 'User IP address',
  longField BIGINT,
  rowTimeField TIMESTAMP,
  WATERMARK wm01 FOR  'longField' AS BOUNDED WITH DELAY '60' SECOND
)
COMMENT 'Kafka Source Table of topic user_ip_address'
WITH (
  type='kafka',
  property-version='1',
  version='0.10',
  kafka.topic='test-kafka-topic',
  kafka.startup-mode = 'latest-offset'
  kafka.specific-offset = 'offset'
  format='json'
  format.property-version = '1'
  format.version='1'
  format.derive-schema='true'

)
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12558) Yarn application can't stop when flink job finished

[lamber-ken (JIRA)]

lamber-ken created FLINK-12558:
--

 Summary: Yarn application can't stop when flink job finished
 Key: FLINK-12558
 URL: https://issues.apache.org/jira/browse/FLINK-12558
 Project: Flink
  Issue Type: Bug
  Components: Deployment / YARN, Runtime / REST
Affects Versions: 1.6.3
Reporter: lamber-ken
 Attachments: image-2019-05-20-18-47-12-497.png

I run a flink +SocketWindowWordCount+ job on yarn cluste mode, when I kill the 
socket, the flink job can't stopped. and I can't reproduct the bug again.

*Steps 1*
{code:java}
nc -lk 
{code}
*Steps 2*
{code:java}
bin/flink run -m yarn-cluster -yn 2 
examples/streaming/SocketWindowWordCount.jar --hostname 10.101.52.12 --port 
{code}
*Steps 3*
 cancel the above nc command

*Steps 4*
 every thing gone
   !image-2019-05-20-18-47-12-497.png!

 ** 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: questions about yarn container（taskmanager）memory allocation

[Xintong Song]

Hi XiangWei,

Thank you for the inputs. I agree with you that it is possible that
containers may use extra memory in 1.8. As for native memory, it is memory
used by JVM and other processes outside JVM. So it's not limited by
MaxDirectMemorySize.

The community is working on a refactoring plan about resource management of
Flink. AFAIK, the native memory is on the table, but I'm not sure whether
the metaspace memory is considered. I think we should create a jira issue
on this.

Thank you~

Xintong Song



On Mon, May 20, 2019 at 4:47 PM XiangWei Huang 
wrote:

> Hi all,
> Currently i am running my flink application in yarn session mode and
> using below commnad :
> *bin/yarn-session.sh -d -s 3 -jm 1024 -tm 4096*
> when taskmanager complete to started,i found the container launching
> command is :
> * bin/java -Xms2765m -Xmx2765m -XX:MaxDirectMemorySize=1331m ..*
> from flink source code,i'v learned how direct memory size and heap memory
> size calculated :
> container total memory: 4096M
> network memory fraction: 0.1
> cut off memory fraction: 0.25
> networkMemory = 4096*0.1 = 409.6
> cut off memory = (4096 - 409.6) * 0.25 = 921.6
> *directMemorySize* = 409.6 + 921.6 = 1331.2
> *heapsize *= 4096 - 1331.2 = 2764.8
>
> Below is the environment i'm using to run yarn and flink:
> *  jdk version: 1.8*
> * flink version: 1.6.1*
> *OS: centos7*
>
> there are two questions about memory allocation i want to ask:
> 1. Since the jdk version i an using is 1.8,is it necessary to consider the 
> *metaspace
> memory *into calculation.According to the current way to calculate memory
> size（without metaspace memory） may cause
> container running beyond physical memory limit.
> 2. Is the native memory that rocksdb using part of direct memory（limit by
> jvm parameter  *MaxDirectMemorySize*）,if not how do i control the size it
> used.
>

[jira] [Created] (FLINK-12559) Introduce metadata handlers on window aggregate

[godfrey he (JIRA)]

godfrey he created FLINK-12559:
--

 Summary: Introduce metadata handlers on window aggregate
 Key: FLINK-12559
 URL: https://issues.apache.org/jira/browse/FLINK-12559
 Project: Flink
  Issue Type: New Feature
  Components: Table SQL / Planner
Reporter: godfrey he
Assignee: godfrey he


In [FLINK-11822|https://issues.apache.org/jira/browse/FLINK-11822], we have 
introduced all Flink metadata handlers,  several {{RelNode}}s (e.g. window 
aggregate) have not be implemented. So this issue aims to introduce metadata 
handlers on window aggregate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Flink Forward Europe 2019 - Call for Presentations open until 17th May

[Oytun Tez]

Thanks for the update, Robert!

I am planning to prepare a use case content with how we use Flink at
MotaWord, focusing more on Flink as "application framework", rather than
confining our mindset to Flink as "stream processor", on non-Uber,
non-Alibaba scales. Hopefully over the weekend, I should be ready to submit
CFP.




---
Oytun Tez

*M O T A W O R D*
The World's Fastest Human Translation Platform.
oy...@motaword.com — www.motaword.com


On Fri, May 17, 2019 at 11:02 AM Robert Metzger  wrote:

> Hey all,
>
> Short update on the Flink Forward Call For Presentations: We've extended
> the submission deadline till May 31 ... so there's more time to finish the
> talk abstracts.
>
> Also, the organizers are now able to cover travel costs for speakers in
> cases where an employer can not cover them.
>
>
>
> On Fri, May 10, 2019 at 9:38 AM Fabian Hueske  wrote:
>
>> Hi Tim,
>>
>> Thanks for submitting a talk!
>> This sounds like a good and interesting use case to me.
>> Machine learning on streaming data is definitely a relevant and
>> interesting
>> topic for Flink Forward!
>>
>> Best,
>> Fabian
>>
>>
>> Am Mo., 6. Mai 2019 um 19:52 Uhr schrieb Tim Frey :
>>
>> > Hi All,
>> >
>> > Sounds interesting. I submitted a talk about using Flink for machine
>> > learning.
>> > However, I would also be happy to gain some community feedback if the
>> > topic is to the right interest of the community.
>> >
>> > In short, we use flink to train machine learning models and to then use
>> > the same models for predict then. Our goal was to determine if it is
>> > possible to predict crypto currency exchange rates by utilizing social
>> data
>> > from Twitter.
>> > I would talk about our experiences and describe how we leveraged online
>> > learning in conjunction with social data to determine if we are able to
>> > predict future currency exchange rates. I’ll point out the general
>> > architecture and describe the most interesting findings.
>> >
>> > Best
>> > Tim
>> >
>> > -Ursprüngliche Nachricht-
>> > Von: Robert Metzger 
>> > Gesendet: Montag, 6. Mai 2019 09:44
>> > An: Fabian Hueske 
>> > Cc: user ; dev ;
>> > commun...@flink.apache.org
>> > Betreff: Re: Flink Forward Europe 2019 - Call for Presentations open
>> until
>> > 17th May
>> >
>> > Thanks for announcing the Call for Presentations here!
>> >
>> > Since the deadline is approaching, I wanted to bump up this thread to
>> > remind everybody to submit talks!
>> > Please reach out to me or Fabian directly if you have any questions or
>> if
>> > you need any support!
>> >
>> >
>> >
>> > On Thu, Apr 11, 2019 at 3:47 PM Fabian Hueske 
>> wrote:
>> >
>> > > Hi all,
>> > >
>> > > Flink Forward Europe returns to Berlin on October 7-9th, 2019.
>> > > We are happy to announce that the Call for Presentations is open!
>> > >
>> > > Please submit a proposal if you'd like to present your Apache Flink
>> > > experience, best practices, new features, or use cases in front of an
>> > > international audience of highly skilled and enthusiastic Flink users
>> > > and committers.
>> > >
>> > > Flink Forward will run tracks for the following topics:
>> > > * Use Case
>> > > * Operations
>> > > * Technology Deep Dive
>> > > * Ecosystem
>> > > * Research
>> > >
>> > > For the first time, we'll also have a Community track.
>> > >
>> > > Please find the submission form at
>> > > https://berlin-2019.flink-forward.org/call-for-presentations
>> > >
>> > > The deadline for submissions is May 17th, 11:59pm (CEST).
>> > >
>> > > Best regards,
>> > > Fabian
>> > > (PC Chair for Flink Forward Berlin 2019)
>> > >
>> >
>> >
>>
>

[jira] [Created] (FLINK-12560) Documentation language build flags

[Seth Wiesman (JIRA)]

Seth Wiesman created FLINK-12560:


 Summary: Documentation language build flags
 Key: FLINK-12560
 URL: https://issues.apache.org/jira/browse/FLINK-12560
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation
Reporter: Seth Wiesman
Assignee: Seth Wiesman


Most documentation writers are only ever focused on one language at a time. 
Adding language-specific build flags can dramatically reduce render time during 
local development. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12561) Unable to insert into cassandra table using FileWordCount example using Remote Cassandra Host with credentials

[Ramesh Srinivasalu (JIRA)]

Ramesh Srinivasalu created FLINK-12561:
--

 Summary: Unable to insert into cassandra table using FileWordCount 
example using Remote Cassandra Host with credentials
 Key: FLINK-12561
 URL: https://issues.apache.org/jira/browse/FLINK-12561
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Cassandra
Affects Versions: 1.8.0, 1.7.2
 Environment: DEV/QA/Prod
Reporter: Ramesh Srinivasalu
 Fix For: 1.7.3


eculativeRetry=99.0PERCENTILE,droppedColumns={},triggers=[],isDense=false]
09:33:55,072 INFO org.apache.cassandra.db.ColumnFamilyStore - Initializing 
system_auth.role_members
09:33:55,317 INFO org.apache.cassandra.transport.Server - Netty using native 
Epoll event loop
09:33:55,356 INFO org.apache.cassandra.transport.Server - Using Netty Version: 
[netty-buffer=netty-buffer-4.0.33.Final.69b5aef, 
netty-codec=netty-codec-4.0.33.Final.69b5aef, 
netty-codec-http=netty-codec-http-4.0.27.Final.054e7c5, 
netty-codec-socks=netty-codec-socks-4.0.27.Final.054e7c5, 
netty-common=netty-common-4.0.33.Final.69b5aef, 
netty-handler=netty-handler-4.0.33.Final.69b5aef, 
netty-tcnative=netty-tcnative-1.1.32.Fork1.9e735d2, 
netty-transport=netty-transport-4.0.33.Final.69b5aef, 
netty-transport-native-epoll=netty-transport-native-epoll-4.0.27.Final.054e7c5, 
netty-transport-rxtx=netty-transport-rxtx-4.0.27.Final.054e7c5, 
netty-transport-sctp=netty-transport-sctp-4.0.27.Final.054e7c5, 
netty-transport-udt=netty-transport-udt-4.0.27.Final.054e7c5]
09:33:55,356 INFO org.apache.cassandra.transport.Server - Starting listening 
for CQL clients on /127.0.0.1:9042...
09:33:55,389 INFO org.apache.cassandra.service.CassandraDaemon - Not starting 
RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool 
(enablethrift) to start it
09:33:55,389 INFO 
org.apache.flink.connectors.cassandra.EmbeddedCassandraService - Embedded 
Cassasndra service started
09:33:55,389 INFO 
org.apache.flink.connectors.cassandra.datamodel.DataServiceFacade - Bringing up 
Embedded Cassandra service ... DONE
09:33:55,550 INFO com.datastax.driver.core.NettyUtil - Did not find Netty's 
native epoll transport in the classpath, defaulting to NIO.
[WARNING]
com.datastax.driver.core.exceptions.AuthenticationException: Authentication 
error on host testremoteserver.com/192.121.22.1:9042: Host 
testremoteserver.com/192.121.22.1:9042 requires authentication, but no 
authenticator found in Cluster configuration
 at 
com.datastax.driver.core.AuthProvider$1.newAuthenticator(AuthProvider.java:40)
 at com.datastax.driver.core.Connection$5.apply(Connection.java:250)
 at com.datastax.driver.core.Connection$5.apply(Connection.java:234)
 at 
org.apache.flink.cassandra.shaded.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:906)
 at 
org.apache.flink.cassandra.shaded.com.google.common.util.concurrent.Futures$1$1.run(Futures.java:635)
 at 
org.apache.flink.cassandra.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
 at 
org.apache.flink.cassandra.shaded.com.google.common.util.concurrent.Futures$1.run(Futures.java:632)
 at 
org.apache.flink.cassandra.shaded.com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
 at 
org.apache.flink.cassandra.shaded.com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
 at 
org.apache.flink.cassandra.shaded.com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
 at 
org.apache.flink.cassandra.shaded.com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:185)
 at com.datastax.driver.core.Connection$Future.onSet(Connection.java:1174)
 at 
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1005)
 at 
com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:928)
 at 
com.datastax.shaded.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
 at 
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
 at 
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
 at 
com.datastax.shaded.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
 at 
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
 at 
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:304)
 at 
com.datastax.shaded.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
 at 
com.datastax.shaded.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:318)
 at 
com.datasta

[ANNOUNCE] Seattle Flink Meetup at AWS on May 30

[Bowen Li]

Hi Greater Seattle folks!

We are hosting our next meetup with AWS Kinesis Analytics team on May 30
next Thursday in downtown Seattle.

We feature two talks this time:

   1. *"AWS Kinesis Analytics: running Flink serverless in multi-tenant
   environment"* by Kinesis Analytics team on:
  - How to run Apache Flink applications using Amazon Kinesis Data
  Analytics service
  - Challenges they faced building a multi-tenant serverless Flink
  service
   2. *"How to contribute to Flink: a hands-on guide to start your Apache
   open-source journey"* by me to get developers onboard given the
   considerably rising interest of contributing to Flink from local tech
   community

Please find details and RSVP at
https://www.meetup.com/seattle-flink/events/260865206/  See you next week!

Cheers,
Bowen

[DISCUSS] FLIP to Restructure, Update & Rework Apache Flink's Documentation

[Konstantin Knauf]

Hi all,

following our unsuccessful application to Season of Docs Fabian, Seth,
Stephan and myself have drafted a FLIP for reworking our documentation and
implementing some (or most) of the changes, we would have liked to tackle
in SoD.

The overarching goals of the FLIP are:

* Top-level structure with clear separation of concerns

* Expand and restructure existing Concepts section

* Align Application Development with current state and Flink roadmap

* Improve documentation on Deployment & Operations

** *Improve getting-started experience

* Use of a common terminology

* Add a documentation style guide


It is quite an ambitions endeavor, but we can tackle these changes in
multiple phases and we will benefit from it even if the whole FLIP takes
some time to be implemented.

The whole description of the FLIP (proposed documentation structure &
sections, changes, implementation plan) is available on Google Docs [1] for
now for simplicity. I will move it to Confluence once we have reached
consensus.

Please let us know what you think and I am looking forward to your
feedback, opinions and comments on the proposal.

Cheers,

Konstantin

[1]
https://docs.google.com/document/d/1pPM4vTWUUiJb73pd8OqHA1EWVl9cv6kXoCB5F7_J6gM/edit?usp=sharing


-- 

Konstantin Knauf | Solutions Architect

+49 160 91394525


Planned Absences: 20. - 21.06.2019




Follow us @VervericaData

--

Join Flink Forward  - The Apache Flink
Conference

Stream Processing | Event Driven | Real Time

--

Data Artisans GmbH | Invalidenstrasse 115, 10115 Berlin, Germany

--
Data Artisans GmbH
Registered at Amtsgericht Charlottenburg: HRB 158244 B
Managing Directors: Dr. Kostas Tzoumas, Dr. Stephan Ewen

[jira] [Created] (FLINK-12562) Improve code in ExpandColumnFunctionsRule

[Hequn Cheng (JIRA)]

Hequn Cheng created FLINK-12562:
---

 Summary: Improve code in ExpandColumnFunctionsRule
 Key: FLINK-12562
 URL: https://issues.apache.org/jira/browse/FLINK-12562
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API
Reporter: Hequn Cheng
Assignee: Hequn Cheng


Enable most IntelliJ warnings and resolve these warnings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Fwd: [DISCUSS] Flink SQL DDL Design

[Yuzhao Chen]

Thanks for all you discussions Timo, Jark Wu, Lin Li, Fabian

I have fired a new design doc in [1] which reorganized from the MVP doc[2] and 
the initial doc[3] that both proposed by Shuyi Chen.

The main diff is that i extend the create table DDL to support complex sql type 
(array, map and struct), also i have added 3 points that need to reach a 
consensus：

1. Where we should put the parse code ? (blocking)
2. How to integrate the sql parse logic with TableEnvironment and SqlClient ? 
(blocking)
3. What format should should the table ddl with clause be like ? (non-blocking)

For points 1 and 2, i have put some solutions in [1], we need a duscussion 
because they are blocking, for 3 i have fired a new JIRA issue to track[4].

[1] 
https://docs.google.com/document/d/1OmVyuPk9ibGUC-CnPHbXvCg_fdG1TeC3lXSnqcUEYmM/edit?usp=sharing
[2] 
https://docs.google.com/document/d/1ug1-aVBSCxZQk58kR-yaK2ETCgL3zg0eDUVGCnW2V9E/edit
[3] 
https://docs.google.com/document/d/1TTP-GCC8wSsibJaSUyFZ_5NBAHYEB1FVmPpP7RgDGBA/edit
[4] https://issues.apache.org/jira/browse/FLINK-12557

Best,
Danny Chan
>
> Thanks for all you discussions Timo, Jark Wu, Lin Li, Fabian
>
> I have fired a new design doc in [1] which is reorganized from the MVP doc[2] 
> and the initial doc[3] that both proposed by Shuyi Chen.
>
> The main diff is that i extend the create table DDL to support complex sql 
> type (array, map and struct), also i have added 3 points that need to reach a 
> consensus：
>
> 1. Where we should put the parse code ? (blocking)
> 2. How to integrate the sql parse logic with TableEnvironment and SqlClient ? 
> (blocking)
> 3. What format should the table ddl with clause be like ? (non-blocking)
>
> For points 1 and 2, i have put some solutions in [1] (highlight title as 
> red), we need a duscussion because they are blocking, for 3 i have fired a 
> new JIRA issue to track[4].
>
> [1] 
> https://docs.google.com/document/d/1OmVyuPk9ibGUC-CnPHbXvCg_fdG1TeC3lXSnqcUEYmM/edit?usp=sharing
> [2] 
> https://docs.google.com/document/d/1ug1-aVBSCxZQk58kR-yaK2ETCgL3zg0eDUVGCnW2V9E/edit
> [3] 
> https://docs.google.com/document/d/1TTP-GCC8wSsibJaSUyFZ_5NBAHYEB1FVmPpP7RgDGBA/edit
> [4] https://issues.apache.org/jira/browse/FLINK-12557

Contributor

[何泽雄]

Hi,

I want to contribute to Apache Flink.
Would you please give me the contributor permission?
My JIRA ID is zxhe.

My JIRA Fullname is ZeXiong.

[jira] [Created] (FLINK-12563) Introduce vector data format in blink

[Jingsong Lee (JIRA)]

Jingsong Lee created FLINK-12563:


 Summary: Introduce vector data format in blink
 Key: FLINK-12563
 URL: https://issues.apache.org/jira/browse/FLINK-12563
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / Runtime
Reporter: Jingsong Lee
Assignee: Jingsong Lee


Introduce VectorizedColumnBatch: A VectorizedColumnBatch is a set of rows, 
organized with each column as a vector.

Introduce ColumnVector with subclasses: IntColumnVector, LongColumnVector and 
etc..

Introduce ColumnarRow: wrap VectorizedColumnBatch as a BaseRow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12564) Remove getBufferProvider method from ResultPartitionWriter interface

[zhijiang (JIRA)]

zhijiang created FLINK-12564:


 Summary: Remove getBufferProvider method from 
ResultPartitionWriter interface
 Key: FLINK-12564
 URL: https://issues.apache.org/jira/browse/FLINK-12564
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Network
Reporter: zhijiang
Assignee: zhijiang


Currently `ResultPartitionWriter#getBufferProvider` is used for requesting 
`BufferBuilder` in `RecordWriter`, then the `BufferConsumer` created from 
`BufferBuilder` is added into `ResultPartitionWriter` via `addBufferConsumer` 
method.

We could merge these two methods in `ResultPartitionWriter` in order not to 
expose `getBufferProvider`. `ResultPartitionWriter` could internally request 
`BufferBuilder` and  add the created `BufferConsumer` into one sub partition, 
then return the `BufferBuilder` for `RecordWriter` writing serialized data.

Since we also change the `ResultPartitionWriter#addBufferConsumer` to 
`ResultPartitionWriter#requestBufferBuilder`, then another new method 
`ResultPartitionWriter#broadcastEvents` should be introduced for handling the 
case of events.

In future it might worth further abstracting the `ResultPartitionWriter` to be 
not only related to  `BufferBuilder`. We could provide `writeRecord(int 
targetIndex)` to replace `requestBufferBuilder`, then the serialization process 
could be done inside specific `ResultPartitionWriter` instance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Job recovery with task manager restart

[Thomas Weise]

Hi Till,

Thanks for the background.

It seems that we cannot always rely on the user code to not cause leaks and
that can wreak havoc even when everything in Flink works as expected (user
code managed threads may not terminate, class loader GC may not work due to
references in the parent class loader, direct memory not deallocated and so
on).

I think it would be nice to have the option to terminate the TMs. Maybe it
can be solved by making TMs exit similar to task cancellation [1] and let
the user configure after how many job starts this should occur?

For the session mode case, it would affect other jobs that share the TMs,
but that's not avoidable. A resource leak would eventually compromise the
entire cluster.

Thanks,
Thomas

[1] https://issues.apache.org/jira/browse/FLINK-4715


On Fri, May 17, 2019 at 12:50 AM Till Rohrmann  wrote:

> Hi Thomas and Hwanju,
>
> thanks for starting this discussion. As far as I know, there has not been a
> lot of prior discussion or related work with respect to this topic.
> Somewhat related is the discussion about job isolation in a session cluster
> [1].
>
> Whenever there is resource leak on Flink's side, we should try to fix it.
> However, I see that user code might be out of our control and for this such
> a feature might be useful.
>
> How would such a feature behave in detail? Would you like that all TMs
> which executed a task of a restarting job are being restarted? What happens
> if these TMs execute other jobs (in session mode)? Should the decision to
> restart happen locally on the TM (number of tolerated task failures) or
> centralized on the RM where different resolution strategies could run?
>
> To mitigate the problem of class loader leaks and GC pressure, we thought
> about binding the class loader to a slot. As long as a JM owns this slot
> (so also across job restarts), the user code class loader should then be
> reusable.
>
> [1] https://issues.apache.org/jira/browse/FLINK-9662
>
> Cheers,
> Till
>
> On Thu, May 16, 2019 at 7:28 PM Kim, Hwanju 
> wrote:
>
> > Hi Thomas,
> >
> > I have a sort of question regarding the class loader issue, as it seems
> > interesting.
> > My understanding is that at least user class loader is unregistered and
> > re-registered (from/to library cache on TM) across task restart. If I
> > understand it correctly, unregistered one should be GCed as long as no
> > object loaded by the user class loader is lingering across task restart.
> > Indeed, however, there is no guarantee that UDF cleans up everything on
> > close(). I've seen that some libraries used in UDF let a daemon thread
> > outlive a task, so any object loaded by unregistered user class loader in
> > the thread causes the class loader to be leaked (also daemon threads are
> > also leaked since those keep being spawned, albeit singleton, due to
> newly
> > registered class loader). If a job keeps restarting, this behavior leads
> to
> > metaspace OOM or out of threads/OOM. So, my question is if the memory
> issue
> > you've seen is due to whether Flink issue or the side-effect that UDF
> > causes (as I described). Second question is if there's anything else
> other
> > than class loader issue. Of course, I also wonder if any prior discussion
> > is going on.
> >
> > Best,
> > Hwanju
> >
> > On 5/16/19, 8:01 AM, "Thomas Weise"  wrote:
> >
> > Hi,
> >
> > When a job fails and is recovered by Flink, task manager JVMs are
> > reused.
> > That can cause problems when the failed job wasn't cleaned up
> > properly, for
> > example leaving behind the user class loader. This would manifest in
> > rising
> > base for memory usage, leading to a death spiral.
> >
> > It would be good to provide an option that guarantees isolation, by
> > restarting the task manager processes. Managing the processes would
> > depend
> > on how Flink is deployed, but the recovery sequence would need to
> > provide a
> > hook for the user.
> >
> > Has there been prior discussion or related work?
> >
> > Thanks,
> > Thomas
> >
> >
> >
>

[jira] [Created] (FLINK-12565) PythonStreamBinderTest#testProgram test failer

[vinoyang (JIRA)]

vinoyang created FLINK-12565:


 Summary: PythonStreamBinderTest#testProgram test failer
 Key: FLINK-12565
 URL: https://issues.apache.org/jira/browse/FLINK-12565
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Reporter: vinoyang


{code:java}
04:15:51.246 [INFO] Running 
org.apache.flink.streaming.python.api.PythonStreamBinderTest
Working directory: 
/tmp/flink_streaming_plan_7159fb6f-a167-425b-95ad-466d37f9ded4
Excluded tests: ['test_kafka09']

Submitting job ... 'test_split_select'
04:16:07.167 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time 
elapsed: 15.917 s <<< FAILURE! - in 
org.apache.flink.streaming.python.api.PythonStreamBinderTest
04:16:07.167 [ERROR] 
testProgram(org.apache.flink.streaming.python.api.PythonStreamBinderTest)  Time 
elapsed: 7.884 s  <<< ERROR!
org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
at 
org.apache.flink.streaming.python.api.PythonStreamBinderTest.testProgram(PythonStreamBinderTest.java:81)
Caused by: org.apache.flink.util.FlinkException: 
Python function failed: Traceback (most recent call last):
  File 
"/tmp/flink_streaming_plan_7159fb6f-a167-425b-95ad-466d37f9ded4/test_split_select.py",
 line 33, in flatMap
collector.collect((1, value))
SystemError: __getattribute__ not found on type null. See 
http://bugs.jython.org/issue2487 for details.
{code}
log details : [https://api.travis-ci.org/v3/job/533608290/log.txt]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12566) Remove row interval type

[Timo Walther (JIRA)]

Timo Walther created FLINK-12566:


 Summary: Remove row interval type
 Key: FLINK-12566
 URL: https://issues.apache.org/jira/browse/FLINK-12566
 Project: Flink
  Issue Type: Sub-task
  Components: Table SQL / API
Reporter: Timo Walther
Assignee: Timo Walther


The row interval type just adds additional complexity and prevents SQL queries 
from supporting count windows. A regular {{BIGINT}} type is sufficient to 
represent a count.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12567) Rework DescriptorProperties to adapt unified DDL with clause and Descriptor key value pairs

[Danny Chan (JIRA)]

Danny Chan created FLINK-12567:
--

 Summary: Rework DescriptorProperties to adapt unified DDL with 
clause and Descriptor key value pairs
 Key: FLINK-12567
 URL: https://issues.apache.org/jira/browse/FLINK-12567
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.8.0
Reporter: Danny Chan
Assignee: Danny Chan
 Fix For: 1.9.0


After introduce DDLs, we need to unify the k-v properties format for DDL with 
clause and Descriptor API, this tool class also need to rework.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-12568) Implement TableSink and OutputFormat to write Hive tables

[Rui Li (JIRA)]

Rui Li created FLINK-12568:
--

 Summary: Implement TableSink and OutputFormat to write Hive tables
 Key: FLINK-12568
 URL: https://issues.apache.org/jira/browse/FLINK-12568
 Project: Flink
  Issue Type: Sub-task
Reporter: Rui Li
Assignee: Rui Li






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)