Re: [DISCUSS] FLIP-19: Improved BLOB storage architecture

[Biao Liu]

I have the same concern with Chesnay Schepler. AFIK Flink does not support
DC as well as Mapreduce and Spark. We only support DC in DataSet API. And
DC in flink do not support local files. Is this a good change to refactor
DC too?

I have another concern, currently BLOB server has some conflicts with
FLIP-6 architecture. We start JM while submitting job instead of starting
it before in FLIP-6. If BLOB server is embedded in JM we can not upload
jars and files before JM started. But the fact is that we need jars
uploaded before starting JM. Correct me is I was wrong.
To solve this problem we can separate submitting job into different stage.
Or we can separate BLOB server as a independent component parallel with RM.

Maybe we can think more about these in FLIP-19, what do you think? @Nico

[jira] [Created] (FLINK-6925) Add CONCAT/CONCAT_WS supported in SQL

[sunjincheng (JIRA)]

sunjincheng created FLINK-6925:
--

 Summary: Add CONCAT/CONCAT_WS supported in SQL
 Key: FLINK-6925
 URL: https://issues.apache.org/jira/browse/FLINK-6925
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng


CONCAT(str1,str2,...)Returns the string that results from concatenating the 
arguments. May have one or more arguments. If all arguments are nonbinary 
strings, the result is a nonbinary string. If the arguments include any binary 
strings, the result is a binary string. A numeric argument is converted to its 
equivalent nonbinary string form.

CONCAT() returns NULL if any argument is NULL.

* Syntax:
CONCAT(str1,str2,...) 

* Arguments
** str1,str2,... -

* Return Types
  string

* Example:
  CONCAT('F', 'lin', 'k') -> 'Flink'
  CONCAT('M', NULL, 'L') -> NULL
  CONCAT(14.3) -> '14.3'

* See more:
** [MySQL| 
https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat]


CONCAT_WS() stands for Concatenate With Separator and is a special form of 
CONCAT(). The first argument is the separator for the rest of the arguments. 
The separator is added between the strings to be concatenated. The separator 
can be a string, as can the rest of the arguments. If the separator is NULL, 
the result is NULL.

* Syntax:
CONCAT_WS(separator,str1,str2,...)

* Arguments
** separator -
** str1,str2,... -

* Return Types
  string

* Example:
  CONCAT_WS(',','First name','Second name','Last Name') -> 'First name,Second 
name,Last Name'

* See more:
** [MySQL| 
https://dev.mysql.com/doc/refman/5.7/en/string-functions.html#function_concat-ws]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6926) Add MD5/SHA1/SHA2 supported in SQL

[sunjincheng (JIRA)]

sunjincheng created FLINK-6926:
--

 Summary: Add MD5/SHA1/SHA2 supported in SQL
 Key: FLINK-6926
 URL: https://issues.apache.org/jira/browse/FLINK-6926
 Project: Flink
  Issue Type: Sub-task
  Components: Table API & SQL
Affects Versions: 1.4.0
Reporter: sunjincheng


MD5(str)Calculates an MD5 128-bit checksum for the string. The value is 
returned as a string of 32 hexadecimal digits, or NULL if the argument was 
NULL. The return value can, for example, be used as a hash key. See the notes 
at the beginning of this section about storing hash values efficiently.

The return value is a nonbinary string in the connection character set.

* Example:
 MD5('testing') - 'ae2b1fca515949e5d54fb22b8ed95575'

* See more:
** [MySQL| 
https://dev.mysql.com/doc/refman/5.7/en/encryption-functions.html#function_sha1]


SHA1(str), SHA(str)Calculates an SHA-1 160-bit checksum for the string, as 
described in RFC 3174 (Secure Hash Algorithm). The value is returned as a 
string of 40 hexadecimal digits, or NULL if the argument was NULL. One of the 
possible uses for this function is as a hash key. See the notes at the 
beginning of this section about storing hash values efficiently. You can also 
use SHA1() as a cryptographic function for storing passwords. SHA() is 
synonymous with SHA1().

The return value is a nonbinary string in the connection character set.

* Example:
  SHA1('abc') -> 'a9993e364706816aba3e25717850c26c9cd0d89d'



SHA2(str, hash_length)Calculates the SHA-2 family of hash functions (SHA-224, 
SHA-256, SHA-384, and SHA-512). The first argument is the cleartext string to 
be hashed. The second argument indicates the desired bit length of the result, 
which must have a value of 224, 256, 384, 512, or 0 (which is equivalent to 
256). If either argument is NULL or the hash length is not one of the permitted 
values, the return value is NULL. Otherwise, the function result is a hash 
value containing the desired number of bits. See the notes at the beginning of 
this section about storing hash values efficiently.

The return value is a nonbinary string in the connection character set.

* Example:
SHA2('abc', 224) -> '23097d223405d8228642a477bda255b32aadbce4bda0b3f7e36c9da7'
* See more:
** [MySQL| 
https://dev.mysql.com/doc/refman/5.7/en/encryption-functions.html#function_sha2]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6927) Support pattern group in CEP

[Dian Fu (JIRA)]

Dian Fu created FLINK-6927:
--

 Summary: Support pattern group in CEP
 Key: FLINK-6927
 URL: https://issues.apache.org/jira/browse/FLINK-6927
 Project: Flink
  Issue Type: Bug
  Components: CEP
Reporter: Dian Fu
Assignee: Dian Fu


We should add support for pattern group. This would enrich the set of supported 
patterns. For example, users can write patterns like this with this feature 
available:
{code}
 A --> (B --> C.times(3)).optional() --> D
{code}
or
{code}
A --> (B --> C).times(3) --> D
{code}




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6928) Kafka source: default topic needs to exist

[Erik van Oosten (JIRA)]

Erik van Oosten created FLINK-6928:
--

 Summary: Kafka source: default topic needs to exist
 Key: FLINK-6928
 URL: https://issues.apache.org/jira/browse/FLINK-6928
 Project: Flink
  Issue Type: Bug
  Components: Kafka Connector
Affects Versions: 1.2.1, 1.3.0
Reporter: Erik van Oosten


When using a Kafka source, the defaultTopic needs to exist even when it is 
never used. It would be nice if fetching partition information for the default 
topic would be delayed until the moment a topic is actually used.

Cause: {{FlinkKafkaProducerBase.open}} fetches partition information for the 
default topic.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Tracking API

[Aljoscha Krettek]

Hi,

Would this be what you’re looking for: 
https://stackoverflow.com/questions/1069066/get-current-stack-trace-in-java 
 ?

Best,
Aljoscha

> On 14. Jun 2017, at 12:46, Ameet BD  wrote:
> 
> Hello,
> 
> I am trying to print which Flink API (e.g Complex Event Processing or
> Streaming SQL) was called that resulted in a call down the function stack
> in task handling code. Please tell me how it could be done using existing
> code base (1.3.0).
> Regards,
> Ameet

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

[Till Rohrmann]

@Jark: You should now have the permissions to create pages in the Flink
wiki.

Cheers,
Till

On Thu, Jun 15, 2017 at 5:11 AM, Jark Wu  wrote:

> Hi Till,
>
> Could you grant me the edit permission of Flink WIKI? My id is imjark.
>
> Thanks,
> Jark Wu
>
> 2017-06-15 0:07 GMT+08:00 Till Rohrmann :
>
> > I think that the integration of SQL and CEP would make a good FLIP.
> >
> > Cheers,
> > Till
> >
> > On Wed, Jun 14, 2017 at 2:40 PM, Jark Wu  wrote:
> >
> > > Hi,
> > >
> > > Do you think whether we should create a FLIP for this proposal to track
> > > progress?
> > >
> > > Regards,
> > > Jark
> > >
> > > 2017-06-13 16:59 GMT+08:00 Dian Fu :
> > >
> > > > Hi Fabian,
> > > >
> > > > Thanks a lot. Agree that we can start working by adding the missing
> > > > features of the CEP library.
> > > >
> > > > Best regards,
> > > > Dian
> > > >
> > > > On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > @Dian Fu: I gave you contributor permissions. :-)
> > > > >
> > > > > I don't think we have to wait for Calcite 1.13 to start working on
> > the
> > > > > missing features of the CEP library and extending the prototype.
> > > > > We might want to wait with the integration into flink-table until
> > > Calcite
> > > > > 1.13 is out and we updated the dependency though.
> > > > >
> > > > > Best, Fabian
> > > > >
> > > > > 2017-06-13 9:45 GMT+02:00 jincheng sun :
> > > > >
> > > > > > Hi Jark, Dian,
> > > > > >
> > > > > > Thanks for bring up this discuss and share the prototype.
> > > > > >
> > > > > > +1 to push this great feature forward!
> > > > > >
> > > > > > Cheers,
> > > > > > SunJincheng
> > > > > >
> > > > > > 2017-06-13 15:34 GMT+08:00 Jark Wu :
> > > > > >
> > > > > > > Thank you Yueting for pointing out the mistake in the
> prototype.
> > I
> > > > > > > accidentally introduced it when merge code.
> > > > > > >
> > > > > > > I'm so glad to see so many people are interested in the
> feature.
> > > > Let's
> > > > > > work
> > > > > > > out together to push it forward!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jark Wu
> > > > > > >
> > > > > > >
> > > > > > > 2017-06-13 15:27 GMT+08:00 Liangfei Su :
> > > > > > >
> > > > > > > > +1 for the feature. Myself was a user of Siddhi, this is
> pretty
> > > > user
> > > > > > > > friendly feature to provide to user.
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <
> > dian0511...@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yueting & Dawid Wysakowicz,
> > > > > > > > >
> > > > > > > > > Very glad that you're interested in this feature and you're
> > > > > > definitely
> > > > > > > > > welcome to join this work and also anyone else.:)
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Dian
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > > > > > wysakowicz.da...@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Integrating SQL with CEP seems like a really nice idea.
> > > > > > > Unfortunately I
> > > > > > > > > had
> > > > > > > > > > time just for a brief look at the design doc, but it
> looks
> > > > really
> > > > > > > cool
> > > > > > > > > and
> > > > > > > > > > thorough. Also will have a second run tomorrow and will
> try
> > > to
> > > > > > > provide
> > > > > > > > > more
> > > > > > > > > > comments. Anyway will be glad to help pushing the
> > initiative
> > > > > > forward.
> > > > > > > > > >
> > > > > > > > > > Z pozdrowieniami! / Cheers!
> > > > > > > > > >
> > > > > > > > > > Dawid Wysakowicz
> > > > > > > > > >
> > > > > > > > > > *Data/Software Engineer*
> > > > > > > > > >
> > > > > > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > > > > > >
> > > > > > > > > > 
> > > > > > > > > >
> > > > > > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <
> > yestinm...@gmail.com
> > > >:
> > > > > > > > > >
> > > > > > > > > > > Hi Dian & Jark,
> > > > > > > > > > >
> > > > > > > > > > > I checked out your prototype code, but it didn't pass
> the
> > > > > > CEPITCase
> > > > > > > > > test
> > > > > > > > > > in
> > > > > > > > > > > the flink-table component.
> > > > > > > > > > > It turns out that in the `MatchCodeGenerator.scala`
> file,
> > > > line
> > > > > 74
> > > > > > > > > should
> > > > > > > > > > > use `${classOf[IterativeCondition.
> > > > > Context[_]].getCanonicalName}`
> > > > > > > > > instead
> > > > > > > > > > > of
> > > > > > > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > > > > > > >
> > > > > > > > > > > I've also read your desgin document and it looks fine
> to
> > > me.
> > > > > > > > Actually,
> > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > > working on the same thing recently, I think maybe we
> can
> > > work
> > > > > > > > together
> > > > > > > > > to
> > > > > > > > > > > push this forward.
> > > > > > > > > > >
> > > > > > > > > > > Tha

Re: [DISCUSS] FLIP-19: Improved BLOB storage architecture

[Till Rohrmann]

Hi Biao,

you're right that the BlobServer won't live in the JM in FLIP-6. Instead it
will either be part of the RM or the dispatcher component depending on the
actual implementation. The requirements for the BlobServer should, however,
be the same.

Concerning the question about Flink's distributed cache, I think this is an
orthogonal topic. I think that we should be able to piggy-back on the
BlobServer once we have refactored it. This should simplify the DC's
implementation a bit.

Cheers,
Till

On Thu, Jun 15, 2017 at 9:16 AM, Biao Liu  wrote:

> I have the same concern with Chesnay Schepler. AFIK Flink does not support
> DC as well as Mapreduce and Spark. We only support DC in DataSet API. And
> DC in flink do not support local files. Is this a good change to refactor
> DC too?
>
> I have another concern, currently BLOB server has some conflicts with
> FLIP-6 architecture. We start JM while submitting job instead of starting
> it before in FLIP-6. If BLOB server is embedded in JM we can not upload
> jars and files before JM started. But the fact is that we need jars
> uploaded before starting JM. Correct me is I was wrong.
> To solve this problem we can separate submitting job into different stage.
> Or we can separate BLOB server as a independent component parallel with RM.
>
> Maybe we can think more about these in FLIP-19, what do you think? @Nico
>

[jira] [Created] (FLINK-6929) Add documentation for Table API OVER windows

[Fabian Hueske (JIRA)]

Fabian Hueske created FLINK-6929:


 Summary: Add documentation for Table API OVER windows
 Key: FLINK-6929
 URL: https://issues.apache.org/jira/browse/FLINK-6929
 Project: Flink
  Issue Type: Improvement
  Components: Documentation, Table API & SQL
Affects Versions: 1.3.1, 1.4.0
Reporter: Fabian Hueske


The Table API documentation is currently lacking a description of OVER windows.
The page has a placeholder section with a TODO: 
{{./docs/dev/table/tableApi.md}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Reorganize Table API / SQL documentation

[Fabian Hueske]

Hi,

I merged the feature branch back to the master and the release-1.3 branch
and deleted it.
There are still a few gaps in the docs that need to be filled (there are
JIRAs for those), but the overall structure looks good IMO and most aspects
of the APIs are covered.

Cheers, Fabian

2017-05-28 15:21 GMT+02:00 Fabian Hueske :

> Hi everybody,
>
> I added a couple of subissues to https://issues.apache.org/
> jira/browse/FLINK-5354 and started to work on the first two.
> I'll open PRs for those soon.
>
> Please pick an issue and assign it to yourself if you want to help with
> the Table API / SQL documentation.
>
> Thanks,
> Fabian
>
>
> 2017-05-23 17:49 GMT+02:00 Fabian Hueske :
>
>> Hi everybody,
>>
>> I pushed the branch to the ASF Flink repository as a feature branch to
>> keep all PRs in one place:
>>
>> -->  https://github.com/apache/flink/tree/tableDocs
>>
>> Thanks,
>> Fabian
>>
>> 2017-05-23 16:25 GMT+01:00 Fabian Hueske :
>>
>>> Hi everybody,
>>>
>>> I prepared a branch that creates the proposed structure and copied the
>>> existing documentation into the corresponding pages / sections.
>>> There are plenty of gaps that need to be filled or reworked.
>>>
>>> --> https://github.com/fhueske/flink/tree/tableDocs
>>>
>>> How do we go on from here?
>>> I think the easiest would be if everybody who's interested in working on
>>> the documentation picks a page and prepares a PR against my branch (we
>>> could also push this into a feature branch in the Flink repository if
>>> somebody prefers that). The PRs are cross-checked and we merge everything
>>> into to the master when the docs are ready.
>>>
>>> Any opinions or other proposals?
>>>
>>> Cheers, Fabian
>>>
>>> 2017-05-23 10:31 GMT+01:00 Fabian Hueske :
>>>
 Hi everybody,

 Thanks for the feedback. I'll go ahead and create the proposed
 structure and move the content of the existing docs with comments of what
 needs to be adapted.
 I'll put this into branch of my Github repo and let you know when I'm
 done.
 From there, we can distribute working on the missing parts / parts that
 need adaption.

 Cheers, Fabian

 2017-05-19 9:44 GMT+01:00 jincheng sun :

> Hi, Fabian,
>
>   Thanks for the sketch. The structure is pretty well to me, And glad
> to
> join in the discussion in google doc.
>
> Cheers,
> SunJincheng
>
> 2017-05-19 14:55 GMT+08:00 Shaoxuan Wang :
>
> > Hello Fabian,
> > Thanks for drafting the proposal. I like the entire organization in
> general
> > and left a few comments. I think this will be a very good kick off to
> > reorganize the tableAPI&SQL doc.
> >
> > -shaoxuan
> >
> > On Fri, May 19, 2017 at 7:06 AM, Fabian Hueske 
> wrote:
> >
> > > Hi everybody,
> > >
> > > I came up with a proposal for the structure of the Table API / SQL
> > > documentation:
> > >
> > > https://docs.google.com/document/d/1ENY8tcPadZjoZ4AQ_
> > > lRRwWiVpScDkm_4rgxIGWGT5E0/edit?usp=sharing
> > >
> > > Feedback and comments are very welcome.
> > > Once we agree on a structure, we can create skeletons and
> distribute the
> > > work.
> > >
> > > Cheers,
> > > Fabian
> > >
> > > 2017-05-18 21:01 GMT+02:00 Haohui Mai :
> > >
> > > > +1
> > > >
> > > > The Table / SQL component has made significant progress in the
> last few
> > > > months (kudos to all contributors).
> > > >
> > > > It is a good time to have a documentation to reflect all the
> changes in
> > > the
> > > > Table / SQL side.
> > > >
> > > >
> > > >
> > > > On Thu, May 18, 2017 at 8:12 AM Robert Metzger <
> rmetz...@apache.org>
> > > > wrote:
> > > >
> > > > > Thank you Fabian for working on the proposal.
> > > > >
> > > > > On Thu, May 18, 2017 at 3:51 PM, Fabian Hueske <
> fhue...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > Thanks for starting this discussion Robert.
> > > > > >
> > > > > > I think with the next release the Table API / SQL should be
> moved
> > up
> > > in
> > > > > the
> > > > > > Application Development menu.
> > > > > > I also though about restructuring the docs, but it won't be
> trivial
> > > to
> > > > do
> > > > > > this, IMO because there are many orthogonal aspects:
> > > > > > - Stream/Batch
> > > > > > - Table/SQL
> > > > > > - Scala/Java
> > > > > >
> > > > > > and sometimes also common concepts.
> > > > > > At the moment there are also many new features missing like
> OVER
> > > > windows,
> > > > > > UDAGGs, retraction, StreamTableSinks, time indicator
> attributes,
> > > filter
> > > > > > pushdown, ...
> > > > > >
> > > > > > I will try to sketch a new structure in a Google Doc in the
> next
> > days
> > > > and
> > > > > > share it i

Re: Tracking API

[Ameet BD]

Hello,

I want to know which CEP API resulted in a call to a certain invocation of
submitTask function. I think the  stack trace will not tell me that because
they are in different thread contexts.

Regards,
Ameet

On Thu, Jun 15, 2017 at 3:16 PM, Aljoscha Krettek 
wrote:

> Hi,
>
> Would this be what you’re looking for: https://stackoverflow.com/
> questions/1069066/get-current-stack-trace-in-java <
> https://stackoverflow.com/questions/1069066/get-current-
> stack-trace-in-java> ?
>
> Best,
> Aljoscha
>
> > On 14. Jun 2017, at 12:46, Ameet BD  wrote:
> >
> > Hello,
> >
> > I am trying to print which Flink API (e.g Complex Event Processing or
> > Streaming SQL) was called that resulted in a call down the function stack
> > in task handling code. Please tell me how it could be done using existing
> > code base (1.3.0).
> > Regards,
> > Ameet
>
>

Re: Tracking API

[Aljoscha Krettek]

Ah, now I see what you mean. I’m afraid there’s no built-in functionality for 
that. You probably have to follow the API calls yourself.

Best,
Aljoscha

> On 15. Jun 2017, at 12:54, Ameet BD  wrote:
> 
> Hello,
> 
> I want to know which CEP API resulted in a call to a certain invocation of
> submitTask function. I think the  stack trace will not tell me that because
> they are in different thread contexts.
> 
> Regards,
> Ameet
> 
> On Thu, Jun 15, 2017 at 3:16 PM, Aljoscha Krettek 
> wrote:
> 
>> Hi,
>> 
>> Would this be what you’re looking for: https://stackoverflow.com/
>> questions/1069066/get-current-stack-trace-in-java <
>> https://stackoverflow.com/questions/1069066/get-current-
>> stack-trace-in-java> ?
>> 
>> Best,
>> Aljoscha
>> 
>>> On 14. Jun 2017, at 12:46, Ameet BD  wrote:
>>> 
>>> Hello,
>>> 
>>> I am trying to print which Flink API (e.g Complex Event Processing or
>>> Streaming SQL) was called that resulted in a call down the function stack
>>> in task handling code. Please tell me how it could be done using existing
>>> code base (1.3.0).
>>> Regards,
>>> Ameet
>> 
>> 

Re: FlinkML on slack

[Stavros Kontopoulos]

Ziyad added.

Stavros

On Sun, Jun 11, 2017 at 4:45 PM, Ziyad Muhammed  wrote:

> Hi Stavros
>
> Could you please send me an invite to the slack channel?
>
> Best
> Ziyad
>
>
> On Sun, Jun 11, 2017 at 1:53 AM, Stavros Kontopoulos <
> st.kontopou...@gmail.com> wrote:
>
> > @Henry @Tao @Martin invitations sent... Thnx @Theo for handling the
> Apache
> > compliance issues.
> >
> > Best,
> > Stavros
> >
> > On Sat, Jun 10, 2017 at 10:27 PM, Henry Saputra  >
> > wrote:
> >
> > > Hi Stavros,
> > >
> > > Could you also send me invite to the Slack?
> > >
> > > My email is hsapu...@apache.org
> > >
> > > Thanks,
> > >
> > > Henry
> > >
> > >
> > > On Thu, Jun 8, 2017 at 2:21 AM, Stavros Kontopoulos <
> > > st.kontopou...@gmail.com> wrote:
> > >
> > > > Hi Aljoscha,
> > > >
> > > > Slack is invite only to the best of my knowledge, I just sent you an
> > > > invitation.
> > > >
> > > > Best,
> > > > Stavros
> > > >
> > > >
> > > > On Thu, Jun 8, 2017 at 11:31 AM, Aljoscha Krettek <
> aljos...@apache.org
> > >
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Is the slack invite based? If yes, could you please send me one?
> > > > >
> > > > > Best,
> > > > > Aljoscha
> > > > >
> > > > > > On 7. Jun 2017, at 21:56, Stavros Kontopoulos <
> > > > st.kontopou...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > We took the initiative to create the organization for FlinkML on
> > > slack
> > > > > > (thnx Eron).
> > > > > > There is now a channel for model-serving
> > > > > >  1CjWL9aLxPrKytKxUF5c3ohs0ickp0
> > > > > fdEXPsPYPEywsE/edit#>.
> > > > > > Another is coming for flink-jpmml.
> > > > > > You are invited to join the channels and the efforts. @Gabor
> @Theo
> > > > please
> > > > > > consider adding channels for the other efforts there as well.
> > > > > >
> > > > > > FlinkMS on Slack  (
> > > > > https://flinkml.slack.com/)
> > > > > >
> > > > > > Details for the efforts here: Flink Roadmap doc
> > > > > >  1afQbvZBTV15qF3vobVWUjxQc49h3U
> > > > > d06MIRhahtJ6dw/edit#>
> > > > > >
> > > > > > Github  (https://github.com/FlinkML)
> > > > > >
> > > > > >
> > > > > > Stavros
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: FlinkML on slack

[Lokesh Amarnath]

Hi Stravros,

Could you also please add me to the Slack channel? My email id is:
lokesh.amarn...@gmail.com.

Thanks,
Lokesh



On Thu, Jun 15, 2017 at 6:27 PM, Stavros Kontopoulos <
st.kontopou...@gmail.com> wrote:

> Ziyad added.
>
> Stavros
>
> On Sun, Jun 11, 2017 at 4:45 PM, Ziyad Muhammed  wrote:
>
> > Hi Stavros
> >
> > Could you please send me an invite to the slack channel?
> >
> > Best
> > Ziyad
> >
> >
> > On Sun, Jun 11, 2017 at 1:53 AM, Stavros Kontopoulos <
> > st.kontopou...@gmail.com> wrote:
> >
> > > @Henry @Tao @Martin invitations sent... Thnx @Theo for handling the
> > Apache
> > > compliance issues.
> > >
> > > Best,
> > > Stavros
> > >
> > > On Sat, Jun 10, 2017 at 10:27 PM, Henry Saputra <
> henry.sapu...@gmail.com
> > >
> > > wrote:
> > >
> > > > Hi Stavros,
> > > >
> > > > Could you also send me invite to the Slack?
> > > >
> > > > My email is hsapu...@apache.org
> > > >
> > > > Thanks,
> > > >
> > > > Henry
> > > >
> > > >
> > > > On Thu, Jun 8, 2017 at 2:21 AM, Stavros Kontopoulos <
> > > > st.kontopou...@gmail.com> wrote:
> > > >
> > > > > Hi Aljoscha,
> > > > >
> > > > > Slack is invite only to the best of my knowledge, I just sent you
> an
> > > > > invitation.
> > > > >
> > > > > Best,
> > > > > Stavros
> > > > >
> > > > >
> > > > > On Thu, Jun 8, 2017 at 11:31 AM, Aljoscha Krettek <
> > aljos...@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Is the slack invite based? If yes, could you please send me one?
> > > > > >
> > > > > > Best,
> > > > > > Aljoscha
> > > > > >
> > > > > > > On 7. Jun 2017, at 21:56, Stavros Kontopoulos <
> > > > > st.kontopou...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > We took the initiative to create the organization for FlinkML
> on
> > > > slack
> > > > > > > (thnx Eron).
> > > > > > > There is now a channel for model-serving
> > > > > > >  > 1CjWL9aLxPrKytKxUF5c3ohs0ickp0
> > > > > > fdEXPsPYPEywsE/edit#>.
> > > > > > > Another is coming for flink-jpmml.
> > > > > > > You are invited to join the channels and the efforts. @Gabor
> > @Theo
> > > > > please
> > > > > > > consider adding channels for the other efforts there as well.
> > > > > > >
> > > > > > > FlinkMS on Slack  (
> > > > > > https://flinkml.slack.com/)
> > > > > > >
> > > > > > > Details for the efforts here: Flink Roadmap doc
> > > > > > >  > 1afQbvZBTV15qF3vobVWUjxQc49h3U
> > > > > > d06MIRhahtJ6dw/edit#>
> > > > > > >
> > > > > > > Github  (
> https://github.com/FlinkML)
> > > > > > >
> > > > > > >
> > > > > > > Stavros
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-6930) Selecting window start / end on row-based Tumble/Slide window causes NPE

[Fabian Hueske (JIRA)]

Fabian Hueske created FLINK-6930:


 Summary: Selecting window start / end on row-based Tumble/Slide 
window causes NPE
 Key: FLINK-6930
 URL: https://issues.apache.org/jira/browse/FLINK-6930
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Affects Versions: 1.3.0, 1.4.0
Reporter: Fabian Hueske


Selecting the start and end properties of a row-based window causes a 
NullPointerException.
The following program:

{code}
val windowedTable = table
  .window(Tumble over 2.rows on 'proctime as 'w)
  .groupBy('w, 'string)
  .select('string as 'n, 'int.count as 'cnt, 'w.start as 's, 'w.end as 'e)
{code}

causes 

{code}
Caused by: java.lang.NullPointerException
at 
org.apache.calcite.runtime.SqlFunctions.toLong(SqlFunctions.java:1556)
at 
org.apache.calcite.runtime.SqlFunctions.toLong(SqlFunctions.java:1551)
at DataStreamCalcRule$40.processElement(Unknown Source)
at 
org.apache.flink.table.runtime.CRowProcessRunner.processElement(CRowProcessRunner.scala:67)
at 
org.apache.flink.table.runtime.CRowProcessRunner.processElement(CRowProcessRunner.scala:35)
at 
org.apache.flink.streaming.api.operators.ProcessOperator.processElement(ProcessOperator.java:66)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:528)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:503)
at 
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:483)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:890)
at 
org.apache.flink.streaming.api.operators.AbstractStreamOperator$CountingOutput.collect(AbstractStreamOperator.java:868)
at 
org.apache.flink.streaming.api.operators.TimestampedCollector.collect(TimestampedCollector.java:51)
at 
org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply(IncrementalAggregateWindowFunction.scala:75)
at 
org.apache.flink.table.runtime.aggregate.IncrementalAggregateWindowFunction.apply(IncrementalAggregateWindowFunction.scala:37)
at 
org.apache.flink.streaming.runtime.operators.windowing.functions.InternalSingleValueWindowFunction.process(InternalSingleValueWindowFunction.java:46)
at 
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.emitWindowContents(WindowOperator.java:599)
at 
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.processElement(WindowOperator.java:456)
at 
org.apache.flink.streaming.runtime.io.StreamInputProcessor.processInput(StreamInputProcessor.java:207)
at 
org.apache.flink.streaming.runtime.tasks.OneInputStreamTask.run(OneInputStreamTask.java:69)
at 
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:265)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
at java.lang.Thread.run(Thread.java:745)
{code}

We should validate that the start and end window properties are not accessed if 
the window is defined on row-counts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6931) Support custom compression formats for checkpoints (+Upgrade/Compatibility)

[Stefan Richter (JIRA)]

Stefan Richter created FLINK-6931:
-

 Summary: Support custom compression formats for checkpoints 
(+Upgrade/Compatibility)
 Key: FLINK-6931
 URL: https://issues.apache.org/jira/browse/FLINK-6931
 Project: Flink
  Issue Type: Improvement
  Components: State Backends, Checkpointing
Reporter: Stefan Richter


With FLINK-6773, we introduced optional snappy compression for keyed state in 
full checkpoints and savepoints. We should offer users a way to register their 
own compression formats with the {{ExecutionConfig}}. For this, we should also 
have a compatibility story, very similar to what 
{{TypeSerializerConfigSnapshot}} doesfor type serializers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[DISCUSS] Table API / SQL features for Flink 1.4.0

[Fabian Hueske]

Hi everybody,

I would like to start a discussion about the targeted feature set of the
Table API / SQL for Flink 1.4.0.
Flink 1.3.0 was released about 2 weeks ago and we have 2.5 months (~11
weeks, until begin of September) left until the feature freeze for Flink
1.4.0.

I think it makes sense to start with a collection of desired features. Once
we have a list of requested features, we might want to prioritize and maybe
also assign responsibilities.

When we prioritize, we should keep in mind that:
- we want to have a consistent API. Larger features should be developed in
a feature branch first.
- the next months are typical time for vacations
- we have been bottlenecked by committer resources in the last release.

I think the following features would be a nice addition to the current
state:

- Conversion of a stream into an upsert table (with retraction, updating to
the last row per key)
- Joins for streaming tables
  - Stream-Stream (time-range predicate) there is already a PR for
processing time joins
  - Table-Table (with retraction)
- Support for late arriving records in group window aggregations
- Exposing a keyed result table as queryable state

Which features are others looking for?

Cheers,
Fabian

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

[Fabian Hueske]

Hi everybody,

I just stumbled over this blog post [1] which discusses new features in SQL
2016.
Apparently the match_recognize clause is part of that. The blogpost also
contains a slide set that presents the pattern matching feature and a link
to a 90 page technical report.

I thought this might be helpful as a references.

Cheers, Fabian

[1] http://modern-sql.com/blog/2017-06/whats-new-in-sql-2016

2017-06-15 11:53 GMT+02:00 Till Rohrmann :

> @Jark: You should now have the permissions to create pages in the Flink
> wiki.
>
> Cheers,
> Till
>
> On Thu, Jun 15, 2017 at 5:11 AM, Jark Wu  wrote:
>
> > Hi Till,
> >
> > Could you grant me the edit permission of Flink WIKI? My id is imjark.
> >
> > Thanks,
> > Jark Wu
> >
> > 2017-06-15 0:07 GMT+08:00 Till Rohrmann :
> >
> > > I think that the integration of SQL and CEP would make a good FLIP.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jun 14, 2017 at 2:40 PM, Jark Wu  wrote:
> > >
> > > > Hi,
> > > >
> > > > Do you think whether we should create a FLIP for this proposal to
> track
> > > > progress?
> > > >
> > > > Regards,
> > > > Jark
> > > >
> > > > 2017-06-13 16:59 GMT+08:00 Dian Fu :
> > > >
> > > > > Hi Fabian,
> > > > >
> > > > > Thanks a lot. Agree that we can start working by adding the missing
> > > > > features of the CEP library.
> > > > >
> > > > > Best regards,
> > > > > Dian
> > > > >
> > > > > On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske 
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > @Dian Fu: I gave you contributor permissions. :-)
> > > > > >
> > > > > > I don't think we have to wait for Calcite 1.13 to start working
> on
> > > the
> > > > > > missing features of the CEP library and extending the prototype.
> > > > > > We might want to wait with the integration into flink-table until
> > > > Calcite
> > > > > > 1.13 is out and we updated the dependency though.
> > > > > >
> > > > > > Best, Fabian
> > > > > >
> > > > > > 2017-06-13 9:45 GMT+02:00 jincheng sun  >:
> > > > > >
> > > > > > > Hi Jark, Dian,
> > > > > > >
> > > > > > > Thanks for bring up this discuss and share the prototype.
> > > > > > >
> > > > > > > +1 to push this great feature forward!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > SunJincheng
> > > > > > >
> > > > > > > 2017-06-13 15:34 GMT+08:00 Jark Wu :
> > > > > > >
> > > > > > > > Thank you Yueting for pointing out the mistake in the
> > prototype.
> > > I
> > > > > > > > accidentally introduced it when merge code.
> > > > > > > >
> > > > > > > > I'm so glad to see so many people are interested in the
> > feature.
> > > > > Let's
> > > > > > > work
> > > > > > > > out together to push it forward!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Jark Wu
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-06-13 15:27 GMT+08:00 Liangfei Su  >:
> > > > > > > >
> > > > > > > > > +1 for the feature. Myself was a user of Siddhi, this is
> > pretty
> > > > > user
> > > > > > > > > friendly feature to provide to user.
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <
> > > dian0511...@gmail.com>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Yueting & Dawid Wysakowicz,
> > > > > > > > > >
> > > > > > > > > > Very glad that you're interested in this feature and
> you're
> > > > > > > definitely
> > > > > > > > > > welcome to join this work and also anyone else.:)
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Dian
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > > > > > > wysakowicz.da...@gmail.com> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Integrating SQL with CEP seems like a really nice idea.
> > > > > > > > Unfortunately I
> > > > > > > > > > had
> > > > > > > > > > > time just for a brief look at the design doc, but it
> > looks
> > > > > really
> > > > > > > > cool
> > > > > > > > > > and
> > > > > > > > > > > thorough. Also will have a second run tomorrow and will
> > try
> > > > to
> > > > > > > > provide
> > > > > > > > > > more
> > > > > > > > > > > comments. Anyway will be glad to help pushing the
> > > initiative
> > > > > > > forward.
> > > > > > > > > > >
> > > > > > > > > > > Z pozdrowieniami! / Cheers!
> > > > > > > > > > >
> > > > > > > > > > > Dawid Wysakowicz
> > > > > > > > > > >
> > > > > > > > > > > *Data/Software Engineer*
> > > > > > > > > > >
> > > > > > > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > > > > > > >
> > > > > > > > > > > 
> > > > > > > > > > >
> > > > > > > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <
> > > yestinm...@gmail.com
> > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Dian & Jark,
> > > > > > > > > > > >
> > > > > > > > > > > > I checked out your prototype code, but it didn't pass
> > the
> > > > > > > CEPITCase
> > > > > > > > > > test
> > > > > > > > > > > in
> > > > > > > > > > > > the flink-table component.
>

[jira] [Created] (FLINK-6932) Update the inaccessible Dataflow Model paper link

[mingleizhang (JIRA)]

mingleizhang created FLINK-6932:
---

 Summary: Update the inaccessible Dataflow Model paper link
 Key: FLINK-6932
 URL: https://issues.apache.org/jira/browse/FLINK-6932
 Project: Flink
  Issue Type: Bug
  Components: Documentation
Reporter: mingleizhang
Assignee: mingleizhang


 I tried to access the [#Dataflow Model paper] link which under 
[https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/event_time.html],
 then it gives me an error [ 404 ] instead.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: [DISCUSS] Table API / SQL features for Flink 1.4.0

[Shaoxuan Wang]

Nice timing, Fabian!

Your checklist aligns our plans very well. Here are the things we are
working on & planning to contribute to release 1.4:
1. DDL (with property waterMark config for source-table, and emit config on
result-table)
2. unbounded stream-stream joins (with retraction supported)
3. backend state user interface for UDAGG
4. UDOP (as oppose to UDF(scalars to scalar)/UDTF(scalar to
table)/UDAGG(table to scalar), this allows user to define a table to table
conversion business logic)

Some of them already have PR/jira, while some are not. We will send out the
design doc for the missing ones very soon. Looking forward to the 1.4
release.

Btw, what is "Table-Table (with retraction)" you have mentioned in your
plan?

Regards,
Shaoxuan



On Thu, Jun 15, 2017 at 10:29 PM, Fabian Hueske  wrote:

> Hi everybody,
>
> I would like to start a discussion about the targeted feature set of the
> Table API / SQL for Flink 1.4.0.
> Flink 1.3.0 was released about 2 weeks ago and we have 2.5 months (~11
> weeks, until begin of September) left until the feature freeze for Flink
> 1.4.0.
>
> I think it makes sense to start with a collection of desired features. Once
> we have a list of requested features, we might want to prioritize and maybe
> also assign responsibilities.
>
> When we prioritize, we should keep in mind that:
> - we want to have a consistent API. Larger features should be developed in
> a feature branch first.
> - the next months are typical time for vacations
> - we have been bottlenecked by committer resources in the last release.
>
> I think the following features would be a nice addition to the current
> state:
>
> - Conversion of a stream into an upsert table (with retraction, updating to
> the last row per key)
> - Joins for streaming tables
>   - Stream-Stream (time-range predicate) there is already a PR for
> processing time joins
>   - Table-Table (with retraction)
> - Support for late arriving records in group window aggregations
> - Exposing a keyed result table as queryable state
>
> Which features are others looking for?
>
> Cheers,
> Fabian
>

[jira] [Created] (FLINK-6933) Refactor NFACompiler to reduce code duplication

[Dian Fu (JIRA)]

Dian Fu created FLINK-6933:
--

 Summary: Refactor NFACompiler to reduce code duplication
 Key: FLINK-6933
 URL: https://issues.apache.org/jira/browse/FLINK-6933
 Project: Flink
  Issue Type: Bug
  Components: CEP
Reporter: Dian Fu
Assignee: Dian Fu


I find that part of the code in NFACompiler is duplicate, this JIRA tries to 
eliminate the code duplication.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (FLINK-6934) Consider moving LRUCache class

[mingleizhang (JIRA)]

mingleizhang created FLINK-6934:
---

 Summary: Consider moving LRUCache class
 Key: FLINK-6934
 URL: https://issues.apache.org/jira/browse/FLINK-6934
 Project: Flink
  Issue Type: Improvement
  Components: Local Runtime
Reporter: mingleizhang
Assignee: mingleizhang


LRUCache class is not used any more. So, I would suggest remove it.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)