[jira] [Commented] (FLINK-5792) Improve “UDF/UDTF" to support constructor with parameter.

2017-02-17 Thread sunjincheng (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871584#comment-15871584
 ] 

sunjincheng commented on FLINK-5792:


HI,[~clarkyzl]  Thanks for your attention to this JIRA. After 
https://github.com/apache/flink/pull/3330 merge into master, I'll open the PR 
of FLINK-5794.

> Improve “UDF/UDTF" to support constructor with parameter.
> -
>
> Key: FLINK-5792
> URL: https://issues.apache.org/jira/browse/FLINK-5792
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently UDF/UDTF in the codegen phase using a nonparametric constructor to 
> create the instance, causing the user can not include the state value in the 
> UDF/UDTF. The UDF/UDTF's codegen phase can use a serialized mechanism so that 
> the UDTF can contain state values.
> 1. UserDefinedFunction inherits Serializable.
> 2. Modify CodeGenerator about UDF/UDTF part.
> 3. Modify TableAPI about UDF/UDTF
> 4. Add Test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (FLINK-5792) Improve “UDF/UDTF" to support constructor with parameter.

2017-02-16 Thread Zhuoluo Yang (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15871230#comment-15871230
 ] 

Zhuoluo Yang commented on FLINK-5792:
-

Hi [~sunjincheng121], I think this feature is very important to FLINK-5802. 
Because we need to pass something (eg, Hive Udf ) to the Flink's UDF/UDTF. A 
serialization will be a good idea, IMHO. 

> Improve “UDF/UDTF" to support constructor with parameter.
> -
>
> Key: FLINK-5792
> URL: https://issues.apache.org/jira/browse/FLINK-5792
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently UDF/UDTF in the codegen phase using a nonparametric constructor to 
> create the instance, causing the user can not include the state value in the 
> UDF/UDTF. The UDF/UDTF's codegen phase can use a serialized mechanism so that 
> the UDTF can contain state values.
> 1. UserDefinedFunction inherits Serializable.
> 2. Modify CodeGenerator about UDF/UDTF part.
> 3. Modify TableAPI about UDF/UDTF
> 4. Add Test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (FLINK-5792) Improve “UDF/UDTF" to support constructor with parameter.

2017-02-16 Thread sunjincheng (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869488#comment-15869488
 ] 

sunjincheng commented on FLINK-5792:


BYW, when I implement the issue, I had try to use `kryo` serialize 
the UDTF/UDF object, It will make byte [] very small.
But unfortunately, It asked the serialized member must have a zero-parameter 
constructor which not friendly to user.

> Improve “UDF/UDTF" to support constructor with parameter.
> -
>
> Key: FLINK-5792
> URL: https://issues.apache.org/jira/browse/FLINK-5792
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently UDF/UDTF in the codegen phase using a nonparametric constructor to 
> create the instance, causing the user can not include the state value in the 
> UDF/UDTF. The UDF/UDTF's codegen phase can use a serialized mechanism so that 
> the UDTF can contain state values.
> 1. UserDefinedFunction inherits Serializable.
> 2. Modify CodeGenerator about UDF/UDTF part.
> 3. Modify TableAPI about UDF/UDTF
> 4. Add Test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (FLINK-5792) Improve “UDF/UDTF" to support constructor with parameter.

2017-02-16 Thread sunjincheng (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869473#comment-15869473
 ] 

sunjincheng commented on FLINK-5792:


HI, [~fhueske] Thanks for your attention to this JIRA.   Thanks [~jark] review 
the PR.
I had serialize UDF object at code-gen stage and deserialize in the  `open()` 
method. JackWu is right. the current implement is similar to the approach-2. 
What do you think about the current PR?

> Improve “UDF/UDTF" to support constructor with parameter.
> -
>
> Key: FLINK-5792
> URL: https://issues.apache.org/jira/browse/FLINK-5792
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently UDF/UDTF in the codegen phase using a nonparametric constructor to 
> create the instance, causing the user can not include the state value in the 
> UDF/UDTF. The UDF/UDTF's codegen phase can use a serialized mechanism so that 
> the UDTF can contain state values.
> 1. UserDefinedFunction inherits Serializable.
> 2. Modify CodeGenerator about UDF/UDTF part.
> 3. Modify TableAPI about UDF/UDTF
> 4. Add Test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (FLINK-5792) Improve “UDF/UDTF" to support constructor with parameter.

2017-02-15 Thread Fabian Hueske (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869413#comment-15869413
 ] 

Fabian Hueske commented on FLINK-5792:
--

Ah, great. Thanks for pointing to the PR [~jark]. 

> Improve “UDF/UDTF" to support constructor with parameter.
> -
>
> Key: FLINK-5792
> URL: https://issues.apache.org/jira/browse/FLINK-5792
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently UDF/UDTF in the codegen phase using a nonparametric constructor to 
> create the instance, causing the user can not include the state value in the 
> UDF/UDTF. The UDF/UDTF's codegen phase can use a serialized mechanism so that 
> the UDTF can contain state values.
> 1. UserDefinedFunction inherits Serializable.
> 2. Modify CodeGenerator about UDF/UDTF part.
> 3. Modify TableAPI about UDF/UDTF
> 4. Add Test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (FLINK-5792) Improve “UDF/UDTF" to support constructor with parameter.

2017-02-15 Thread Jark Wu (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869404#comment-15869404
 ] 

Jark Wu commented on FLINK-5792:


Hi [~fhueske], [~sunjincheng121] has create a PR for this issue: 
https://github.com/apache/flink/pull/3330 

Sun's way is similar to approach-2, which serialized the UDF object into a 
string. Maybe we can move the discussion under the PR.

> Improve “UDF/UDTF" to support constructor with parameter.
> -
>
> Key: FLINK-5792
> URL: https://issues.apache.org/jira/browse/FLINK-5792
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently UDF/UDTF in the codegen phase using a nonparametric constructor to 
> create the instance, causing the user can not include the state value in the 
> UDF/UDTF. The UDF/UDTF's codegen phase can use a serialized mechanism so that 
> the UDTF can contain state values.
> 1. UserDefinedFunction inherits Serializable.
> 2. Modify CodeGenerator about UDF/UDTF part.
> 3. Modify TableAPI about UDF/UDTF
> 4. Add Test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (FLINK-5792) Improve “UDF/UDTF" to support constructor with parameter.

2017-02-15 Thread Fabian Hueske (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869395#comment-15869395
 ] 

Fabian Hueske commented on FLINK-5792:
--

I agree, it makes sense to ship a serialized UDF object.
Regular Flink function are also serialized and distributed to the workers.

[~sunjincheng121] can you explain a bit how you want to distribute the UDF?
I see two options:
1. make the UDF a member of wrapping function. It might be a bit tricky to pass 
the reference into the code-gen'd function.
2. add a final byte[] field into the code-gen'd function that holds the 
serialized UDF object and deserialize during initialization. This will blow up 
the code-gen'd string but might work well.

Best, Fabian

> Improve “UDF/UDTF" to support constructor with parameter.
> -
>
> Key: FLINK-5792
> URL: https://issues.apache.org/jira/browse/FLINK-5792
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently UDF/UDTF in the codegen phase using a nonparametric constructor to 
> create the instance, causing the user can not include the state value in the 
> UDF/UDTF. The UDF/UDTF's codegen phase can use a serialized mechanism so that 
> the UDTF can contain state values.
> 1. UserDefinedFunction inherits Serializable.
> 2. Modify CodeGenerator about UDF/UDTF part.
> 3. Modify TableAPI about UDF/UDTF
> 4. Add Test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)


[jira] [Commented] (FLINK-5792) Improve “UDF/UDTF" to support constructor with parameter.

2017-02-15 Thread Jark Wu (JIRA)

[ 
https://issues.apache.org/jira/browse/FLINK-5792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869029#comment-15869029
 ] 

Jark Wu commented on FLINK-5792:


+1

The serialization way will be more flexible.

> Improve “UDF/UDTF" to support constructor with parameter.
> -
>
> Key: FLINK-5792
> URL: https://issues.apache.org/jira/browse/FLINK-5792
> Project: Flink
>  Issue Type: Improvement
>  Components: Table API & SQL
>Reporter: sunjincheng
>Assignee: sunjincheng
>
> Currently UDF/UDTF in the codegen phase using a nonparametric constructor to 
> create the instance, causing the user can not include the state value in the 
> UDF/UDTF. The UDF/UDTF's codegen phase can use a serialized mechanism so that 
> the UDTF can contain state values.
> 1. UserDefinedFunction inherits Serializable.
> 2. Modify CodeGenerator about UDF/UDTF part.
> 3. Modify TableAPI about UDF/UDTF
> 4. Add Test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)