[jira] [Created] (FLINK-11242) The predicate `'o_c_id === Null(Types.STRING)` not work in TableAPI

[sunjincheng (JIRA)]

sunjincheng created FLINK-11242:
---

 Summary: The predicate `'o_c_id === Null(Types.STRING)` not work 
in TableAPI
 Key: FLINK-11242
 URL: https://issues.apache.org/jira/browse/FLINK-11242
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Affects Versions: 1.8.0
Reporter: sunjincheng


Schema:
{code:java}
Table(c_id:String,  o_c_id:String){code}
Data: 
{code:java}
c_001,c_001
c_002,c_002
c_003,null{code}
TableAPI：
{code:java}
val result = tab
  .select('c_id, 'o_c_id)
  .where('o_c_id === Null(Types.STRING)){code}
Result:

We expect  `c_003, null`, but got empty.

BTW, we can got the correct result by follows TableAPI:
{code:java}
val result = tab
  .select('c_id, 'o_c_id)
  .where('o_c_id isNull){code}
 

I think is better for we let `.where('o_c_id === Null(Types.STRING))` work 
well. What do you think?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11243) GroupBy udf() can not be select in TableAPI

[sunjincheng (JIRA)]

sunjincheng created FLINK-11243:
---

 Summary: GroupBy udf() can not be select in TableAPI
 Key: FLINK-11243
 URL: https://issues.apache.org/jira/browse/FLINK-11243
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Affects Versions: 1.8.0
Reporter: sunjincheng


The group key can not be select in the follows TableAPI:
{code:java}
val result = order
  .groupBy('o_time.substring(1, 16))
  .select('o_time.substring(1, 16), 'o_id.count){code}
Exception:
{code:java}
org.apache.flink.table.api.ValidationException: Cannot resolve field [o_time] 
given input [('o_time).substring(1, 16), TMP_0].

at 
org.apache.flink.table.plan.logical.LogicalNode.failValidation(LogicalNode.scala:156){code}
BTW, We can only use the following description to avoid the above problem.
{code:java}
val result = order
  .select('o_id, 'c_id, 'o_time.substring(1, 16) as 'key)
  .groupBy('key)
  .select('key, 'o_id.count)
{code}

But I think it is better to make `groupby udf()` can be select in TableAPI. 
What do you think?






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11244) flink web.port can not be configured correctly in yarn mode

[Kaicheng Sun (JIRA)]

Kaicheng Sun created FLINK-11244:


 Summary: flink web.port can not be configured correctly in yarn 
mode
 Key: FLINK-11244
 URL: https://issues.apache.org/jira/browse/FLINK-11244
 Project: Flink
  Issue Type: Bug
  Components: Metrics, REST, YARN
Affects Versions: 1.4.2
 Environment: ubuntu 16.04

openjdk-8
Hadoop 2.7.1.2.3.6.0-3796

flink 1.4.0
Reporter: Kaicheng Sun
 Attachments: 捕获.PNG

Using yarn to manage flink resources, after changing  web.port configuration in 
flink-conf.yaml, the change has no affect, and the web.port in the flink 
dashboard is always 0.

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11245) Update documentation for catalogs in SQL-Client

[Eron Wright (JIRA)]

Eron Wright  created FLINK-11245:


 Summary: Update documentation for catalogs in SQL-Client
 Key: FLINK-11245
 URL: https://issues.apache.org/jira/browse/FLINK-11245
 Project: Flink
  Issue Type: Sub-task
  Components: Documentation, Table API & SQL
Reporter: Eron Wright 
Assignee: Eron Wright 


Add to the SQL-Client documentation, information about using catalogs in an 
environment file.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-11246) Fix distinct AGG visibility issues

[sunjincheng (JIRA)]

sunjincheng created FLINK-11246:
---

 Summary: Fix distinct AGG visibility issues
 Key: FLINK-11246
 URL: https://issues.apache.org/jira/browse/FLINK-11246
 Project: Flink
  Issue Type: Bug
  Components: Table API & SQL
Reporter: sunjincheng


1. DistinctAggregateFunction# distinct -> remove private[flink] 

2. case class DistinctAgg#distinct  -> remove private[flink]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem

[Eron Wright]

Hi folks, there's clearly some incremental steps to be taken to introduce
catalog support to SQL Client, complementary to what is proposed in the
Flink-Hive Metastore design doc.  I was quietly working on this over the
holidays.   I posted some new sub-tasks, PRs, and sample code
to FLINK-10744.

What inspired me to get involved is that the catalog interface seems like a
great way to encapsulate a 'library' of Flink tables and functions.  For
example, the NYC Taxi dataset (TaxiRides, TaxiFares, various UDFs) may be
nicely encapsulated as a catalog (TaxiData).   Such a library should be
fully consumable in SQL Client.

I implemented the above.  Some highlights:

1. A fully-worked example of using the Taxi dataset in SQL Client via an
environment file.
- an ASCII video showing the SQL Client in action:
https://asciinema.org/a/C8xuAjmZSxCuApgFgZQyeIHuo

- the corresponding environment file (will be even more concise once
'FLINK-10696 Catalog UDFs' is merged):
*https://github.com/EronWright/flink-training-exercises/blob/3be008d64be975ced0f1a7e3901a8c5353f72a7e/src/main/dist/conf/sql-client-defaults.yaml
*

- the typed API for standalone table applications:
*https://github.com/EronWright/flink-training-exercises/blob/3be008d64be975ced0f1a7e3901a8c5353f72a7e/src/main/java/com/dataartisans/flinktraining/examples/table_java/examples/ViaCatalog.java#L50
*

2. Implementation of the core catalog descriptor and factory.  I realize
that some renames may later occur as per the design doc, and would be happy
to do that as a follow-up.
https://github.com/apache/flink/pull/7390

3. Implementation of a connect-style API on TableEnvironment to use catalog
descriptor.
https://github.com/apache/flink/pull/7392

4. Integration into SQL-Client's environment file:
https://github.com/apache/flink/pull/7393

I realize that the overall Hive integration is still evolving, but I
believe that these PRs are a good stepping stone. Here's the list (in
bottom-up order):
- https://github.com/apache/flink/pull/7386
- https://github.com/apache/flink/pull/7388
- https://github.com/apache/flink/pull/7389
- https://github.com/apache/flink/pull/7390
- https://github.com/apache/flink/pull/7392
- https://github.com/apache/flink/pull/7393

Thanks and enjoy 2019!
Eron W


On Sun, Nov 18, 2018 at 3:04 PM Zhang, Xuefu 
wrote:

> Hi Xiaowei,
>
> Thanks for bringing up the question. In the current design, the properties
> for meta objects are meant to cover anything that's specific to a
> particular catalog and agnostic to Flink. Anything that is common (such as
> schema for tables, query text for views, and udf classname) are abstracted
> as members of the respective classes. However, this is still in discussion,
> and Timo and I will go over this and provide an update.
>
> Please note that UDF is a little more involved than what the current
> design doc shows. I'm still refining this part.
>
> Thanks,
> Xuefu
>
>
> --
> Sender:Xiaowei Jiang 
> Sent at:2018 Nov 18 (Sun) 15:17
> Recipient:dev 
> Cc:Xuefu ; twalthr ; piotr <
> pi...@data-artisans.com>; Fabian Hueske ; suez1224 <
> suez1...@gmail.com>
> Subject:Re: [DISCUSS] Integrate Flink SQL well with Hive ecosystem
>
> Thanks Xuefu for the detailed design doc! One question on the properties
> associated with the catalog objects. Are we going to leave them completely
> free form or we are going to set some standard for that? I think that the
> answer may depend on if we want to explore catalog specific optimization
> opportunities. In any case, I think that it might be helpful for
> standardize as much as possible into strongly typed classes and use leave
> these properties for catalog specific things. But I think that we can do it
> in steps.
>
> Xiaowei
> On Fri, Nov 16, 2018 at 4:00 AM Bowen Li  wrote:
> Thanks for keeping on improving the overall design, Xuefu! It looks quite
>  good to me now.
>
>  Would be nice that cc-ed Flink committers can help to review and confirm!
>
>
>
>  One minor suggestion: Since the last section of design doc already touches
>  some new sql statements, shall we add another section in our doc and
>  formalize the new sql statements in SQL Client and TableEnvironment that
>  are gonna come along naturally with our design? Here are some that the
>  design doc mentioned and some that I came up with:
>
>  To be added:
>
> - USE  - set default catalog
> - USE  - set default schema
> - SHOW CATALOGS - show all registered catalogs
> - SHOW SCHEMAS [FROM catalog] - list schemas in the current default
> catalog or the specified catalog
> - DESCRIBE VIEW view

[jira] [Created] (FLINK-11247) Fix DESCRIBE command to support catalog tables

[Eron Wright (JIRA)]

Eron Wright  created FLINK-11247:


 Summary: Fix DESCRIBE command to support catalog tables
 Key: FLINK-11247
 URL: https://issues.apache.org/jira/browse/FLINK-11247
 Project: Flink
  Issue Type: Sub-task
Reporter: Eron Wright 


When the {{DESCRIBE}} command is applied to a catalog table, it fails with an 
error:

{code}
Flink SQL> DESCRIBE nyc.TaxiRides;  

  
[ERROR] Could not execute SQL statement. Reason:

  
org.apache.flink.table.api.TableException: Table 'nyc.TaxiRides' was not found.
{code}

The reason appears to be that {{LocalExecutor}} calls 
{{TableEnvironment::scan}} with the fully-qualified table name as a parameter 
(e.g. {{scan("nyc.TaxiRides")}}) rather than with an array of components (e.g. 
{{scan("nyc", "TaxiRides")}}).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Apply for flink contributor permission

[Haibo Sun]

Hi guys,
Could anyone kindly give me contributor permission? My JIRA username is
sunhaibotb.

Thanks,
Haibo

subscribe to flink dev maillist

[Zhang Shaoquan]

subscribe to flink dev maillist