Re: Hive Source With Kerberos认证问题

2020-03-15 Thread Rui Li
那提交作业的用户跟你login的用户是同一个用户不?我怀疑是不是因为用户不同,所以metastore的client没有用到正确的credentials。 On Tue, Mar 10, 2020 at 6:42 PM 叶贤勋 wrote: > 在doAs方法中是可以的。我现在hive connector中操作hive涉及认证的代码都在doAs中执行,可以解决认证问题。 > 前面提到的stacktrace是用我们公司自己封装的hive-exec > jar打印出来的,所以跟源码对应不上,我用官网的hive-exec-2.1.1.jar也是有这个问题。 > > > 叶贤勋 >

Need help on timestamp type conversion for Table API on Pravega Connector

2020-03-15 Thread B.Zhou
Hi community, Pravega connector is a connector that provides both Batch and Streaming Table API implementation. We uses descriptor API to build Table source. When we plan to upgrade to Flink 1.10, we found the unit tests are not passing with our existing Batch Table API. There is a type

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-15 Thread Jingsong Li
Thanks Piotr and Yun for involving. Hi Piotr and Yun, for implementation, FLINK-14254 [1] introduce batch sink table world, it deals with partitions thing, metastore thing and etc.. And it just reuse Dataset/Datastream FileInputFormat and FileOutputFormat. Filesystem can not do without

Re: [DISCUSS] FLIP-115: Filesystem connector in Table

2020-03-15 Thread Jingsong Li
Thanks Piotr and Yun for involving. Hi Piotr and Yun, for implementation, FLINK-14254 [1] introduce batch sink table world, it deals with partitions thing, metastore thing and etc.. And it just reuse Dataset/Datastream FileInputFormat and FileOutputFormat. Filesystem can not do without

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

2020-03-15 Thread Becket Qin
Hi Sidney, The WARN logging you saw was from the AbstractPartitionDiscoverer which is created by FlinkKafkaConsumer itself. It has an internal consumer which shares the client.id of the actual consumer fetching data. This is a bug that we should fix. As Rong said, this won't affect the normal

Re: How to change the flink web-ui jobServer?

2020-03-15 Thread Yang Wang
For K8s deployment(including standalone session/perjob, native integration), Flink could not support port range for rest options. You need to set the `rest.bind-port` exactly same with `rest.port`. Hi LakeShen I am trying to understand your problem and do not think it is about the port

[ANNOUNCE] Weekly Community Update 2020/11

2020-03-15 Thread Konstantin Knauf
Dear community, happy to share this week's community digest with an outlook on Apache Flink 1.11 & 1.10.1, an update on the recent development around Apache Flink Stateful Functions, a couple of SQL FLIPs (planner hints, Hbase catalog) and a bit more. Flink Development == *

Re: KafkaConsumer keeps getting InstanceAlreadyExistsException

2020-03-15 Thread Rong Rong
We also had seen this issue before running Flink apps in a shared cluster environment. Basically, Kafka is trying to register a JMX MBean[1] for application monitoring. This is only a WARN suggesting that you are registering more than one MBean with the same client id "consumer-1", it should not

KafkaConsumer keeps getting InstanceAlreadyExistsException

2020-03-15 Thread Sidney Feiner
Hey, I've been using Flink for a while now without any problems when running apps with a FlinkKafkaConsumer. All my apps have the same overall logic (consume from kafka -> transform event -> write to file) and the only way they differ from each other is the topic they read (remaining kafka

Re: datadog http reporter metrics

2020-03-15 Thread Yitzchak Lieberman
Anyone? On Wed, Mar 11, 2020 at 11:23 PM Yitzchak Lieberman < yitzch...@sentinelone.com> wrote: > Hi. > > Did someone encountered problem with sending metrics with datadog http > reporter? > My setup is flink version 1.8.2 deployed on k8s with 1 job manager and 10 > task managers. > Every

a question on window trigger and delta output

2020-03-15 Thread kant kodali
Hi All, I set a transformation like this and my events in the stream have a sequential timestamp like 1,2,3, and I set the watermark to event time. myStream .keyBy(0) .timeWindow(Time.of(1000, TimeUnit.MILLISECONDS)) .aggregate(new myAggregateFunction())

Re: Use flink to calculate sum of the inventory under certain conditions

2020-03-15 Thread Jiawei Wu
Thanks Arvid and Kurt. That's very helpful discussion. Currently we will continue this with Lambda, but I'll definitely do a A-A test between Lambda and Flink for this case. Regards, Jiawei On Wed, Mar 11, 2020 at 5:40 PM Kurt Young wrote: > > The second reason is this query need to scan the