User-based authentication in Flink

Hello Folks, I need to use user-based authentication in Flink Dashboard for multi-tenancy. I read the document, however, I could not found how to enable the authentication. If authentication is not supported currently, I would like to contribute to the community. How can I proceed it?

Re: Broadcast Config through Connected Stream

Hi, Any suggestions on this could be achieved? Thanks On Thu, Sep 7, 2017 at 8:02 AM, Navneeth Krishnan wrote: > Hi All, > > Any suggestions on this would really help. > > Thanks. > > On Tue, Sep 5, 2017 at 2:42 PM, Navneeth Krishnan < > reachnavnee...@gmail.com>

Re: Queryable State

Hi, Any idea on how to solve this issue? Thanks On Wed, Sep 13, 2017 at 10:12 AM, Navneeth Krishnan < reachnavnee...@gmail.com> wrote: > Hi, > > I am sure I have provided the right job manager details because the > connection timeout ip is the task manager where the state is kept. I guess >

Re: Keyed function type erasure problem.

Thanks -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

StreamCorruptedException

I am running Flink 1.3.0 against Kafka 0.10. I managed to bring the flink cluster up and have been running my flink CEP job for more than 3 hours when I see the following exception : The messages consumed from Kafka are protobuf messages and I use a protobuf serializer. i have no clue as to where

Re: Streaming API has a long delay at the beginning of the process.

Hi, Fabian > If I understand you correctly, the problem is only for the first events > that are processed. Yes. More Precisely, first 300 kafka-messages. > AFAIK, Flink lazily instantiates its operators which means that a source > task starts to consume records from Kafka before the subsequent

Re: NoResourceAvailable exception

P.S.: I tried on my laptop with the same configuration of the job-task manager (ram, slots, parallelism etc...) and it works perfectly. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: NoResourceAvailable exception

Update. the previous error probably was caused because I didn't restart the cluster before a re-execution. (maybe) Then, I tried to execute the program on a cluster of one node on my laptop and, after solved some little issues, everything works fine. Now I'm trying to deploy the same jar on the

Re: NoResourceAvailable exception

The error message says that the total number of slots is 0, It is thus very likely that no task manager is connected to the jobmanager. How exactly are you starting the cluster? On 14.09.2017 18:03, AndreaKinn wrote: Hi, I'm executing a program on a flink cluster. I tried the same on a local

Re: Flink flick cancel vs stop

I too am curious about stop vs cancel. I'm trying to understand the motivations a bit more. The current behavior of stop is basically that the sources become bounded, leading to the job winding down. The interesting question is how best to support 'planned' maintenance procedures such as app

Re: Cassandra Connector Problem (Possible Guava Conflict?)

Hi, I have the same problem but trying your solution so substituting this: org.apache.maven.plugins maven-shade-plugin 2.4.1

Re: Eception while running Table API job

Sorry..the error was caused by the fact that I've moved the table jar only on the job manager machine. After copying the jar from opt to lib in all the other TM machines the job was able to continue! Best, Flavio On Thu, Sep 14, 2017 at 6:56 PM, Flavio Pompermaier wrote:

Eception while running Table API job

Hi to all, I've tested my Flink 1.3.1 on my local machine and everything was fine. So I've tried to run it on the cluster but I've got the following weird exception (I've already moved flink-table_2.10-1.3.1.jar from opt to lib): Caused by: java.lang.RuntimeException: The initialization of the

NoResourceAvailable exception

Hi, I'm executing a program on a flink cluster. I tried the same on a local node with Eclipse and it worked fine. To start, following Flink recommendations on the cluster I set numberOfTaskSlots equals to the Cpu cores (2) while I set parallelism to 1. Unfortunately when I try to execute I obtain

Re: Can't start cluster

SOLVED using binaries. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Change Kafka cluster without loosing the Flink's state

Hi Gordon, Thanks again for your answer. But I am not sure if I understood this part: "The workaround, for now, would be to explicitly disable chaining of the consumer source with any stateful operators before taking the savepoint and changing the operator UID." So my code looks like this:

Taskmanager unable to rejoin job manager

Hi all, I am having an issue where one of our task managers that is running in high availability mode is timing out on the connection to zookeeper. This is causing it to retry the connection to zookeeper, which succeeds. The issue is once the taskmanager is back connected to zookeeper it is

Re: Table API and registration of DataSet/DataStream

I see...anyway for me it continue to be very misleading to have different syntax for where clauses (SQL vs scala)... Why not make them compatible? Is it that complex? On Thu, Sep 14, 2017 at 4:26 PM, Fabian Hueske wrote: > Hi Flavio, > > 1) The Java Table API does not aim to

Re: Flink doesn't free YARN slots after restarting

Sorry for my late answer Bowen, I think this only works if you implement your own WindowAssigner. With the built-in sliding window this is not possible since all windows have the same offset. Cheers, Till On Fri, Aug 25, 2017 at 9:44 AM, Bowen Li wrote: > Hi Till, >

Re: Table API and registration of DataSet/DataStream

Hi Flavio, 1) The Java Table API does not aim to resemble SQL but the Scala Table API which is integrated with the host language (Scala). Hence the different syntax for expressions. 2) Yes, that would be one way to do it. If that adds to much boilerplate code, you could encapsulate the code in

Re: Table API and registration of DataSet/DataStream

Hi Fabian, basically these were my problems with Table API. 1 ) Table.sql() has a different where syntax than Table.where() , and this is very annoying (IMHO). Ex: Table.sql("SELECT * FROM XXX WHERE Y IS NOT NULL) vs Table.i.where("Y.isNotNull"). 2) If I understood correctly, my program that

Re: Change Kafka cluster without loosing the Flink's state

Simply like this: env.addSource(new FlinkKafkaConsumer(...)).uid(“some-unique-id”) The same goes for any other operator. However, do keep in mind this bug that was just recently uncovered:  https://issues.apache.org/jira/browse/FLINK-7623. What I described in my previous reply would not work as

Dot notation not working for accessing case classes nested fields

Hi, I have the following case classes: case class Event(instantValues: InstantValues) case class InstantValues(speed: Int, altitude: Int, time: DateTime) in a DataStream[Event] I'd like to perform a maxBy operation on the field time of instantValue for each event and according to the docs here

Re: Change Kafka cluster without loosing the Flink's state

Hi Gordon, Thanks for your quick reply. I have following consumer: jobConfiguration.getEnv().addSource( new FlinkKafkaConsumer010<>(properties.getProperty(TOPIC), deserializer, properties)); How can I set the UID for the consumer ? Thanks again for help! Regards, Konstantin -- Sent from:

Re: Table API and registration of DataSet/DataStream

Not sure what you mean by "translate a where clause to a filter function". Isn't that exactly what Table.filter(String condition) is doing? It translates a SQL-like condition (represented as String) into an operator that filter the Table. 2017-09-09 23:49 GMT+02:00 Flavio Pompermaier

Re: ETL with changing reference data

Hi Peter, in principle, joining the data stream with the reference data would be the most natural approach to enrich data. However, stream joins for the Table API / SQL are currently under development and not available yet. You can of course try to solve the issue manually using UDFs but this

Re: DataSet: partitionByHash without materializing/spilling the entire partition?

Hi Urs, thanks for the additional details and plans you provided. Regarding your questions: Q1: Does the moved sort affect the spilling behavior? Yes, it might. If you look at the the out-output.png plan, you'll notice that is says "Sort (combining)" whereas the two-output.png plan only says

Re: Empty state restore seems to be broken for Kafka source (1.3.2)

Good job for figuring this out! This certainly seems to explain our problems. Thanks! Gyula Aljoscha Krettek ezt írta (időpont: 2017. szept. 14., Cs, 14:46): > After a bit more digging I found that the "isRestored" flag doesn't work > correctly if there are operators

Re: Empty state restore seems to be broken for Kafka source (1.3.2)

After a bit more digging I found that the "isRestored" flag doesn't work correctly if there are operators chained to the sink that have state: https://issues.apache.org/jira/browse/FLINK-7623 Blocker issue for 1.3.3 and 1.4.0. Best, Aljoscha

Re: Can't start cluster

Just a question: >From download page I have to download binaries or source package? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Too few memory segments provided exception

If it might help anyone else, I had a similar issue when running my unit tests, I could solve it by increasing memory of sbt export SBT_OPTS="-Xmx3G -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -Xss1G" -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Event-time and first watermark

Coming back to this after a while. What is the place were you would have expected such a note? Unfortunately the documentation about watermarks and process function is a bit spread across the documentation. If you could point me to where you would expect it that would be very helpful. Best,

Re: Delay in Flink timers

Hi, Yes, execution of these methods is protected by a synchronized block. This is not a fair lock so incoming data might starve timer callbacks. What is the number of timers we are talking about here? Best, Aljoscha > On 11. Sep 2017, at 19:38, Chesnay Schepler wrote: > >

Re: Streaming API has a long delay at the beginning of the process.

Hi, If I understand you correctly, the problem is only for the first events that are processed. AFAIK, Flink lazily instantiates its operators which means that a source task starts to consume records from Kafka before the subsequent tasks have been started. That's why the latency of the first

Re: heap dump shows StoppableSourceStreamTask retained by java.lang.finalizer

Hi Steven, thanks for reporting this issue. Looping in Till who's more familiar with the task lifecycles. Thanks, Fabian 2017-09-12 7:08 GMT+02:00 Steven Wu : > Hi , > > I was using Chaos Monkey to test Flink's behavior against frequent killing > of task manager nodes. I

Re: How to user ParameterTool.fromPropertiesFile() to get resource file inside my jar

Hi, Are you using Maven to create the Jar or your IDE? I think this might be a problem only when creating the Jar via the IDE. Best, Aljoscha > On 11. Sep 2017, at 04:46, Tony Wei wrote: > > Hi Aljoscha, > > I found the root cause of my problem from this reference >

Re: Change Kafka cluster without loosing the Flink's state

Hi Konstantin, After migrating and connecting to the new Kafka cluster, do you want the Kafka consumer to start fresh without any partition offset state (and therefore will re-establish its partition-to-subtask assignments), while keeping all other operator state in the pipeline intact? If so,

Re: Keyed function type erasure problem.

Hi, The problem is that Flink cannot determine the key type produced by the KeySelector because the type is a generic. The type information is lost at runtime due to Java's type erasure. You can try to implement the ResultTypeQueryable interface with the KeySelector and tell Flink the key type

Change Kafka cluster without loosing the Flink's state

Hi guys, We have a running apache flink streaming job which interacts with apache kafka (consumer and producer). Now we would like to change the kafka cluster without loosing Flink's state. Is it possible to do it ? If yes, what is the right way to do it ? Thanks in advance! Best,

Re: question on sideoutput from ProcessWindow function

Hi, Chen is correct! I think it would be nice, though, to also add that functionality for ProcessWindowFunction and I think this should be easy to do since the interface is very similar to ProcessFunction and we could also add that to the Context. Best, Aljoscha > On 9. Sep 2017, at 06:22,

Re: got Warn message - "the expected leader session ID did not equal the received leader session ID " when using LocalFlinkMiniCluster to interpret scala code

Hi XiangWei, the problem is that the LocalFlinkMiniCluster can no longer be used in combination with a RemoteExecutionEnvironment. The reason is that the LocalFlinkMiniCluster uses now an internal leader election service and assigns leader ids to its components. Since this is an internal service

Re: Flink flick cancel vs stop

Hey Elias, sorry for the delay here. No, stop is not deprecated but not fully implemented yet. One missing part is migration of the existing source functions as you say. Let me pull in Till for more details on this. @Till: Is there more missing than migrating the sources? Here is the PR and

Re: Bulk Iteration

Hello Alieh, If you set the logging to a more verbose level, then Flink prints a log msg at every iteration. If you need the current iteration number inside your code, then you should create your UDF as an AbstractRichFunction, where you can call getIterationRuntimeContext(), which has