Hi,
it maintain itself a individual instance of
FlinkKafkaConsumerBase, and it contains a individual pendingOffsetsToCommit
, am right ?
That is correct! The FlinkKafkaConsumerBase is code executed for each parallel
subtask instance, and therefore have their own pendingOffsetsToCommit which
w
Hi,Piotr Nowojski
i think you are right, but i think it is executed in parallel, but in
each parallel , it maintain itself a individual instance of
FlinkKafkaConsumerBase, and it contains a individual pendingOffsetsToCommit
, am right ?
thanks, aitozi
--
View this message in context:
Hi,
I have few questions regarding event time windowing. My scenario is devices
from various timezones will send messages with timestamp and I need to
create a window per device for 10 seconds. The messages will mostly arrive
in order.
Here is my sample code to perform windowing and aggregating t
This was user induced problem - me. I wasn't calling streamenv.execute() :(
On Tue, Aug 1, 2017 at 1:29 PM, Mohit Anchlia
wrote:
> This doesn't work even with TextInputFormat. Not sure what's wrong.
>
> On Tue, Aug 1, 2017 at 9:53 AM, Mohit Anchlia
> wrote:
>
>> I don't see the print output.
>>
This doesn't work even with TextInputFormat. Not sure what's wrong.
On Tue, Aug 1, 2017 at 9:53 AM, Mohit Anchlia
wrote:
> I don't see the print output.
>
> On Tue, Aug 1, 2017 at 2:08 AM, Fabian Hueske wrote:
>
>> Hi Mohit,
>>
>> these are just INFO log statements that do not necessarily indic
Hello,
I am using flink 1.2 and writing records to S3 using rolling sink.
I am encountering this S3 write error quite frequently :
TimerException{com.amazonaws.services.s3.model.AmazonS3Exception: Status Code:
404, AWS Service: Amazon S3, AWS Request ID: B573887B1850BF28, AWS Error Code:
nu
Thanks that worked. However, what I don't understand is wouldn't the open
call that I am inheriting have this logic already inbuilt? I am inheriting
FileInputFormat.
On Tue, Aug 1, 2017 at 1:42 AM, Fabian Hueske wrote:
> An InputFormat processes multiple InputSplits. open() is called for each
>
I want a count of events that are put into a bucketing sink, but can't find
a ready-made way of doing that. Is there an easier way than to implement a
counter for each bucket via the metrics? If metrics counters is the easy
way, what do I do to make sure that I don't have a memory leak from expired
I don't see the print output.
On Tue, Aug 1, 2017 at 2:08 AM, Fabian Hueske wrote:
> Hi Mohit,
>
> these are just INFO log statements that do not necessarily indicate a
> problem.
> Is the program working otherwise or do you observe other problems?
>
> Best, Fabian
>
> 2017-08-01 0:32 GMT+02:00
Hi Yann,
I'm not aware of anyone that has started work on this. I'm quite interested in
how this turns out for you.
Best,
Aljoscha
> On 20. Jul 2017, at 14:49, Yann Pauly wrote:
>
> Hi all,
>
> We want to integrate our Flink instances with our Azkaban scheduler.
> For that we will have to cr
Just some clarification: Flink state is never shared between different parallel
operator instances. If you want to make those changes available to all parallel
instances of the operation you have to broadcast the control stream, i.e.
control.broadcast().
Best,
Aljoscha
> On 24. Jul 2017, at 17
Cool, good to hear!
It is one of those "it a feature, not a bug" situations ;-)
Flink's HA mode supports multiple masters, so the CLI needs to have a way
to find which master is "leader" (active, versus the passive masters on
standby). That discovery goes through ZooKeeper as well (which is the
g
Hi,
In your original program, the problem is that there is both an ERROR and
SUCCESS event in your List. Thus you add both "success" and "error" to the list
of split outputs. To discern between those different types you first have to
flatten that DataStream> into a DataStream using a flatMap().
Hi,
I think the problem here is that the SerializationUtils don't use the correct
ClassLoader when deserialising. When running a Flink job on a cluster the user
code (from the user-code jar) is only available in the user-code ClassLoader.
If you can pass a ClassLoader to your deserialize method
Hi Fabian,
I am not really sure using CoProcessFunction would be useful for my use
case. My use case, in short, can be explained as follows:
1) create 2 different local state store, where both have 1-N relationship.
For eg. 1 -> [A,B,C] and A -> [1,2,3]
2) Based on the key A, get list of element
"No, those are two different queries. "
This is enough. The second part does not applies since i'm calculating
EventTime from table source.
Thanks
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SQL-API-with-Table-API-tp14599p14605.html
Sen
No, those are two different queries.
The second query would also not work because the revenue table does not
have the EventTime attribute.
Best, Fabian
2017-08-01 13:03 GMT+02:00 nragon :
> Hi,
>
> Can i expect the output from this:
>
> Table revenue = tableEnv.sql(
> "SELECT TUMBLE_START(E
Hi,
pendingOffsetsToCommit is a private field which is not accessed from outside of
the FlinkKafkaConsumerBase class. It is only used in state manipulation
methods, which are not executed in parallel.
Thanks, Piotrek
> On Aug 1, 2017, at 1:20 PM, aitozi wrote:
>
> Hello:
>
>i am new to
Let's move the discussions to FLINK-7330.
On 01.08.2017 13:15, Chesnay Schepler wrote:
One problem i know of is that windows paths with a scheme are not
detected as windows paths, as documented in FLINK-6889.
They generally still work though (/maybe /by chance).
I just verified that calling Fi
Hello:
i am new to Flink, ijust read the source code . i am doubt that , why in
FlinkKafkaConsumerBase.java (version1.2), like method :
notifyCheckpointComplete may change the pendingOffsetsToCommit in parallel
, why dont need to be surrouned with "synchronized"
thanks
--
View this me
One problem i know of is that windows paths with a scheme are not
detected as windows paths, as documented in FLINK-6889.
They generally still work though (/maybe /by chance).
I just verified that calling FileInputFormat#setFilePath() works for
both "file:///" and "file:/" on Windows.
(I'm assu
Hi,
Can i expect the output from this:
Table revenue = tableEnv.sql(
"SELECT TUMBLE_START(EventTime, INTERVAL '30' MINUTE) as tStart, " +
"TUMBLE_END(EventTime, INTERVAL '30' MINUTE) as tEnd, "
+
"cID, " +
"cName, " +
Hi,
flink currently only seems to support a single kerberos ticket for
deployment. Are there plans to support different users per each job?
regards,
Georg
Hey! It´s working now!!
I will do a summary for those who might have the same problem in the future:
- Flink 1.3.0 dockerized on Mesos:
- Add the HA configuration values in your flink app:
https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/jobmanager_high_availability.html#config-
The average would be computed over the aggregated 15-minute count values.
The sliding window would emit every 15 minutes the average of all records
that arrived within the last 6 hours.
Since the preceding 15-minute tumbling window emits 1 record every 15 mins,
this would be the avg over 24 records
Hi Mohit,
these are just INFO log statements that do not necessarily indicate a
problem.
Is the program working otherwise or do you observe other problems?
Best, Fabian
2017-08-01 0:32 GMT+02:00 Mohit Anchlia :
> I even tried existing format but still same error:
>
> FileInputFormat fileInputFo
Hi Basanth,
Flink is not a storage system (neither for stream nor for batch data).
Flink applications can be stateful and maintain very large state (in the
order of several TBs) but state is always associated with an application.
State can be queryable, so outside applications can run key-lookup (
An InputFormat processes multiple InputSplits. open() is called for each
InputSplit.
If you don't reset reached to false in open() you will only read a single
(i.e., the first) InputSplit and skip all others.
I'd override open as follows:
public void open(FileInputSplit fileSplit) throws IOExcept
Hi,
there is no built-in support for key changes.
You might be able to feedback a changed key with an iteration edge, but not
sure how well that works.
Best, Fabian
2017-08-01 7:32 GMT+02:00 Govindarajan Srinivasaraghavan <
govindragh...@gmail.com>:
> Hi,
>
> I have a keyby state but the key ca
Hi,
in principle you can use any data type with Flink including byte[].
However, all of your functions need the logic to interpret the bytes and
you have to implement custom key extractors (if you need to keyBy or
partition your stream).
Best, Fabian
2017-08-01 2:09 GMT+02:00 Basanth Gowda :
>
Hi Stephan,
So, do you mean to remove the “-m” param from the flink CLI call? And on the
other hand, that I should add the Zookeeper configuration in both sides, the
remote flink and locally in the flink CLI config, right?
Regards
On 31 Jul 2017, at 22:21, Stephan Ewen
mailto:se...@apache.or
Hi Peter,
this kind of use case is supported, but it is best practice to split
independent pipelines into individual jobs.
One reason for that is to isolate failures and restarts.
For example, I would split the program you posted into two programs, one
for the "foo" topic and one of the "bar" topi
32 matches
Mail list logo
