I wrote a simple helper class, the redis connection are initialized in
the constructor and there are
set and get methods to store and retreive values from your map
functions. If you find any better way
to do this please share :). I am using redis scala client.
object class RedisHelper
{
val re
Even thought there are multiple instance of map object transient value
object state is accessible across the object, so as the stream is flowing
in the value can be updated based on application logic.
On Fri, May 13, 2016 at 11:26 AM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com>
I don't think the valuestate defined in one map function is accessible in
other map function this is my understanding, also you need to be aware
there will be instance of map function created for each of your tuple in
your stream, I had a similar use case where I had to pass in some state
from one
expected.
On Wed, May 11, 2016 at 11:42 PM, Aljoscha Krettek
wrote:
> Hi,
> are you per change using Kafka 0.9?
>
> Cheers,
> Aljoscha
>
> On Tue, 10 May 2016 at 08:37 Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> Robert,
>> Re
ailing list
> already.
>
> – Ufuk
>
> On Fri, May 6, 2016 at 12:07 PM, Balaji Rajagopalan
> wrote:
> > Thanks Robert appreciate your help.
> >
> > On Fri, May 6, 2016 at 3:07 PM, Robert Metzger
> wrote:
> >>
> >> Hi,
> >>
I have a requirement where I want to do aggregation on one data stream
every 5 minutes, a different data stream every 1 minute. I wrote a example
code to test this out but the behavior is different from what I expected ,
I expected the window2 to be called 5 times, and window 1 to called once ,
but
to the current offset ("earliest" is the opposite).
>
> How heavy is the processing you are doing? 4500 events/second sounds not
> like a lot of throughput.
>
> On Wed, May 4, 2016 at 8:24 AM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> I
What is the flink documentation you were following to set up your cluster ,
can you point to that ?
On Tue, May 3, 2016 at 6:21 PM, Punit Naik wrote:
> Hi
>
> I did all the settings required for cluster setup. but when I ran the
> start-cluster.sh script, it only started one jobmanager on the ma
I am using the flink connector to read from a kafka stream, I ran into the
problem where the flink job went down due to some application error, it was
down for sometime, meanwhile the kafka queue was growing as expected no
consumer to consume from the given group , and when I started the flink it
s
gt; Cheers,
> Till
>
> On Tue, Apr 19, 2016 at 2:01 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> The are two files in the /usr/share/flink/conf directory, and I was
>> trying to do the rolling of application logs which goes to following
>
Leader: 0 Replicas: 0 Isr: 0
On Wed, Apr 20, 2016 at 10:35 PM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com> wrote:
> /usr/share/kafka_2.11-0.8.2.1/bin$ ./kafka-topics.sh --describe --topic
> capi --zookeeper (someserver)
>
> Topic:capi PartitionCount:1 Replicati
.0 and 0.8.2.1 and for both versions
> everything worked fine.
> How many partitions does your topic have?
>
> Can you send me the full logs of the Kafka consumer?
>
> On Tue, Apr 19, 2016 at 6:05 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>&
issue?
>
> On Tue, Apr 19, 2016 at 2:06 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> It does not seem to fully work if there is no data in the kafka stream,
>> the flink application emits this error and bails, could this be missed use
>
In your pom.xml add the maven.plugins like this, and you will have to add
all the dependent artifacts, this works for me, if you fire mvn clean
compile package, the created jar is a fat jar.
org.apache.maven.plugins
maven-dependency-plugin
2.9
fail on a leader
> change, then Flink will restart the Kafka consumers and they'll find the
> new leaders.
> Starting from Flink 1.0.0 the Kafka consumer will handle leader changes
> without failing.
>
> Regards,
> Robert
>
> On Tue, Apr 19, 2016 at 12:17 PM, Balaji R
The are two files in the /usr/share/flink/conf directory, and I was trying
to do the rolling of application logs which goes to following directory in
task nodes.
/var/log/hadoop-yarn/containers/application_*/container_*/taskmanager.log
out err
Changing the logback.xml and logback-yarn.xml has no
I am facing this exception repeatedly while trying to consume from kafka
topic. It seems it was reported in 1.0.0 and fixed in 1.0.0, how can I be
sure that is fixed in the version of flink that I am using, does it require
me to install patch updates ?
Caused by: java.lang.RuntimeException: Unabl
I had fought with 0.8.0.2 kafka and flink 0.10.2 scala version 2.11, was
never able to get it working confounded with noclassdeffounderror, moved to
flink 1.0.0 with kafka 0.8.0.2 scala version 2.11 things worked for me, if
moving to flink 1.0.0 is an option for you do so.
balaji
On Mon, Apr 18,
Answered based on my understanding.
On Mon, Apr 18, 2016 at 8:12 AM, Prez Cannady
wrote:
> Some background.
>
> I’m running Flink application on a single machine, instrumented by Spring
> Boot and launched via the Maven Spring Boot plugin. Basically, I’m trying
> to figure out how much I can squ
Not a solution for your problem,but an alternative, I wrote my own sink
function where I handle all sql activities(insert/update/select), used a
3rd lib for connection pooling, the code has been running stable in
production without any issue.
On Fri, Apr 15, 2016 at 1:41 PM, Maximilian Bode <
maxi
the non-matching records after 5
> minutes into redis? Does flink give you control on which records is not
> matching in the current window such that you can copy into a long-term
> storage?
>
>
>
> On Wed, Apr 13, 2016 at 11:20 PM, Balaji Rajagopalan <
> balaji.rajagopa..
You can implement join in flink (which is a inner join) the below mentioned
pseudo code . The below join is for a 5 minute interval, yes will be some
corners cases when the data coming after 5 minutes will be missed out in
the join window, I actually had solved this problem but storing some data
i
cause in the
> stack trace it's written that you access a RedisClientPool from the
> DriverStreamHelper.set method.
>
> Cheers,
> Till
>
>
> On Wed, Apr 6, 2016 at 11:42 AM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> Till,
>&g
ure here.
}
}
On Wed, Apr 6, 2016 at 2:42 PM, Till Rohrmann wrote:
> Hi Balaji,
>
> from the stack trace it looks as if you cannot open a connection redis.
> Have you checked that you can access redis from all your TaskManager nodes?
>
> Cheers,
> Till
>
> On Wed, Apr
I am trying to use AWS EMR yarn cluster where the flink code runs, in one
of apply window function, I try to set some values in redis it fails. I
have tried to access the same redis with no flink code and get/set works,
but from the flink I get into this exception. Any inputs on what might be
goin
In your pom file you can mention against which version of scala you want to
build and also remember to add the scala version in the artifactId in all
the dependencies which takes scala version, there might be some libraries
which are scala agnostic there you do not have to specify the scala
version
Nice write up, one question though my understanding of keyed stream is that
it will fork n number of streams from one stream based on n keys, if that
is true it can be pictorially depicted as well and the apply function will
can be shown to operate over the time period by clearly marking a time
lin
gt; 'org.apache.kafka:kafka-clients:0.9.0.0',
>> 'org.apache.flink:flink-java:1.0.0',
>> 'org.apache.flink:flink-streaming-java_2.10:1.0.0',
>> 'org.apache.flink:flink-connector-kafka-0.9_2.10:1.0.0',
>> 'org.apache.flink:flink-connector
I had a similar use case and ended writing the aggregation logic in the
apply function, could not find any better solution.
On Fri, Apr 1, 2016 at 6:03 AM, Kanak Biscuitwala
wrote:
> Hi,
>
> I would like to write something that does something like a word count, and
> then emits only the 10 highe
I have tested kafka 0.8.0.2 with flink 1.0.0 and it works for me. Can't
talk about kafka 0.9.0.1.
On Wed, Mar 30, 2016 at 12:51 PM, Zhun Shen wrote:
> Hi there,
>
> flink version: 1.0.0
> kafka version: 0.9.0.0
> env: local
>
> I run the script below:
> ./bin/flink run -c com.test.flink.FlinkTes
You will have to include dependent jackson jar in flink server lib folder,
or create a fat jar.
balaji
On Tue, Mar 29, 2016 at 4:47 PM, Bart van Deenen
wrote:
> Hi all
>
> I've succesfully built a Flink streaming job, and it runs beautifully in
> my IntelliJ ide, with a Flink instance started o
val stream:DataStream[String] = env
.addSource(new FlinkKafkaConsumer08[String]("topic_name", new
SimpleStringSchema, prop))
val event:DataStream[SomeEventObj] = stream.map(MyMapFunction)
val tenMinute:DataStream[AggEvents] =
ridesByDeviceIdStream.timeWindowAll(Time.of(10,
TimeUnit.MINUTES).tri
Never mind Till figured out a way, instead of doing the aggregation in
reduce, I moved that logic to apply of the window function.
On Thu, Mar 24, 2016 at 11:33 PM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com> wrote:
> Till,
>
> Thanks for your reply, may be I should
ractical, Flink emits the partial
> reduce results.
>
> Cheers,
> Till
>
> On Thu, Mar 24, 2016 at 6:21 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> I have keyed input stream on DateStream(String,Int) and wrote a reduce on
>> the ke
I have keyed input stream on DateStream(String,Int) and wrote a reduce on
the keyedStream. The reduce is simple one summing up the integer values of
the same key.
val stream = DataStream(String,Int)
val keyedStream = stream.keyBy(_._1).reduce( new MyReduceFunction)
keyedStream.print()
class MyRed
Never mind I understand what is going on Aljoscha for each unique key the
value count is reset to 0.
On Wed, Mar 23, 2016 at 4:37 PM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com> wrote:
> (Booking(te7uc4,compact,j...@gmail.com,Mon Feb 29 19:19:40 IST
> 2016),145873098,1
>
> Cheers,
> Aljoscha
> > On 23 Mar 2016, at 11:53, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
> >
> > I wrote the below code which will increment a counter for the data in
> the datastream, and when I print the counter each time it seems the
I wrote the below code which will increment a counter for the data in the
datastream, and when I print the counter each time it seems the value is
reinitialised to 0, and it is not incrementing, any thoughts.
class BookingCntFlatMapFunction extends
RichFlatMapFunction[(Booking,Long,Long),(Booking,
This kind of class not found exception is a little bit misleading, it is
not the class is not found is the real problem rather than the combination
of the different libraries that are using there is a version compatibility
mismatch, so you will have to go back and check if there is any version
mism
The easier way to debug this would be have prints in the
projectjoinresultmapper and see what data you are getting. It is possible
your original dataset has duplicate rows ?
On Thu, Mar 17, 2016 at 6:36 PM, Lydia Ickler
wrote:
> Hi,
> I have a question regarding the Delta Iteration.
> I basicall
If it is a one time thing, you could run the kafka consumer script which
has the --from-beginning option, and redirect the output as a socketstream
and consume in flink.
On Thu, Mar 17, 2016 at 7:24 PM, Till Rohrmann wrote:
> Hi Dominique,
>
> have you tried setting the Kafka property props.put(
There was a similar question before the answer was to use
org.apache.flink.api.common.io.OutputFormat to do the conversion.
On Tue, Mar 15, 2016 at 7:48 PM, subash basnet wrote:
> Hello all,
>
> In WikipediaAnalysis.java we get *result *of type *DataStream Long>>*,
>
> I would want to convert *r
Robert,
I got it working for 1.0.0.
balaji
On Mon, Mar 14, 2016 at 4:41 PM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com> wrote:
> Yep the same issue as before(class not found) with flink 0.10.2 with
> scala version 2.11. I was not able to use scala 2.10 since c
Yep the same issue as before(class not found) with flink 0.10.2 with scala
version 2.11. I was not able to use scala 2.10 since connector for
flink_connector_kafka for 0.10.2 is not available.
balaji
On Mon, Mar 14, 2016 at 4:20 PM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com>
or-kafka_ doesn't exist for 1.0.0. You have to use either
> flink-connector-kafka-0.8_ or flink-connector-kafka-0.9_
>
>
> On Mon, Mar 14, 2016 at 11:17 AM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> What I noticied was that, if I remove the dep
What I noticied was that, if I remove the dependency on
flink-connector-kafka so it is clearly to do something with that
dependency.
On Mon, Mar 14, 2016 at 3:46 PM, Balaji Rajagopalan <
balaji.rajagopa...@olacabs.com> wrote:
> Robert,
>I have moved on to latest version of fl
n you send me the full build file to further investigate the issue?
>
> On Fri, Mar 11, 2016 at 4:56 PM, Balaji Rajagopalan <
> balaji.rajagopa...@olacabs.com> wrote:
>
>> Robert,
>> That did not fix it ( using flink and connector same version) . Tried
>>
from the
> "org.apache.flink" group.
>
> You said these are the versions you are using:
>
> flink.version = 0.10.2
> kafka.verison = 0.8.2
> flink.kafka.connection.verion=0.9.1
>
> For the connector, you also need to use 0.10.2.
>
>
>
> On Fri, Mar 1
I am tyring to use the flink kafka connector, for this I have specified the
kafka connector dependency and created a fat jar since default flink
installation does not contain kafka connector jars. I have made sure that
flink-streaming-demo-0.1.jar has the
kafka.javaapi.consumer.SimpleConsumer.class
You could try this link.
https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/examples.html
On Fri, Mar 11, 2016 at 9:56 AM, janardhan shetty
wrote:
> Hi,
>
> I was looking at the examples for Flink applications and the comment in
> quickstart/job results in 404 for the web page.
>
You could I suppose write the dateset to a sink a file and then read the
file to a data stream.
On Fri, Mar 11, 2016 at 4:18 AM, Prez Cannady
wrote:
>
> I’d like to pour some data I’ve collected into a DataSet via JDBC into a
> Kafka topic, but I think I need to transform my DataSet into a DataS
def main(args: Array[String]): Unit = {
val env: StreamExecutionEnvironment =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
try {
val data1:DataStream[String] = env.readTextFile(“somefile.txt”);
data1.print()
52 matches
Mail list logo
