Yeah, that's why I added some test code in the window() to call store.all()
and iterate through. I traced into it in my local environment and verified
that the iterator is functioning with store.all().
-Yi
On Thu, Jul 23, 2015 at 4:26 PM, Shekar Tippur ctip...@gmail.com wrote:
Yi,
In my
Yi,
In my case, I am able to append to the key but I am not able to get the
store and iterate through.
If you look at http://pastebin.com/fKGpHwW6, line 146, I am able to get the
store value. but in window routine - line 187, I am unable to get the
values from store.
- Shekar
Yi,
I am not sure I see an attachment. Is it possible to paste that on
pastebin?
Shekar
On Jul 21, 2015 4:27 PM, Yi Pan nickpa...@gmail.com wrote:
Hi, Shekar,
I have strip down your use case just to the KV-store operation and have
verified that it works fine. Please see the attached diff
Yi,
I am not sure if attachments gets filtered when sent to the group. I have
seen this earlier as well.
- Shekar
Hi, Shekar,
Here it is: http://pastebin.com/fKGpHwW6
-Yi
On Wed, Jul 22, 2015 at 8:05 AM, Shekar Tippur ctip...@gmail.com wrote:
Yi,
I am not sure I see an attachment. Is it possible to paste that on
pastebin?
Shekar
On Jul 21, 2015 4:27 PM, Yi Pan nickpa...@gmail.com wrote:
Hi,
Thanks Yi. I got the pastebin link.
I am looking at it.
Shekar
On Jul 22, 2015 5:09 PM, Yi Pan nickpa...@gmail.com wrote:
Hmm... did you get my pastebin post? If not, I can send the diff directly
to your gmail account.
On Wed, Jul 22, 2015 at 11:23 AM, Shekar Tippur ctip...@gmail.com wrote:
Hi, Shekar,
I have strip down your use case just to the KV-store operation and have
verified that it works fine. Please see the attached diff file.
If you have any further questions, please let me know.
-Yi
On Mon, Jul 20, 2015 at 12:35 PM, Shekar Tippur ctip...@gmail.com wrote:
Yi,
Here
Yi,
Thanks a lot.
- Shekar
Hi, Shekar,
It would also be helpful if you can post your job configuration on the
pastebin s.t. I can test the same config.
Thanks!
-Yi
On Mon, Jul 20, 2015 at 11:11 AM, Shekar Tippur ctip...@gmail.com wrote:
Yi,
Thanks a lot.
- Shekar
Any takers on this please?
- Shekar
Hi, Shekar,
If possible, could you share your code somewhere? I can try to dig into it
this weekend.
Thanks!
-Yi
On Fri, Jul 17, 2015 at 1:31 PM, Shekar Tippur ctip...@gmail.com wrote:
Any takers on this please?
- Shekar
Any takers on this please?
- Shekar
On Fri, Jul 3, 2015 at 9:46 AM, Shekar Tippur ctip...@gmail.com wrote:
Any answer on how to get all the kv values and reinitialise the kv store?
Had one more question on implementing sliding window.
If i use a kv store like rocksdb, and I use yarn (say 3
Any answer on how to get all the kv values and reinitialise the kv store?
Had one more question on implementing sliding window.
If i use a kv store like rocksdb, and I use yarn (say 3 node cluster), the
job that it runs to aggregate gets distributed as well and I am guessing
the aggregation
Hi, Shekar,
Sorry I was not able to follow up w/ you in time. It is great that you have
found the configure problem and made it work!
As for the exception on the iterator, could you send us the log w/ the
exception?
Thanks!
-Yi
On Thu, Jul 2, 2015 at 4:36 PM, Shekar Tippur ctip...@gmail.com
Yi,
Looks like it is working now. There was a redundant line in the config.
I am able to initialize kv store and add values.
In the window code, I am unable to retrieve them and mark them as 0.
Here is my window code:
public void window(MessageCollector collector,
TaskCoordinator
Yi,
There is no exception. I want to do couple of things in the window.
- Get all the keys and values and publish to another store (like graphite)
as a list
- Remove all entries.
I can iterate thro the list later but I want to be able to get all kv
values and delete all of them in an atomic
Also, next.getValue() or next.getKey() does not yield anything.
KeyValueIteratorString, String i = store.all();
while(i.hasNext()){
Entry String, String next = i.next();
log.info(Removed Key, next.getValue());
}
On Thu, Jul 2, 2015 at 5:36 PM, Shekar Tippur
Yi/Milinda,
I am trying to initialize a kv store. I have the following properties
defined:
stores.store-name.key.serde=json
stores.store-name.msg.serde=json
stores.store-name.changelog=argos.windowchangelog
How do I define a key serde as I am getting this exception:
Exception in thread main
Do you have
serializers.registry.json.class
=org.apache.samza.serializers.JsonSerdeFactory
in your config file?
Fang, Yan
yanfang...@gmail.com
On Wed, Jul 1, 2015 at 2:59 PM, Shekar Tippur ctip...@gmail.com wrote:
Yi/Milinda,
I am trying to initialize a kv store. I have the following
Yan,
yes. I do have it.
- Shekar
On Wed, Jul 1, 2015 at 3:09 PM, Yan Fang yanfang...@gmail.com wrote:
Do you have
serializers.registry.json.class
=org.apache.samza.serializers.JsonSerdeFactory
in your config file?
Fang, Yan
yanfang...@gmail.com
On Wed, Jul 1, 2015 at 2:59 PM,
So do you use the store-name as the kv storage name in your StreamTask
code?
Fang, Yan
yanfang...@gmail.com
On Wed, Jul 1, 2015 at 3:41 PM, Shekar Tippur ctip...@gmail.com wrote:
Yan,
yes. I do have it.
- Shekar
On Wed, Jul 1, 2015 at 3:09 PM, Yan Fang yanfang...@gmail.com wrote:
Do
I do have this in init as well...
public void init(Config config, TaskContext context) {
store = (KeyValueStoreString, Integer) context.getStore(store);
}
You are right. These are primitive types but I was trying to address this
exception:
Exception in thread main
Hi Shekar,
You can use Kafka's partitioning capabilities to partition your stream
based on application. That will make sure events related to a application
will always ended up in same partition. With this you will have multiple
applications in same partition and each partition will be mapped to
Shekar,
You won't be creating a partition per application. By using the application
name as the partitioning key you ensure all events for a given application
are consistently mapped to the same partition. Multiple applications will
be mapped to each partition without any need for a priori
Benjamin,
Thanks for the explanation. We dont have any specific partition scheme as
yet. We just have 2 topics - raw and processed and we use default
partitioning scheme.
Can you share any code snippet so I can understand it better?
- Shekar
Hi, Shekar,
First, I would like to clarify what you meant by sliding window: is it
defined as windows with size N and advance step size of 1 (which means that
windows overlap and each input message would contribute to multiple counts
in different windows)? Or windows with size N and advance step
Hi Shekar,
Please have a look at [1].
Milinda
[1] https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example
On Mon, Jun 29, 2015 at 1:06 PM, Shekar Tippur ctip...@gmail.com wrote:
Benjamin,
Thanks for the explanation. We dont have any specific partition scheme as
yet. We
Yi,
My use case is more of the latter. Your explanation makes sense now. I was
also looking into Milinda's wiki. She has a section for Kafka
partition SimplePartitioner, which is simple enough as well.
Thanks for all the inputs. Let me see what I come up with while
implementing it.
- Shekar
On
Milinda,
I see that the document you mentioned addresses windowing but I also need
to group by different applications.
ApplicationCount
---
A100
B40
C69
- Shekar
On Fri, Jun 26, 2015 at 11:39 AM,
Hello,
My apologies if I have raised it earlier.
Here is the use case:
I have a stream that is partitioned based on application name. I want to be
able to count hte number of events happening for that particular
application in the past 5 minutes (sliding window) and update either
another topic or
Never mind. I see it here:
http://samza.apache.org/learn/documentation/0.8/container/windowing.html
Thanks again Milinda.
- Shekar
On Fri, Jun 26, 2015 at 11:39 AM, Shekar Tippur ctip...@gmail.com wrote:
Thanks Milinda.
Is this feature available on 0.8 version of Samza?
- Shekar
On Fri,
Thanks Milinda.
Is this feature available on 0.8 version of Samza?
- Shekar
On Fri, Jun 26, 2015 at 11:24 AM, Milinda Pathirage mpath...@umail.iu.edu
wrote:
Hi Shekar,
You can use Samza's local storage (
http://samza.apache.org/learn/documentation/0.9/container/state-management.html
)
to
32 matches
Mail list logo
