Samza yarn application container is getting killed due to out of memory
error.
Wondering of you guys can give me some pointers on how to troubleshoot this
error.
Background: I had to change the Kafka log location and restart kafka. There
is no change in application or yarn configuration.
Hello,
I am looking for direction on implementing samza over Kinesis.
I see that jira ticket is in unresolved state.
https://issues.apache.org/jira/plugins/servlet/mobile#issue/SAMZA-489
I also saw that with Samza release 0.10 this implementation is in place.
Appreciate any pointers on this.
Just following on the same thread, is there a feature/bug to force the
jmxrmi port? This would be helpful to get the container metrics.
- Shekar
On Sat, Aug 6, 2016 at 6:14 PM, Shekar Tippur <ctip...@gmail.com> wrote:
> Thanks a lot guys. It was a system problem. /etc/hosts file
t; other details about your cluster you can provide would be useful.
>
> -Jake
>
> On Fri, Aug 5, 2016 at 7:33 AM, Shekar Tippur <ctip...@gmail.com> wrote:
>
> > Any pointers on this please. I am completely blocked.
> >
> > - Shekar
> >
> > On Thu, A
Any pointers on this please. I am completely blocked.
- Shekar
On Thu, Aug 4, 2016 at 5:05 PM, Shekar Tippur <ctip...@gmail.com> wrote:
> This server is not connected to vpn.
>
> - Shekar
>
This server is not connected to vpn.
- Shekar
2:23 PM, Yi Pan <nickpa...@gmail.com> wrote:
> Hi, Shekar,
>
> Please see my replies below:
>
> On Thu, Aug 4, 2016 at 12:18 PM, Shekar Tippur <ctip...@gmail.com> wrote:
>
> > Sorry .. got trigger happy.
> > Couple of observations:
> > 1. rmi
Sorry .. got trigger happy.
Couple of observations:
1. rmi port seem to be random ones Even with task.opts and setting the port
to 1099
2. Do I need to start rmiregistry manually?
3. I dont have any firewall settings.
On Thu, Aug 4, 2016 at 12:14 PM, Shekar Tippur <ctip...@gmail.com> wrote:
on? Could you also paste your
> configuration, especially task.opts?
>
> -Yi
>
> On Wed, Aug 3, 2016 at 5:56 PM, Shekar Tippur <ctip...@gmail.com> wrote:
>
> > I am trying to submit a Samza job to yarn and I get a error:
> >
> > Exception in
I am trying to submit a Samza job to yarn and I get a error:
Exception in thread "main" java.io.IOException: Cannot bind to URL
[rmi://localhost:44960/jmxrmi]: javax.naming.ServiceUnavailableException
[Root exception is java.rmi.ConnectException: Connection refused to host:
localhost; nested
Weird,
I see a response
http://mail-archives.apache.org/mod_mbox/samza-dev/201606.mbox/browser but
I dint get a email. Anyways, Jagadish - Thanks for responding.
Yes. We were able to produce a message via console producer and consume it.
- Shekar
On Thu, Jun 9, 2016 at 1:42 AM, Shekar Tippur
Looks like there is a Jira ticket with no conclusion
https://issues.apache.org/jira/browse/SAMZA-750
Is there a workaround without upgrading the code?
On Mon, Jun 13, 2016 at 11:25 PM, Shekar Tippur <ctip...@gmail.com> wrote:
>
>
> Hello,
>
> Running Samza 8.0 with
>
> Hello,
>
> Running Samza 8.0 with latest hadoop, I get the below exception. What do I
> need to change to get this to work?
>
> [root@bus01 yarn]# /usr/hdp/2.3.4.7-4/hadoop-yarn/bin/yarn version
>
> Hadoop 2.7.1.2.3.4.7-4
>
>
> Exception in thread "main" java.lang.IllegalArgumentException:
Hello,
Running Samza 8.0 with latest hadoop, I get the below exception. What do I
need to change to get this to work?
[root@bus01 yarn]# /usr/hdp/2.3.4.7-4/hadoop-yarn/bin/yarn version
Hadoop 2.7.1.2.3.4.7-4
Exception in thread "main" java.lang.IllegalArgumentException: Invalid
ContainerId:
I have a strange issue. We just replaced failed zookeeper server. We
killed the job in the interim and resubmitted the job to yarn.
The events were getting processed but at some point, the events stopped
processing. I have checked RM and application logs but they seem to be fine.
We then
Thanks Jordi. This really helps.
- Shekar
On Thu, Aug 6, 2015 at 12:21 AM, Jordi Blasi Uribarri jbl...@nextel.es
wrote:
Hi,
As a little present (and I know this is not the way to get the code in the
project, but I am new to this sharing). I just made a simple script to kill
all the jobs
Yi,
In my case, I am able to append to the key but I am not able to get the
store and iterate through.
If you look at http://pastebin.com/fKGpHwW6, line 146, I am able to get the
store value. but in window routine - line 187, I am unable to get the
values from store.
- Shekar
file.
If you have any further questions, please let me know.
-Yi
On Mon, Jul 20, 2015 at 12:35 PM, Shekar Tippur ctip...@gmail.com wrote:
Yi,
Here is the config:
http://pastebin.com/mCALEACs
- Shekar
On Mon, Jul 20, 2015 at 12:27 PM, Yi Pan nickpa...@gmail.com wrote:
Hi, Shekar
Yi,
I am not sure if attachments gets filtered when sent to the group. I have
seen this earlier as well.
- Shekar
Thanks Yi. I got the pastebin link.
I am looking at it.
Shekar
On Jul 22, 2015 5:09 PM, Yi Pan nickpa...@gmail.com wrote:
Hmm... did you get my pastebin post? If not, I can send the diff directly
to your gmail account.
On Wed, Jul 22, 2015 at 11:23 AM, Shekar Tippur ctip...@gmail.com wrote
Yi,
Thanks a lot.
- Shekar
Any takers on this please?
- Shekar
Any takers on this please?
- Shekar
On Fri, Jul 3, 2015 at 9:46 AM, Shekar Tippur ctip...@gmail.com wrote:
Any answer on how to get all the kv values and reinitialise the kv store?
Had one more question on implementing sliding window.
If i use a kv store like rocksdb, and I use yarn (say 3
numbers get skewed? Is that a right assessment?
On Thu, Jul 2, 2015 at 5:47 PM, Shekar Tippur ctip...@gmail.com wrote:
Also, next.getValue() or next.getKey() does not yield anything.
KeyValueIteratorString, String i = store.all();
while(i.hasNext()){
Entry String, String next
();
}
eventsSeen = 0;
i.close();
}
How do I retrieve the key and is there a way to remove it? i.remove throws
an exception.
- Shekar
On Wed, Jul 1, 2015 at 7:25 PM, Shekar Tippur ctip...@gmail.com wrote:
Yi,
Here is my config file:
http://pastebin.com/Kf3C9E0h
- S
on the iterator, could you send us the log w/ the
exception?
Thanks!
-Yi
On Thu, Jul 2, 2015 at 4:36 PM, Shekar Tippur ctip...@gmail.com wrote:
Yi,
Looks like it is working now. There was a redundant line in the config.
I am able to initialize kv store and add values.
In the window code
Also, next.getValue() or next.getKey() does not yield anything.
KeyValueIteratorString, String i = store.all();
while(i.hasNext()){
Entry String, String next = i.next();
log.info(Removed Key, next.getValue());
}
On Thu, Jul 2, 2015 at 5:36 PM, Shekar Tippur ctip
)
at org.apache.samza.job.JobRunner.run(JobRunner.scala:62)
at org.apache.samza.job.JobRunner$.main(JobRunner.scala:37)
at org.apache.samza.job.JobRunner.main(JobRunner.scala)
On Mon, Jun 29, 2015 at 12:41 PM, Shekar Tippur ctip...@gmail.com wrote:
Yi,
My use case is more of the latter. Your explanation makes sense now. I
, Shekar Tippur ctip...@gmail.com wrote:
Yi/Milinda,
I am trying to initialize a kv store. I have the following properties
defined:
stores.store-name.key.serde=json
stores.store-name.msg.serde=json
stores.store-name.changelog=argos.windowchangelog
How do I define a key serde as I am
I do have this in init as well...
public void init(Config config, TaskContext context) {
store = (KeyValueStoreString, Integer) context.getStore(store);
}
You are right. These are primitive types but I was trying to address this
exception:
Exception in thread main
Benjamin,
Thanks for the explanation. We dont have any specific partition scheme as
yet. We just have 2 topics - raw and processed and we use default
partitioning scheme.
Can you share any code snippet so I can understand it better?
- Shekar
when next time the application
counter is updated again)
Hope that makes sense.
-Yi
On Mon, Jun 29, 2015 at 10:06 AM, Shekar Tippur ctip...@gmail.com wrote:
Benjamin,
Thanks for the explanation. We dont have any specific partition scheme as
yet. We just have 2 topics - raw
, Shekar Tippur ctip...@gmail.com wrote:
Never mind. I see it here:
http://samza.apache.org/learn/documentation/0.8/container/windowing.html
Thanks again Milinda.
- Shekar
On Fri, Jun 26, 2015 at 11:39 AM, Shekar Tippur ctip...@gmail.com wrote:
Thanks Milinda.
Is this feature available on 0.8
Hello,
My apologies if I have raised it earlier.
Here is the use case:
I have a stream that is partitioned based on application name. I want to be
able to count hte number of events happening for that particular
application in the past 5 minutes (sliding window) and update either
another topic or
Never mind. I see it here:
http://samza.apache.org/learn/documentation/0.8/container/windowing.html
Thanks again Milinda.
- Shekar
On Fri, Jun 26, 2015 at 11:39 AM, Shekar Tippur ctip...@gmail.com wrote:
Thanks Milinda.
Is this feature available on 0.8 version of Samza?
- Shekar
On Fri
window state maintenance. You have to
implement it on top of local storage or if you don't won't fault tolerance
you can keep the state in-memory too (as long as the state fit in memory).
Thanks
Milinda
On Fri, Jun 26, 2015 at 1:53 PM, Shekar Tippur ctip...@gmail.com wrote:
Yan
.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Tue, Jun 23, 2015 at 12:40 PM, Shekar Tippur ctip...@gmail.com wrote:
Yan,
I have restarted Yarn but I still see the same issue. I will post this
question on Yarn mailing list.
yarn-...@hadoop.apache.org
- Shekar
Yan,
I have restarted Yarn but I still see the same issue. I will post this
question on Yarn mailing list.
yarn-...@hadoop.apache.org
- Shekar
This is what I see on Yarn monitoring page:
As we can see, there are 9998 apps pending. There is some 10k limit we are
hitting. I see only 1 app running.
Apps SubmittedApps PendingApps RunningApps CompletedContainers RunningMemory
UsedMemory TotalMemory ReservedVCores UsedVCores TotalVCores
Wang,
I have not caught up but can you please highlight if there are any feature
additions as well?
- Shekar
On Tue, Jun 16, 2015 at 10:14 AM, Guozhang Wang wangg...@gmail.com wrote:
Hi all,
We have been running a couple of our jobs against `0.9.1` branch last week
at LinkedIn with some
+1
On Jun 16, 2015 6:39 PM, Percy Wegmann percy.wegm...@evariant.com wrote:
Thank you!
Sent using CloudMagic
https://cloudmagic.com/k/d/mailapp?ct=picv=6.0.64pv=8.2
On Tue, Jun 16, 2015 at 8:11 PM, Chris Riccomini criccom...@apache.org
wrote:
+1 Here.
On Tue, Jun 16, 2015 at 12:01 PM,
Hello,
I have a scenario where for each incoming message, I see 3 corresponding
processed message.
I am suspecting that I may have submitted the same job multiple number of
times to Yarn.
If this is the case,
a. How do I detect it
b. How do I kill redundant jobs?
Could there be other reasons
Hello,
After submitting Samza job to Yarn, I see a lot of jobs in accepted state.
Please see the attached screenshot. Wondering if this is due to any missed
setting. I see the jobs progressing but would it cause any harm?
- Shekar
runs successfully, can you see the Samza Application
run in the Yarn UI (such as, localhost:8088?) 3) if can not see it, what
does Samza's log say?
Thanks,
Fang, Yan
yanfang...@gmail.com
On Thu, May 14, 2015 at 3:31 PM, Shekar Tippur ctip...@gmail.com
wrote:
Yan,
I have
you paste the entire stacktrace/log? Where there any other errors ?
On Wed, May 13, 2015 at 6:04 PM Shekar Tippur ctip...@gmail.com wrote:
Hello,
I seem to come across a issue with replication. We have 2 nodes where
Kafka
and yarn run.
We have enabled replication factor on Kafka
, Shekar Tippur ctip...@gmail.com wrote:
Here is the complete log:
http://pastebin.com/nX7twETm
Interesting, I see a leader not available exception instead of the earlier
one.
./container_1431601903660_0001_01_02/samza-container-0.log:2015-05-14
04:53:41 BrokerPartitionInfo [WARN] Error
this topic only has one replica 1018019532, which was down
and hence not available.
Guozhang
On Thu, May 14, 2015 at 5:16 AM, Shekar Tippur ctip...@gmail.com wrote:
Here is what I see on Kafka log:
[2015-05-14 04:11:27,752] ERROR Closing socket for /10.180.195.32
because
of error
the log4j.xml to config as you want. Usually, daily base is good
enough.
Thanks,
Fang, Yan
yanfang...@gmail.com
On Thu, May 14, 2015 at 10:56 AM, Shekar Tippur ctip...@gmail.com
wrote:
Hello,
How are the Samza/yarn logs rotated as a best practice? We seem to be
filling our
Hello,
I seem to come across a issue with replication. We have 2 nodes where Kafka
and yarn run.
We have enabled replication factor on Kafka (Replication factor = 2). For
testing redundancy, we shutdown broker01 server.
On the yarn application logs, we see the
exception
, but they exist in
the 0.8.0 release.
Cheers,
Chris
On Mon, Mar 30, 2015 at 5:26 PM, Shekar Tippur ctip...@gmail.com wrote:
One more thing ..
We have this configured as well.
systems.kafka.producer.metadata.broker.list=sprfargas102:6667,sprfargas103:6667
On Mon, Mar 30, 2015 at 5:08
Hello,
I realise this is a Kafka question. Since there are quite a few Kafka
experts here and the answer to this question can help Samza community as
well, bringing this question here.
We are testing Kafka redundancy (just 2 brokers for now).
We have increased replication factor to 2 servers
One more thing ..
We have this configured as well.
systems.kafka.producer.metadata.broker.list=sprfargas102:6667,sprfargas103:6667
On Mon, Mar 30, 2015 at 5:08 PM, Shekar Tippur ctip...@gmail.com wrote:
Hello,
I realise this is a Kafka question. Since there are quite a few Kafka
experts
Felix/Jordan,
1 - 2 is exactly what I was looking for as well. I want to expose
webservices call to Kafka/samza. As there is no concept of a session, I was
wondering how to send back enriched data to the web services request.
Or am I way off on this? Meaning, is this a completely wrong use case
Hello,
Want to confirm a basic understanding of Kafka.
If I have a dataset that needs to be partitioned by 4 columns, then the
progression is
{topic1:partition_key1} - {Group by samza on partition_key1}
-
{topic2:partition_key2} - {Group by samza on partition_key2}
-
{topic3:partition_key3} -
I would like to decouple Samza properties with the custom ones (if
possible).
- Shekar
On Mon, Mar 23, 2015 at 3:12 PM, Shekar Tippur ctip...@gmail.com wrote:
Hello,
I have a custom properties file with name value pairs.
name1 value1
name2 value2
name3 value3
I want to pass
redundancy now.
- Shekar
On Fri, Mar 13, 2015 at 1:20 PM, Chris Riccomini criccom...@apache.org
wrote:
Hey Shekar,
Awesome, thanks! Would love to get any doc updates that would be useful.
Curious: what was wrong?
Cheers,
Chris
On Fri, Mar 13, 2015 at 1:00 PM, Shekar Tippur ctip
! :)
Cheers,
Chris
On Wed, Mar 11, 2015 at 9:43 PM, Shekar Tippur ctip...@gmail.com wrote:
Here is the corresponding log:
2015-03-11 20:43:09,665 INFO [AsyncDispatcher event handler]
localizer.LocalizedResource (LocalizedResource.java:handle(203)) -
Resource
http://sprfargas102:8000/hello
is starting up, and
dumps the full classpath?
Cheers,
Chris
On Thu, Mar 12, 2015 at 10:17 AM, Shekar Tippur ctip...@gmail.com
wrote:
I think all these jars are in place (Under
$HADOOP_YARN_HOME/share/hadoop/hdfs/lib)
- Shekar
On Thu, Mar 12, 2015 at 9:36 AM, Chris Riccomini
)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
Looks like a common issue with yarn but not sure how to resolve as yet.
- Shekar
On Thu, Mar 12, 2015 at 1:44 PM, Shekar Tippur ctip...@gmail.com wrote:
Chris - Here
)
at java.util.concurrent.FutureTask.get(FutureTask.java:187)
at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1030)
at org.apache.hadoop.ipc.Client.call(Client.java:1384)
... 8 more
On Wed, Mar 11, 2015 at 4:56 PM, Shekar Tippur ctip...@gmail.com wrote:
Hello,
Sorry to reopen
Hello,
As I am trying to extend what we have to other sources, I changed some
code. Looks like deserialization is broken.
Here is a sample stream input:
{creation_time:2015-02-19T04:03:51Z,alert_policy_name:Alert
Policy,application_name:PROD AWS,business_unit_l1:CEG}
61 matches
Mail list logo
