date:20160831

[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...

2016-08-31 Thread huafengw

Github user huafengw commented on a diff in the pull request:

https://github.com/apache/incubator-gearpump/pull/78#discussion_r77100932
  
--- Diff: 
examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/Split.scala
 ---
@@ -23,29 +23,34 @@ import java.util.concurrent.TimeUnit
 
 import org.apache.gearpump.Message
 import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.source.Watermark
+import org.apache.gearpump.streaming.source.{DataSource, DataSourceTask, 
Watermark}
 import org.apache.gearpump.streaming.task.{Task, TaskContext}
 
-class Split(taskContext: TaskContext, conf: UserConfig) extends 
Task(taskContext, conf) {
-  import taskContext.output
 
-  override def onStart(startTime: Instant): Unit = {
-self ! Watermark(Instant.now)
-  }
+class Split() extends DataSource {
--- End diff --

I prefer `class Split extends DataSource`, no braces.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...

2016-08-31 Thread huafengw

Github user huafengw commented on a diff in the pull request:

https://github.com/apache/incubator-gearpump/pull/78#discussion_r77101042
  
--- Diff: 
examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/Split.scala
 ---
@@ -23,29 +23,34 @@ import java.util.concurrent.TimeUnit
 
 import org.apache.gearpump.Message
 import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.source.Watermark
+import org.apache.gearpump.streaming.source.{DataSource, DataSourceTask, 
Watermark}
 import org.apache.gearpump.streaming.task.{Task, TaskContext}
 
-class Split(taskContext: TaskContext, conf: UserConfig) extends 
Task(taskContext, conf) {
-  import taskContext.output
 
-  override def onStart(startTime: Instant): Unit = {
-self ! Watermark(Instant.now)
-  }
+class Split() extends DataSource {
+
+
+  override def open(context: TaskContext, startTime: Instant): Unit = {}
 
-  override def onNext(msg: Message): Unit = {
+
+  override def read(): Message = {
 Split.TEXT_TO_SPLIT.lines.foreach { line =>
   line.split("[\\s]+").filter(_.nonEmpty).foreach { msg =>
-output(new Message(msg, System.currentTimeMillis()))
+new Message(msg, System.currentTimeMillis())
   }
 }
-
-import scala.concurrent.duration._
-taskContext.scheduleOnce(Duration(100, TimeUnit.MILLISECONDS))(self !
-  Watermark(Instant.now))
+Message("message")
   }
+
+  override def close(): Unit = {}
+
+  override def getWatermark: Instant = Instant.now()
+
+  Watermark(Instant.now)
--- End diff --

What's this line of code used for?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high

2016-08-31 Thread Huafeng Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453971#comment-15453971
 ] 

Huafeng Wang commented on GEARPUMP-185:
---

Hi Kam, I'm wondering is this problem also caused issue 
[GEARPUMP-161|https://issues.apache.org/jira/browse/GEARPUMP-161]?

> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
>  Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
>Assignee: Kam Kasravi
> Fix For: 0.8.2
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's 
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
>   |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>   |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true 
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
>   |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true 
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
>   |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c 
> /usr/java/jdk1.8.0_72/bin/java  -Xmx512m -cp 
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
>  ...
>   |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a 
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>  
>   |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249 
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp 
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
>  ...
>   |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
>   |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors' 
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280  5.3  3.1 5406960 261356 pts/13 Sl+  09:28   0:10 
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true 
> -DlogFilename=master 
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
>  -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417  7.7  1.8 5392212 154728 pts/27 Sl+  09:29   0:05 
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true 
> -DlogFilename=worker 
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
>  -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0  3.1 4572132 255776 pts/27 Sl+  09:33   0:11 
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M 
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC 
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761  132  5.2 7268592 427200 pts/27 Sl+  09:33   2:31 
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M 
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC 
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 [...] 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 ...
> {code}
> h3. Steps to Reproduce
> To

[jira] [Commented] (GEARPUMP-192) refactor example sources task to use DataSource API

2016-08-31 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/GEARPUMP-192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15453941#comment-15453941
 ] 

ASF GitHub Bot commented on GEARPUMP-192:
-

Github user huafengw commented on a diff in the pull request:

https://github.com/apache/incubator-gearpump/pull/78#discussion_r77100932
  
--- Diff: 
examples/streaming/wordcount/src/main/scala/org/apache/gearpump/streaming/examples/wordcount/Split.scala
 ---
@@ -23,29 +23,34 @@ import java.util.concurrent.TimeUnit
 
 import org.apache.gearpump.Message
 import org.apache.gearpump.cluster.UserConfig
-import org.apache.gearpump.streaming.source.Watermark
+import org.apache.gearpump.streaming.source.{DataSource, DataSourceTask, 
Watermark}
 import org.apache.gearpump.streaming.task.{Task, TaskContext}
 
-class Split(taskContext: TaskContext, conf: UserConfig) extends 
Task(taskContext, conf) {
-  import taskContext.output
 
-  override def onStart(startTime: Instant): Unit = {
-self ! Watermark(Instant.now)
-  }
+class Split() extends DataSource {
--- End diff --

I prefer `class Split extends DataSource`, no braces.


> refactor example sources task to use DataSource API
> ---
>
> Key: GEARPUMP-192
> URL: https://issues.apache.org/jira/browse/GEARPUMP-192
> Project: Apache Gearpump
>  Issue Type: Improvement
>  Components: streaming
>Affects Versions: 0.8.1
>Reporter: Manu Zhang
>Assignee: Kaifang Bao
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (GEARPUMP-195) Single node DAG's should not show node as stalling

2016-08-31 Thread Kam Kasravi (JIRA)


 [ 
https://issues.apache.org/jira/browse/GEARPUMP-195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kam Kasravi updated GEARPUMP-195:
-
Description: DAG's with only one node should not show stalled. If a sink 
node is missing we should either prevent the DAG from being run or not show the 
upstream node status as stalled  (was: DAG's with only one node should not show 
stalled. If a sink node is missing we should either prevent the DAG from being 
run or not show it's status as stalled)

> Single node DAG's should not show node as stalling
> --
>
> Key: GEARPUMP-195
> URL: https://issues.apache.org/jira/browse/GEARPUMP-195
> Project: Apache Gearpump
>  Issue Type: Bug
>Affects Versions: 0.8.1
>Reporter: Kam Kasravi
>Assignee: Kam Kasravi
> Fix For: 0.8.2
>
>
> DAG's with only one node should not show stalled. If a sink node is missing 
> we should either prevent the DAG from being run or not show the upstream node 
> status as stalled



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (GEARPUMP-198) empty kafka topics show very high receive throughput

2016-08-31 Thread Kam Kasravi (JIRA)


 [ 
https://issues.apache.org/jira/browse/GEARPUMP-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kam Kasravi updated GEARPUMP-198:
-
Summary: empty kafka topics show very high receive throughput  (was: empty 
kafka topics show very high throughput)

> empty kafka topics show very high receive throughput
> 
>
> Key: GEARPUMP-198
> URL: https://issues.apache.org/jira/browse/GEARPUMP-198
> Project: Apache Gearpump
>  Issue Type: Bug
>Affects Versions: 0.8.1
>Reporter: Kam Kasravi
>Assignee: Manu Zhang
>Priority: Minor
> Fix For: 0.8.2
>
>
> if the kafkasource reaches the end of the topic it shows very high message 
> throughput even though it's not really sending/receiving messages



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high

2016-08-31 Thread Kam Kasravi (JIRA)


 [ 
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kam Kasravi resolved GEARPUMP-185.
--
   Resolution: Fixed
Fix Version/s: (was: 0.8.0)
   0.8.2

> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
>  Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
>Assignee: Kam Kasravi
> Fix For: 0.8.2
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's 
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
>   |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>   |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true 
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
>   |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true 
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
>   |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c 
> /usr/java/jdk1.8.0_72/bin/java  -Xmx512m -cp 
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
>  ...
>   |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a 
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>  
>   |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249 
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp 
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
>  ...
>   |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
>   |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors' 
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280  5.3  3.1 5406960 261356 pts/13 Sl+  09:28   0:10 
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true 
> -DlogFilename=master 
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
>  -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417  7.7  1.8 5392212 154728 pts/27 Sl+  09:29   0:05 
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true 
> -DlogFilename=worker 
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
>  -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0  3.1 4572132 255776 pts/27 Sl+  09:33   0:11 
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M 
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC 
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761  132  5.2 7268592 427200 pts/27 Sl+  09:33   2:31 
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M 
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC 
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 [...] 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 ...
> {code}
> h3. Steps to Reproduce
> To generate trafic I've used such application:
> {code}
>   kafka

[jira] [Created] (GEARPUMP-200) A missing gear.conf in services and other gearpump commands should result in an ERROR to the log

2016-08-31 Thread Kam Kasravi (JIRA)

Kam Kasravi created GEARPUMP-200:


 Summary: A missing gear.conf in services and other gearpump 
commands should result in an ERROR to the log
 Key: GEARPUMP-200
 URL: https://issues.apache.org/jira/browse/GEARPUMP-200
 Project: Apache Gearpump
  Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Kam Kasravi
Assignee: Kam Kasravi
 Fix For: 0.8.2


Commands like services, master, local, etc are built with Config logic that 
uses gear.conf with fallback to geardefault.conf. If gear.conf is missing from 
the classpath of these commands we should issue a WARNING or ERROR and treat it 
not unlike a missing jar file since it is a critical dependency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high

2016-08-31 Thread Kam Kasravi (JIRA)


 [ 
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kam Kasravi updated GEARPUMP-185:
-
Comment: was deleted

(was: i'm able to confirm that the missing vmargs is related to the user 
argument. Under the applications section of the dashboard UI - the vcap users 
application's config had no vmargs. Launching a new application (which ran as 
yarn) had a config that contained a valid vmargs for the executor)

> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
>  Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
>Assignee: Kam Kasravi
> Fix For: 0.8.0
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's 
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
>   |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>   |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true 
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
>   |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true 
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
>   |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c 
> /usr/java/jdk1.8.0_72/bin/java  -Xmx512m -cp 
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
>  ...
>   |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a 
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>  
>   |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249 
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp 
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
>  ...
>   |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
>   |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors' 
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280  5.3  3.1 5406960 261356 pts/13 Sl+  09:28   0:10 
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true 
> -DlogFilename=master 
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
>  -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417  7.7  1.8 5392212 154728 pts/27 Sl+  09:29   0:05 
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true 
> -DlogFilename=worker 
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
>  -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0  3.1 4572132 255776 pts/27 Sl+  09:33   0:11 
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M 
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC 
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761  132  5.2 7268592 427200 pts/27 Sl+  09:33   2:31 
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M 
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC 
>

[jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high

2016-08-31 Thread Kam Kasravi (JIRA)


[ 
https://issues.apache.org/jira/browse/GEARPUMP-185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15452397#comment-15452397
 ] 

Kam Kasravi commented on GEARPUMP-185:
--

The problem (and resolution) was that the services process CLASSPATH did not 
include gear.conf (gear.conf was not copied into the conf subdirectory) when 
this process was installed in TAP. Once gear.conf was copied into this 
directory everything works as expected where the gear.conf used to build 
gearpump commands (output/target/pack/bin) is used by all commands. I think a 
reasonable improvement is a WARNING emitted to console if gear.conf is not 
found.

> Yarn kills Apache Gearpump's worker if the memory usage is too high
> ---
>
> Key: GEARPUMP-185
> URL: https://issues.apache.org/jira/browse/GEARPUMP-185
> Project: Apache Gearpump
>  Issue Type: Bug
> Environment: yarn
>Reporter: Piotr Grabuszynski
>Assignee: Kam Kasravi
> Fix For: 0.8.0
>
>
> h3. Description
> If the traffic is to big (memory usage is height) Yarn kills Gearpump's 
> worker.
> Clean YARN log (only the part with processes, bigger piece below):
> {code}
> Dump of the process-tree for container_e05_1468841606343_0128_01_03 :
>   |- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS) 
> SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
>   |- 2702 2164 2159 2159 (java) 137198 34820 8656748544 199840 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=0 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true 
> -Dgearpump.config.file=/tmp/gearpump6797336735445779094.conf ...
>   |- 2646 2164 2159 2159 (java) 4268 464 5847887872 70626 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true 
> -Dgearpump.config.file=/tmp/gearpump941454978115257558.conf ...
>   |- 2159 2157 2159 2159 (bash) 0 0 108654592 301 /bin/bash -c 
> /usr/java/jdk1.8.0_72/bin/java  -Xmx512m -cp 
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
>  ...
>   |- 2165 2159 2159 2159 (tee) 38 396 103337984 164 /usr/bin/tee -a 
> /var/log/hadoop-yarn/container/application_1468841606343_0128/container_e05_1468841606343_0128_01_03/stderr
>  
>   |- 2164 2159 2159 2159 (java) 4915 815 2367729664 78249 
> /usr/java/jdk1.8.0_72/bin/java -Xmx512m -cp 
> conf:pack/gearpump-2.11-0.8.0/conf:pack/gearpump-2.11-0.8.0/lib/daemon/*:pack/gearpump-2.11-0.8.0/lib/*:/etc/hadoop/conf.cloudera.YARN:/var/run/cloudera-scm-agent/process/195-yarn-NODEMANAGER
>  ...
>   |- 6417 2164 2159 2159 (java) 3900 1182 5893246976 125356 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=2 -Dgearpump.executorId=0 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
>   |- 6360 2164 2159 2159 (java) 966 58 5845979136 54720 
> /usr/java/jdk1.8.0_72/jre/bin/java -Dgearpump.username=vcap 
> -Dgearpump.applicationId=2 -Dgearpump.executorId=-1 
> -Dgearpump.master.starttime=2016-07-20-08-36 
> -Dgearpump.log.application.dir=logs -Djava.net.preferIPv4Stack=true ...
> {code}
> *As we can see there are no "-Xmx" or "-Xsx" parameters near executors' 
> processes.*
> In standalone mode, we can see both:
> {code}
> // MASTER
> GER\pgr+ 48280  5.3  3.1 5406960 261356 pts/13 Sl+  09:28   0:10 
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true 
> -DlogFilename=master 
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
>  -Djava.rmi.server.hostname=localhost ...
> // WORKER
> GER\pgr+ 48417  7.7  1.8 5392212 154728 pts/27 Sl+  09:29   0:05 
> /usr/lib/jvm/java-8-oracle/bin/java -server -Djava.net.preferIPv4Stack=true 
> -DlogFilename=worker 
> -Dgearpump.home=/home/local/GER/pgrabusz/Documents/gearpump-sources/gearpump-2.11-0.8.0
>  -Djava.rmi.server.hostname=localhost ...
> // APP MASTER
> GER\pgr+ 48707 15.0  3.1 4572132 255776 pts/27 Sl+  09:33   0:11 
> /usr/lib/jvm/java-8-oracle/jre/bin/java -server -Xms512M -Xmx1024M -Xss1M 
> -XX:+HeapDumpOnOutOfMemoryError -XX:+UseConcMarkSweepGC 
> -XX:CMSInitiatingOccupancyFraction=80 -XX:+UseParNewGC -XX:NewRatio=3 
> -Djava.rmi.server.hostname=localhost -Dgearpump.username=GER\pgrabusz 
> -Dgearpump.applicationId=1 -Dgearpump.executorId=-1 ...
> // APP EXECUTOR 0
> GER\pgr+ 48761  132  5.2 7268592 427200

[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...

[GitHub] incubator-gearpump pull request #78: [GEARPUMP-192] refactor example sources...

[jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high

[jira] [Commented] (GEARPUMP-192) refactor example sources task to use DataSource API

[jira] [Updated] (GEARPUMP-195) Single node DAG's should not show node as stalling

[jira] [Updated] (GEARPUMP-198) empty kafka topics show very high receive throughput

[jira] [Resolved] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high

[jira] [Created] (GEARPUMP-200) A missing gear.conf in services and other gearpump commands should result in an ERROR to the log

[jira] [Issue Comment Deleted] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high

[jira] [Commented] (GEARPUMP-185) Yarn kills Apache Gearpump's worker if the memory usage is too high

10 matches

Site Navigation

Mail list logo

Footer information