[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509237#comment-14509237 ] Sangjin Lee commented on YARN-3437: --- Thanks Junping. I initially went with ConcurrentHashMap when I first created this as that is my preference as well. But it was really preventing multiple threads from starting their collector (should that situation arise) that made ConcurrentHashMap not an option. Again, if we want both, we would need to look at the LoadingCache. But since this is really a low contention situation, it would be an overkill. The chances of this code running into a lock contention should be low. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch, YARN-3437.003.patch, YARN-3437.004.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[
https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509717#comment-14509717
]
Zhijie Shen commented on YARN-3437:
---
Sorry for the late comments. This patch has half MR code and half YARN code.
It's not good to commit it as one patch. I have one thought of managing the
commits:
1. The YARN code is nearly duplicate with YARN-3390. As YARN-3390 is almost
ready, we can get that patch in first.
2. Move this jira to MR project, only retain the MR code in the patch and do
some minor rebase according to YARN-3390.
3. TimelineServicePerformanceTest is in different package and has the different
name. Hopefully it won't conflict with YARN-2556. So once YARN-2556 gets
committed, we just need to refactor TimelineServicePerformanceTest to reuse
YARN-2556 code. BTW, can we put TimelineServicePerformanceTest into the same
package of TimelineServicePerformance in YARN-2556, and rename it to
TimelineServicePerformanceTestv2?
How do you think about the plan for the commits?
W.R.T to the patch, I'm a bit concerned that the write which contains one event
per entity is not so typical to represent real use case. And configuration and
metrics are even not covered. Is it more realistic to write an entity with 10
events and 10 metrics, which have 100 points in the time series?
And one nit in the patch: {{entity.setEntityType(TEZ_DAG_ID);}}. How about
not mentioning TEZ in the MR code?
convert load test driver to timeline service v.2
Key: YARN-3437
URL: https://issues.apache.org/jira/browse/YARN-3437
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: YARN-3437.001.patch, YARN-3437.002.patch,
YARN-3437.003.patch, YARN-3437.004.patch
This subtask covers the work for converting the proposed patch for the load
test driver (YARN-2556) to work with the timeline service v.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509905#comment-14509905 ] Sangjin Lee commented on YARN-3437: --- Well it's not entirely true. It seems I still need to change TimelineCollector.getTimelineEntityContext() from protected to public. But creating another YARN JIRA just to make those several lines of changes seems too much. Thoughts folks? convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch, YARN-3437.003.patch, YARN-3437.004.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509932#comment-14509932 ] Zhijie Shen commented on YARN-3437: --- bq. How's that sound? It's also good to me. bq. We would use that one for more realistic load whereas we could keep this mode as a simpler test. Thoughts? It's okay to make it a simpler case, but could we at least cover one config, and one metric, hence we can verify the db that storing this info also works? bq. But creating another YARN JIRA just to make those several lines of changes seems too much A couple of lines change in YARN for MR patch is okay. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch, YARN-3437.003.patch, YARN-3437.004.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509934#comment-14509934 ] Zhijie Shen commented on YARN-3437: --- Oh, previously I said TimelineServicePerformanceTestv2, but actually I meant TimelineServicePerformanceV2. Just a minor suggestion, and it's up to you to find the suitable class name. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch, YARN-3437.003.patch, YARN-3437.004.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[
https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509862#comment-14509862
]
Sangjin Lee commented on YARN-3437:
---
Thanks for your comments [~zjshen].
{quote}
1. The YARN code is nearly duplicate with YARN-3390. As YARN-3390 is almost
ready, we can get that patch in first.
2. Move this jira to MR project, only retain the MR code in the patch and do
some minor rebase according to YARN-3390.
{quote}
Let's do this. While I was working on YARN-3438, I'm realizing that for the
performance tests it is probably OK to use the TimelineCollectors directly and
bypass the TimelineCollectorManager altogether. If we do that, then this could
become purely a MR patch. I'll update this patch to remove the use of
TimelineCollectorManager and move this JIRA to MAPREDUCE. How's that sound?
{quote}
3. TimelineServicePerformanceTest is in different package and has the different
name. Hopefully it won't conflict with YARN-2556. So once YARN-2556 gets
committed, we just need to refactor TimelineServicePerformanceTest to reuse
YARN-2556 code. BTW, can we put TimelineServicePerformanceTest into the same
package of TimelineServicePerformance in YARN-2556, and rename it to
TimelineServicePerformanceTestv2?
{quote}
That's fine. I'll move it back to the same package.
{quote}
W.R.T to the patch, I'm a bit concerned that the write which contains one event
per entity is not so typical to represent real use case. And configuration and
metrics are even not covered. Is it more realistic to write an entity with 10
events and 10 metrics, which have 100 points in the time series?
And one nit in the patch: entity.setEntityType(TEZ_DAG_ID);. How about not
mentioning TEZ in the MR code?
{quote}
Note that this is adding simple entity writes. The more realistic part of the
test is coming in YARN-3438 (I'm nearly finished with that), and it will have
multiple levels of entities as well as metrics and configuration. We would use
that one for more realistic load whereas we could keep this mode as a simpler
test. Thoughts?
I'll change the name of the entity to be something else.
convert load test driver to timeline service v.2
Key: YARN-3437
URL: https://issues.apache.org/jira/browse/YARN-3437
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: YARN-3437.001.patch, YARN-3437.002.patch,
YARN-3437.003.patch, YARN-3437.004.patch
This subtask covers the work for converting the proposed patch for the load
test driver (YARN-2556) to work with the timeline service v.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508742#comment-14508742 ] Junping Du commented on YARN-3437: -- bq. Also, when you said may be move out of collector.start(), did you mean moving the collector.start() call outside the synchronization block? If so, I'd be hesitant to do that. We just had a discussion on this in another JIRA. I see. We don't want the other concurrent thread get a non-start collector. We can improve this later (in some other JIRA), e.g. attach an additional monitor object to specific collector (rather than sync on all collectors), and do something like: startIfNotStarted(). Another NIT is to replace Collections.synchronizedMap with ConcurrentHashMap, the later one has better performance and better safety (never throw ConcurrentModificationException) but just not keep order of element which we don't need in our case. Given we have YARN-3390 to continue refactor work on the same piece of code, we can continue the discussion there. Latest patch LGTM. +1. Will go ahead to commit it within 24h if no object/further comments from others. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch, YARN-3437.003.patch, YARN-3437.004.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508151#comment-14508151 ] Sangjin Lee commented on YARN-3437: --- Could you kindly take a look at the latest patch? Thanks! convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch, YARN-3437.003.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508378#comment-14508378 ] Sangjin Lee commented on YARN-3437: --- Thanks for the review [~djp]! bq. For performance perspective, we should move LOG.info() out of synchronized block (may be move out of collector.start()?). I can move the LOG.info() call outside the synchronized block. That said, I don't think this would have a meaningful performance impact. Aside from the fact that logging calls are usually synchronized themselves, it is reasonable to expect that the contention for this lock (collectors) would be quite low. We're talking about contention when multiple AMs are competing to create collectors on the same node, and the chances that there is any contention on this lock would be very low. Also, when you said may be move out of collector.start(), did you mean moving the collector.start() call outside the synchronization block? If so, I'd be hesitant to do that. We just had a discussion on this in another JIRA (see https://issues.apache.org/jira/browse/YARN-3390?focusedCommentId=14508121page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14508121). bq. we don't need to LOG.ERROR (replace with INFO?) That is a good suggestion. I'll update this (and remove()) to lower the logging level for this. bq. For remove(), similar that we should move collector.stop() and LOG.info() out of synchronized block. This we can do safely. I'll update the patch. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch, YARN-3437.003.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[
https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14508400#comment-14508400
]
Hadoop QA commented on YARN-3437:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727521/YARN-3437.004.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / a100be6 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7466//console |
This message was automatically generated.
convert load test driver to timeline service v.2
Key: YARN-3437
URL: https://issues.apache.org/jira/browse/YARN-3437
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: YARN-3437.001.patch, YARN-3437.002.patch,
YARN-3437.003.patch, YARN-3437.004.patch
This subtask covers the work for converting the proposed patch for the load
test driver (YARN-2556) to work with the timeline service v.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[
https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14507948#comment-14507948
]
Hadoop QA commented on YARN-3437:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12727346/YARN-3437.003.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 0ebe84d |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7459//console |
This message was automatically generated.
convert load test driver to timeline service v.2
Key: YARN-3437
URL: https://issues.apache.org/jira/browse/YARN-3437
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: YARN-3437.001.patch, YARN-3437.002.patch,
YARN-3437.003.patch
This subtask covers the work for converting the proposed patch for the load
test driver (YARN-2556) to work with the timeline service v.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506792#comment-14506792 ] Junping Du commented on YARN-3437: -- bq. The downside is code duplication. Agree. However, I think this is the least price we could pay when developing on different branches in parallel. bq. If others are comfortable with that, I could quickly move it to a different package. I am OK with this in case we have additional JIRA to track removing duplicated code after YARN-2556 land. We can also add some credit to contributors of YARN-2556 in commit/CHANGE messages which should be very common. [~zjshen] and [~jeagles], any ideas on this? convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505593#comment-14505593 ] Junping Du commented on YARN-3437: -- I agree too if we don't have clear plan for YARN-2556 so far. There should be no reason to block other going efforts. An suggestion (optional only) is: can we adjust name (or package path) slightly for duplicated file (TimelineServerPerformance.java) with YARN-2556? We can have an additional patch to remove duplicated file when YARN-2556 get in trunk. I assume this could be easier for YARN-2928 rebase back to trunk/branch-2 as less conflict. Thoughts? convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505382#comment-14505382 ] Sangjin Lee commented on YARN-3437: --- I think we need to make progress on this as this is blocking other JIRAs and also it's tied to the schema evaluation. My vote is to get this committed, and adjust this once YARN-2556 lands and we rebase. Thoughts? convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504842#comment-14504842 ] Junping Du commented on YARN-3437: -- Filed a new JIRA to track fine-grained performance data sounds good to me. Given the patch here have duplicated code with YARN-2556, I would like to understand what's our plan for YARN-2556. [~jeagles], can you share your vision on this? Looks like this JIRA block YARN-3390 (a refactor JIRA) which block YARN-3044 (RM writing events to v2 ATS service). I would like to have a clear path to make all patches goes in as a pipeline with getting ride of any potential deadlock. :) May be the first step is to make YARN-2556 get committed it, and get patch here rebased? [~jeagles], [~sjlee0] and [~zjshen], what's your opinion on this? convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506157#comment-14506157 ] Sangjin Lee commented on YARN-3437: --- The downside is code duplication, but as you said we would not be impacted when/if YARN-2556 lands. If others are comfortable with that, I could quickly move it to a different package. Let me know soon, and I'll update the patch. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503383#comment-14503383 ] Sangjin Lee commented on YARN-3437: --- [~jeagles], thanks for your comment. Yes, I agree the level of reporting in this patch is mostly basic and very high level. I definitely agree that we need to add more reporting in terms of the actual number of writes/puts done on the storage, etc. The first use of this tool is to compare the overall throughput/performance of the storage implementations and identify high level issues. I suspect for that the coarse-grained reporting might be enough. As a follow-up JIRA, I'll add a new JIRA that reports more fine-grained write performance data. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503391#comment-14503391 ] Sangjin Lee commented on YARN-3437: --- Added YARN-3512. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499247#comment-14499247 ] Zhijie Shen commented on YARN-3437: --- Per my comment on [YARN-3390 | https://issues.apache.org/jira/browse/YARN-3390?focusedCommentId=14499245page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14499245]. Please feel free to move forward. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500187#comment-14500187 ] Jonathan Eagles commented on YARN-3437: --- Now that I have dug into timeline server performance (YARN-3448). I have a better understanding of what type of writes are costly. For example, a single entity will generate dozens or writes to the database. The number of primary keys, the number of related entities, and the write batch size (entities per put) greatly affect the time an entity put takes. While this is a good start, I think there should at least be a follow up that addresses these issues to better measure the write performance. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500112#comment-14500112 ] Zhijie Shen commented on YARN-3437: --- Will take a look today. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500088#comment-14500088 ] Sangjin Lee commented on YARN-3437: --- Thanks [~zjshen]! Could someone please commit this patch unless there are further comments? convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch, YARN-3437.002.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[
https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497432#comment-14497432
]
Hadoop QA commented on YARN-3437:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725758/YARN-3437.002.patch
against trunk revision 1b89a3e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7354//console
This message is automatically generated.
convert load test driver to timeline service v.2
Key: YARN-3437
URL: https://issues.apache.org/jira/browse/YARN-3437
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: YARN-3437.001.patch, YARN-3437.002.patch
This subtask covers the work for converting the proposed patch for the load
test driver (YARN-2556) to work with the timeline service v.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494524#comment-14494524 ] Sangjin Lee commented on YARN-3437: --- Forgot to add that I copied it to get the load testing going. Once YARN-2556 gets committed on trunk, I'd need to update this to handle both the old timeline service and the new timeline service. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494521#comment-14494521 ] Sangjin Lee commented on YARN-3437: --- Thanks [~djp] for your comments! I agree with most of your comments. This particular subtask generates a pretty simple model (just a straightforward conversion of YARN-2556) where each map task creates one app collector and simple entities. But we will add more test modes that will have different characteristics. For one, we will add a mode where it iterates over MR job history files and pump them into the timeline storage. The number of mappers is a pretty useful way of controlling the parallelism of tests, and we can add more parameters to have finer-grained parallelism controls (one app collector manager having multiple app collectors, etc.). bq. BTW, it has some duplicated code with YARN-2556 (like TimelineServerPerformance.java). That's because I copied it (as mentioned in the description). :) I had to fix a few issues with the original patch there to get it going. Some appear to be unrelated to porting it to the timeline service v.2. I'll add some comments there to help it get committed. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394437#comment-14394437 ] Junping Du commented on YARN-3437: -- Thanks [~sjlee0] for delivering a patch here! Just quickly go through the patch, looks like we are generating one app collector per map task. I think this is good for scalability test on backend storage which can be a bottleneck in mainstream cases. In addition, do we want to address some extreme cases, e.g. a huge applications will have hundreds of thousands or even millions tasks? If so, then may be we want to know a single app collector's bottleneck as well for accepting/forwarding messages from hundreds of thousands maps. Also, in a real cluster, the mapping from cluster to app, and app to tasks are all 1-N mapping. May be making app aggregator number configurable (just like map task number, and byte per map, etc.) is something we can do for next step? BTW, it has some duplicated code with YARN-2556 (like TimelineServerPerformance.java). Looks like YARN-2556 is in pretty good shape and possible to go to trunk and branch-2 quickly. I would remind to keep watching that JIRA status and do necessary rebase work if that patch go in and we may want to merge it into YARN-2928 branch soon. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[
https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393554#comment-14393554
]
Hadoop QA commented on YARN-3437:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12709078/YARN-3437.001.patch
against trunk revision 6a6a59d.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7206//console
This message is automatically generated.
convert load test driver to timeline service v.2
Key: YARN-3437
URL: https://issues.apache.org/jira/browse/YARN-3437
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Sangjin Lee
Attachments: YARN-3437.001.patch
This subtask covers the work for converting the proposed patch for the load
test driver (YARN-2556) to work with the timeline service v.2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3437) convert load test driver to timeline service v.2
[ https://issues.apache.org/jira/browse/YARN-3437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393530#comment-14393530 ] Sangjin Lee commented on YARN-3437: --- Added a few folks for review. convert load test driver to timeline service v.2 Key: YARN-3437 URL: https://issues.apache.org/jira/browse/YARN-3437 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Sangjin Lee Attachments: YARN-3437.001.patch This subtask covers the work for converting the proposed patch for the load test driver (YARN-2556) to work with the timeline service v.2. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
