[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052053#comment-15052053 ] Illya Yalovyy commented on OOZIE-2402: -- Thank you! Please let me know if you have any questions. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch, OOZIE-2402-4.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15043994#comment-15043994 ] Illya Yalovyy commented on OOZIE-2402: -- It seems like build is broken. I'll wait until it is fixed and re-submit rebased patch. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch, OOZIE-2402-4.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042643#comment-15042643 ] Illya Yalovyy commented on OOZIE-2402: -- Results of the build don't make sense: [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 04:00 min [INFO] Finished at: 2015-12-05T01:48:14+00:00 [INFO] Final Memory: 69M/989M [INFO] ... -1 TESTS - patch does not compile, cannot run testcases Could you please suggest a way to troubleshoot the issue? > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch, OOZIE-2402-4.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated OOZIE-2402: - Attachment: OOZIE-2402-4.patch > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch, OOZIE-2402-4.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15037131#comment-15037131 ] Illya Yalovyy commented on OOZIE-2402: -- No functionality change since previous patch, only style related fixes. This failed test (. testCoordChangeConcurrency(org.apache.oozie.command.coord.TestCoordChangeXCommand)) seems to be irrelevant. I can re-submit my patch if required. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated OOZIE-2402: - Attachment: OOZIE-2402-3.patch Styling is fixed. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15036149#comment-15036149 ] Illya Yalovyy commented on OOZIE-2402: -- [~rkanter], I have addressed your feedback. Please review a new patch. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated OOZIE-2402: - Attachment: OOZIE-2402-2.patch > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15027681#comment-15027681
]
Illya Yalovyy commented on OOZIE-2402:
--
[~rkanter],
Thank you for the prompt review.
Please see my notes below:
1. Will fix it.
2. I'll update related documentation.
3. Will fix it.
4. Will fix it.
5. {{IOUtils.copyBytes(in, out, fs.getConf(), true);}} closes both streams
internally. We need this {{close()}} statement in catch section only for a case
when {{out = fs.create(new Path(dstPath, file.getName()));}} fails with an
exception.
6. Will fix it
7. I wanted to escape overhead of hadoop FS implementation, but I will run some
tests to actually measure the difference. If it is not significant, I will use
{{fs.copyFromLocalFile}} to copy individual files.
8. Will add unit test
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022357#comment-15022357 ] Illya Yalovyy commented on OOZIE-2402: -- [~rkanter], Could you please review this patch? > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy > Attachments: OOZIE-2402-1.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15018512#comment-15018512 ] Illya Yalovyy commented on OOZIE-2402: -- org.apache.oozie.event.TestEventGeneration.testForNoDuplicates and org.apache.oozie.util.TestMetricsInstrumentation.testSamplers have been failing for a while already. Don't seem to be relevant. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy > Attachments: OOZIE-2402-1.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Illya Yalovyy updated OOZIE-2402: - Attachment: OOZIE-2402-1.patch > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy > Attachments: OOZIE-2402-1.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
Illya Yalovyy created OOZIE-2402: Summary: oozie-setup.sh sharelib create takes a long time on large clusters Key: OOZIE-2402 URL: https://issues.apache.org/jira/browse/OOZIE-2402 Project: Oozie Issue Type: Improvement Components: tools Affects Versions: 4.2.0 Reporter: Illya Yalovyy When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. Copy the tarball itself takes only around 10 seconds. It seems like performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
