[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057475#comment-15057475
]
Hadoop QA commented on OOZIE-2402:
--
Testing JIRA OOZIE-2402
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2625/
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch,
> OOZIE-2402-3.patch, OOZIE-2402-4.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051997#comment-15051997 ] Robert Kanter commented on OOZIE-2402: -- Ya, the test-patch script has been having problems lately. Thanks for fixing #4. The patch looks good, but I'd like to actually play around a bit with it first. I'll try to get to that sometime either this week or next week. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch, OOZIE-2402-4.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15052053#comment-15052053 ] Illya Yalovyy commented on OOZIE-2402: -- Thank you! Please let me know if you have any questions. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch, OOZIE-2402-4.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15043994#comment-15043994 ] Illya Yalovyy commented on OOZIE-2402: -- It seems like build is broken. I'll wait until it is fixed and re-submit rebased patch. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch, OOZIE-2402-4.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042256#comment-15042256
]
Robert Kanter commented on OOZIE-2402:
--
Thanks for making the changes. Though it looks like my #4 comment was missed:
{quote}
It would be nice if checkCopyResults could print out the exception for each
failure, instead of just one. Otherwise, if there are multiple problems, the
user will have to keep trying after resolving each issue.
{quote}
Can you do that too? I think if you might just have to remove the {{break;}}
statements.
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch,
> OOZIE-2402-3.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042567#comment-15042567
]
Hadoop QA commented on OOZIE-2402:
--
Testing JIRA OOZIE-2402
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color} - patch does not compile, cannot run testcases
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2618/
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch,
> OOZIE-2402-3.patch, OOZIE-2402-4.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042643#comment-15042643 ] Illya Yalovyy commented on OOZIE-2402: -- Results of the build don't make sense: [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 04:00 min [INFO] Finished at: 2015-12-05T01:48:14+00:00 [INFO] Final Memory: 69M/989M [INFO] ... -1 TESTS - patch does not compile, cannot run testcases Could you please suggest a way to troubleshoot the issue? > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch, OOZIE-2402-4.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036149#comment-15036149 ] Illya Yalovyy commented on OOZIE-2402: -- [~rkanter], I have addressed your feedback. Please review a new patch. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15036532#comment-15036532
]
Hadoop QA commented on OOZIE-2402:
--
Testing JIRA OOZIE-2402
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 6 line(s) with trailing spaces
.{color:red}-1{color} the patch contains 1 line(s) longer than 132
characters
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1703
.Tests failed: 3
.Tests errors: 0
.The patch failed the following testcases:
. testForNoDuplicates(org.apache.oozie.event.TestEventGeneration)
. testSamplers(org.apache.oozie.util.TestMetricsInstrumentation)
. testUpdateSLA(org.apache.oozie.sla.TestSLAService)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2612/
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037007#comment-15037007
]
Hadoop QA commented on OOZIE-2402:
--
Testing JIRA OOZIE-2402
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:green}+1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:green}+1{color} the patch does not introduce any trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:green}+1{color} the patch does adds/modifies 1 testcase(s)
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1703
.Tests failed: 3
.Tests errors: 0
.The patch failed the following testcases:
. testSamplers(org.apache.oozie.util.TestMetricsInstrumentation)
.
testCoordChangeConcurrency(org.apache.oozie.command.coord.TestCoordChangeXCommand)
. testForNoDuplicates(org.apache.oozie.event.TestEventGeneration)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2614/
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch,
> OOZIE-2402-3.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037131#comment-15037131 ] Illya Yalovyy commented on OOZIE-2402: -- No functionality change since previous patch, only style related fixes. This failed test (. testCoordChangeConcurrency(org.apache.oozie.command.coord.TestCoordChangeXCommand)) seems to be irrelevant. I can re-submit my patch if required. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: OOZIE-2402-1.patch, OOZIE-2402-2.patch, > OOZIE-2402-3.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027449#comment-15027449
]
Robert Kanter commented on OOZIE-2402:
--
The overall approach looks good to me. This should be helpful in speeding
things up.
Some minor things:
# Please remove the trailing whitespaces from the 7 lines
# Can you update the docs? Update this section
(http://oozie.apache.org/docs/4.2.0/AG_Install.html#Oozie_Server_Setup) with
the new usage info. You can find the source for this page in AG_Install.twiki
in the code
# In {{concurrentCopyFromLocal}}, I think we should put the call to
{{copyFolderRecursively}} in a try-catch-finally block. For instance, we
should make sure to always call {{threadpool.shutdown()}}.
# It would be nice if {{checkCopyResults}} could print out the exception for
each failure, instead of just one. Otherwise, if there are multiple problems,
the user will have to keep trying after resolving each issue.
# In {{copyFolderRecursively}}, the Streams should be closed in a finally
block, not a catch block. Otherwise, they're only closed if an Exception occurs
# The description for the concurrency should say what the default is (e.g.
"(default=1)")
# Why use Streams instead of {{fs.copyFromLocalFile}}?
# Can you add/update unit tests in {{TestOozieSharelibCLI}}?
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027681#comment-15027681
]
Illya Yalovyy commented on OOZIE-2402:
--
[~rkanter],
Thank you for the prompt review.
Please see my notes below:
1. Will fix it.
2. I'll update related documentation.
3. Will fix it.
4. Will fix it.
5. {{IOUtils.copyBytes(in, out, fs.getConf(), true);}} closes both streams
internally. We need this {{close()}} statement in catch section only for a case
when {{out = fs.create(new Path(dstPath, file.getName()));}} fails with an
exception.
6. Will fix it
7. I wanted to escape overhead of hadoop FS implementation, but I will run some
tests to actually measure the difference. If it is not significant, I will use
{{fs.copyFromLocalFile}} to copy individual files.
8. Will add unit test
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
>Assignee: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15022357#comment-15022357 ] Illya Yalovyy commented on OOZIE-2402: -- [~rkanter], Could you please review this patch? > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy > Attachments: OOZIE-2402-1.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[
https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018502#comment-15018502
]
Hadoop QA commented on OOZIE-2402:
--
Testing JIRA OOZIE-2402
Cleaning local git workspace
{color:green}+1 PATCH_APPLIES{color}
{color:green}+1 CLEAN{color}
{color:red}-1 RAW_PATCH_ANALYSIS{color}
.{color:green}+1{color} the patch does not introduce any @author tags
.{color:green}+1{color} the patch does not introduce any tabs
.{color:red}-1{color} the patch contains 7 line(s) with trailing spaces
.{color:green}+1{color} the patch does not introduce any line longer than
132
.{color:red}-1{color} the patch does not add/modify any testcase
{color:green}+1 RAT{color}
.{color:green}+1{color} the patch does not seem to introduce new RAT
warnings
{color:green}+1 JAVADOC{color}
.{color:green}+1{color} the patch does not seem to introduce new Javadoc
warnings
{color:green}+1 COMPILE{color}
.{color:green}+1{color} HEAD compiles
.{color:green}+1{color} patch compiles
.{color:green}+1{color} the patch does not seem to introduce new javac
warnings
{color:green}+1 BACKWARDS_COMPATIBILITY{color}
.{color:green}+1{color} the patch does not change any JPA
Entity/Colum/Basic/Lob/Transient annotations
.{color:green}+1{color} the patch does not modify JPA files
{color:red}-1 TESTS{color}
.Tests run: 1698
.Tests failed: 2
.Tests errors: 0
.The patch failed the following testcases:
. testForNoDuplicates(org.apache.oozie.event.TestEventGeneration)
. testSamplers(org.apache.oozie.util.TestMetricsInstrumentation)
{color:green}+1 DISTRO{color}
.{color:green}+1{color} distro tarball builds with the patch
{color:red}*-1 Overall result, please check the reported -1(s)*{color}
The full output of the test-patch run is available at
. https://builds.apache.org/job/oozie-trunk-precommit-build/2596/
> oozie-setup.sh sharelib create takes a long time on large clusters
> --
>
> Key: OOZIE-2402
> URL: https://issues.apache.org/jira/browse/OOZIE-2402
> Project: Oozie
> Issue Type: Improvement
> Components: tools
>Affects Versions: 4.2.0
>Reporter: Illya Yalovyy
> Attachments: OOZIE-2402-1.patch
>
>
> When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib.
> Copy the tarball itself takes only around 10 seconds. It seems like
> performance could be improved by loading files concurrently in many threads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (OOZIE-2402) oozie-setup.sh sharelib create takes a long time on large clusters
[ https://issues.apache.org/jira/browse/OOZIE-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018512#comment-15018512 ] Illya Yalovyy commented on OOZIE-2402: -- org.apache.oozie.event.TestEventGeneration.testForNoDuplicates and org.apache.oozie.util.TestMetricsInstrumentation.testSamplers have been failing for a while already. Don't seem to be relevant. > oozie-setup.sh sharelib create takes a long time on large clusters > -- > > Key: OOZIE-2402 > URL: https://issues.apache.org/jira/browse/OOZIE-2402 > Project: Oozie > Issue Type: Improvement > Components: tools >Affects Versions: 4.2.0 >Reporter: Illya Yalovyy > Attachments: OOZIE-2402-1.patch > > > When cluster has 256+ nodes it can take up to 5 minutes to create a sharelib. > Copy the tarball itself takes only around 10 seconds. It seems like > performance could be improved by loading files concurrently in many threads. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
