[jira] [Commented] (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13018015#comment-13018015
]
Allen Wittenauer commented on MAPREDUCE-2265:
-
It needs to get fixed in 0.22 because it will become a defacto (shell) API.
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13018026#comment-13018026
]
Nigel Daley commented on MAPREDUCE-2265:
Allen, can you contribute a patch?
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12983262#action_12983262
]
Owen O'Malley commented on MAPREDUCE-2265:
--
{quote}
We already do this for the native libs, why is the task-controller any
different?
{quote}
Agreed, we need to push this change all of the way through, but you've chosen
to start here. I'm precisely arguing that all of the deployment directories
should assume a single architecture with the exception of 32/64 bit libraries
where you clearly need both. Does that sound reasonable? I think that users
would rather download a x86 redhat tarball than a tarball that includes all
possible combinations of hardware and operating system.
{quote}
the scripts never call the task-controller, only the TT does,
{quote}
The scripts calls jsvc and it should be in the same place as task-controller.
{quote}
Then the tarball environment and the product of a build start looking rather
different, we have extra install complexity, etc.
{quote}
Running out of the build directory isn't a good option until we undo the
project split anyways. Once we do, then we could make a single build directory
that contains a deployment directory.
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12983438#action_12983438
]
Allen Wittenauer commented on MAPREDUCE-2265:
-
Due to multi-architecture JVMs on various OSes (Solaris, OS X, and likely
others), I doubt we'll ever be able to remove the requirement from native libs
since the user can supply the -d flag to flip back and forth. Due to the
linking requirements, we're sort of stuck here. Executables are a different
beast. I personally can't think of a modern OS where the 64-bit variant
couldn't run its 32-bit variant's executables. They might be out there, but
likely rare. So I don't think we're looking at a 32-bit RHEL vs. 64-bit RHEL
problem. We're more likely looking at a RHEL5 vs. RHEL6 issue or a SunOS 5.10
SPARC vs. i386 issue. To which I say:
- Keep the platform structure in libexec
- Make libexec/taskcontroller a shell wrapper (or whatever) that determines the
proper platform and calls the proper exec
This also enables us to provide a different binary for OSes that provide
different functionalities. (For example, on Solaris, I'd likely use a pfexec
equivalent anyway.)
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981669#action_12981669
]
Todd Lipcon commented on MAPREDUCE-2265:
I suppose jsvc only applies to HDFS currently, but let's discuss here about
task-controller and then we can open a JIRA later about jsvc - no sense
confusing the discussion by splitting it in two places.
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981811#action_12981811
]
Allen Wittenauer commented on MAPREDUCE-2265:
-
This stuff definitely should get moved, but I'd prefer seeing them in libexec.
sbin should be for the admin-only hadoop commands.
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981828#action_12981828
]
Todd Lipcon commented on MAPREDUCE-2265:
OK, $HADOOP_HOME/libexec/platform/task-controller good by you for this change?
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981834#action_12981834
]
Owen O'Malley commented on MAPREDUCE-2265:
--
Let's just use $HADOOP_HOME/libexec. The platform only helps if you plan on
running the executable from a shared mount point, which should be strongly
discouraged. Nothing brings a cluster to its knees faster than running off of a
NFS mounted HADOOP_HOME.
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981835#action_12981835
]
Todd Lipcon commented on MAPREDUCE-2265:
Owen: it's also useful if you want to distribute a tarball that already has the
native bits compiled for common platforms (ie Linux 32 and 64 bit)
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981936#action_12981936
]
Owen O'Malley commented on MAPREDUCE-2265:
--
But it is *only* useful for universal binary tarballs. For RPM's for instance,
you can just put the right one in to place and remove a unnecessary level of
indirection.
For the fat binary tarballs, how about if it looks like:
libexec/arch/$ARCH/*
libexec/task-controller is a symlink to one of the executables that are in the
arch folder?
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981944#action_12981944
]
Todd Lipcon commented on MAPREDUCE-2265:
bq. But it is only useful for universal binary tarballs. For RPM's for
instance, you can just put the right one in to place
Sure, but then the RPM install and the tarball install would have a different
layout (or require the user to manually create the symlink to the correct
one... one extra step which is unnecessary)
Is there a particular objection to just having the arch directories with no
symlink, and using the same pattern for this as we already do for native? I
know it will require some changes to how the task controller resolves the conf
dir path, etc, but happy to make and test those.
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981947#action_12981947
]
Owen O'Malley commented on MAPREDUCE-2265:
--
I want a consistent layout between tarballs and RPMs.
I don't want to include the platform name simply because then all of the
scripts need to consistently figure out the platform name. If we can have the
resolution done once at install time, that is much less cumbersome. It will
also leak into bin and lib.
For universal tar balls, I think that a post-untar setup script makes the most
sense. I guess you could also have different hadoop home directories in the
tarball with symlinks between them.
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2265) task-controller and jsvc should install into sbin/platform/ directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12981958#action_12981958
]
Todd Lipcon commented on MAPREDUCE-2265:
bq. I don't want to include the platform name simply because then all of the
scripts need to consistently figure out the platform name
We already do this for the native libs, why is the task-controller any
different? Besides, the scripts never call the task-controller, only the TT
does, where the PlatformName class is readily available.
bq. For universal tar balls, I think that a post-untar setup script makes the
most sense
Then the tarball environment and the product of a build start looking rather
different, we have extra install complexity, etc.
bq. I guess you could also have different hadoop home directories in the
tarball with symlinks between them.
Building two entirely parallel HADOOP_HOMES for different architectures out of
a series of symlinks? That sounds like a disaster waiting to happen.
task-controller and jsvc should install into sbin/platform/ directory
---
Key: MAPREDUCE-2265
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2265
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: build, task-controller, tasktracker
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Fix For: 0.22.0
Currently the task-controller and jsvc live in the bin/ directory
regardless of build platform. This is incorrect since these components are
native compiled code and thus are built for a particular architecture. So,
when we ship a build of 22, we will want to ship both 32-bit and 64-bit
artifacts so users can use these components without rebuilding on their own.
Additionally, it doesn't make sense for them to be in bin/ since they're not
user-facing in any way (i.e a user would never directly invoke them). So I
would propose putting them in an sbin directory.
The final proposed path is $HADOOP_HOME/sbin/platform/{jsvc,task-controller}
Note this is not an incompatible change since these components were not
present in any prior apache release.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
