A question about Hadoop 1 job user id used for group mapping, which could lead to performance degradatioin

[Jian Fang]

Hi,

I looked at Hadoop 1.X source code and found some logic that I could not
understand.

In the org.apache.hadoop.mapred.Child class, there were two UGIs defined as
follows.

    UserGroupInformation current = UserGroupInformation.getCurrentUser();
    current.addToken(jt);

    UserGroupInformation taskOwner
     =
UserGroupInformation.createRemoteUser(firstTaskid.getJobID().toString());
    taskOwner.addToken(jt);

But it is the taskOwner that is actually passed as a UGI to task tracker
and then to HDFS. The first one was not referenced any where.

    final TaskUmbilicalProtocol umbilical =
      taskOwner.doAs(new PrivilegedExceptionAction<TaskUmbilicalProtocol>()
{
        @Override
        public TaskUmbilicalProtocol run() throws Exception {
          return
(TaskUmbilicalProtocol)RPC.getProxy(TaskUmbilicalProtocol.class,
              TaskUmbilicalProtocol.versionID,
              address,
              defaultConf);
        }
    });

What puzzled me is that the job id is actually passed in as the user name
to task tracker. On the Name node side, when it tries to map the
non-existing user name, i.e., task id, to a group, it always returns empty
array. As a result, we always see annoying warning messages such as

 WARN org.apache.hadoop.security.UserGroupInformation (IPC Server handler
63 on 9000): No groups available for user job_201401071758_0002

Sometimes, the warning messages were thrown so fast, hundreds or even
thousands per second for a big cluster, the system performance was degraded
dramatically.

Could someone please explain why this logic was designed in this way? Any
benefit to use non-existing user for the group mapping? Or is this a bug?

Thanks in advance,

John