[ https://issues.apache.org/jira/browse/HIVE-8071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Damien Carol updated HIVE-8071: ------------------------------- Description: For every run of the hive CLI there is a delay for the shell startup {noformat} 14/07/31 23:07:19 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS 14/07/31 23:07:19 INFO tez.DagUtils: Hive jar directory is hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/ 14/07/31 23:07:19 INFO tez.DagUtils: Localizing resource because it does not exist: file:/home/gopal/tez-autobuild/dist/hive/lib/hive-exec-0.14.0-SNAPSHOT.jar to dest: hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/hive-exec-0.14.0-SNAPSHOTde1f82f0b5561d3db9e3080dfb2897210a3bda4ca5e7b14e881e381115837fd8. jar 14/07/31 23:07:19 INFO tez.DagUtils: Looks like another thread is writing the same file will wait. 14/07/31 23:07:19 INFO tez.DagUtils: Number of wait attempts: 5. Wait interval: 5000 14/07/31 23:07:19 INFO tez.DagUtils: Resource modification time: 1406870512963 14/07/31 23:07:20 INFO tez.TezSessionState: Opening new Tez Session (id: 02d6b558-44cc-4182-b2f2-6a37ffdd25d2, scratch dir: hdfs://mac-10:8020/tmp/hive-gopal/_tez_session_dir/02d6b558-44cc-4182-b2f2-6a37ffdd25d2) {noformat} Traced this to a method which does PRIVATE LRs - this is marked as PRIVATE even if it is from a common install dir. {code} public LocalResource localizeResource(Path src, Path dest, Configuration conf) throws IOException { .... return createLocalResource(destFS, dest, LocalResourceType.FILE, LocalResourceVisibility.PRIVATE); } {code} was: For every run of the hive CLI there is a delay for the shell startup 14/07/31 23:07:19 INFO Configuration.deprecation: fs.default.name is deprecated. Instead, use fs.defaultFS 14/07/31 23:07:19 INFO tez.DagUtils: Hive jar directory is hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/ 14/07/31 23:07:19 INFO tez.DagUtils: Localizing resource because it does not exist: file:/home/gopal/tez-autobuild/dist/hive/lib/hive-exec-0.14.0-SNAPSHOT.jar to dest: hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/hive-exec-0.14.0-SNAPSHOTde1f82f0b5561d3db9e3080dfb2897210a3bda4ca5e7b14e881e381115837fd8. jar 14/07/31 23:07:19 INFO tez.DagUtils: Looks like another thread is writing the same file will wait. 14/07/31 23:07:19 INFO tez.DagUtils: Number of wait attempts: 5. Wait interval: 5000 14/07/31 23:07:19 INFO tez.DagUtils: Resource modification time: 1406870512963 14/07/31 23:07:20 INFO tez.TezSessionState: Opening new Tez Session (id: 02d6b558-44cc-4182-b2f2-6a37ffdd25d2, scratch dir: hdfs://mac-10:8020/tmp/hive-gopal/_tez_session_dir/02d6b558-44cc-4182-b2f2-6a37ffdd25d2) Traced this to a method which does PRIVATE LRs - this is marked as PRIVATE even if it is from a common install dir. {code} public LocalResource localizeResource(Path src, Path dest, Configuration conf) throws IOException { .... return createLocalResource(destFS, dest, LocalResourceType.FILE, LocalResourceVisibility.PRIVATE); } {code} > hive shell tries to write hive-exec.jar for each run > ---------------------------------------------------- > > Key: HIVE-8071 > URL: https://issues.apache.org/jira/browse/HIVE-8071 > Project: Hive > Issue Type: Bug > Components: Tez > Reporter: Rajesh Balamohan > Assignee: Rajesh Balamohan > Attachments: HIVE-8071.1.patch > > > For every run of the hive CLI there is a delay for the shell startup > {noformat} > 14/07/31 23:07:19 INFO Configuration.deprecation: fs.default.name is > deprecated. Instead, use fs.defaultFS > 14/07/31 23:07:19 INFO tez.DagUtils: Hive jar directory is > hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/ > 14/07/31 23:07:19 INFO tez.DagUtils: Localizing resource because it does not > exist: > file:/home/gopal/tez-autobuild/dist/hive/lib/hive-exec-0.14.0-SNAPSHOT.jar to > dest: > hdfs://mac-10:8020/user/gopal/apps/2014-Jul-31/hive/hive-exec-0.14.0-SNAPSHOTde1f82f0b5561d3db9e3080dfb2897210a3bda4ca5e7b14e881e381115837fd8. > jar > 14/07/31 23:07:19 INFO tez.DagUtils: Looks like another thread is writing the > same file will wait. > 14/07/31 23:07:19 INFO tez.DagUtils: Number of wait attempts: 5. Wait > interval: 5000 > 14/07/31 23:07:19 INFO tez.DagUtils: Resource modification time: 1406870512963 > 14/07/31 23:07:20 INFO tez.TezSessionState: Opening new Tez Session (id: > 02d6b558-44cc-4182-b2f2-6a37ffdd25d2, scratch dir: > hdfs://mac-10:8020/tmp/hive-gopal/_tez_session_dir/02d6b558-44cc-4182-b2f2-6a37ffdd25d2) > {noformat} > Traced this to a method which does PRIVATE LRs - this is marked as PRIVATE > even if it is from a common install dir. > {code} > public LocalResource localizeResource(Path src, Path dest, Configuration > conf) > throws IOException { > .... > return createLocalResource(destFS, dest, LocalResourceType.FILE, > LocalResourceVisibility.PRIVATE); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)