[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-18 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138619#comment-14138619
 ] 

Lefty Leverenz commented on HIVE-7553:
--

Thanks for the release note, [~Ferd].

The *reload* command can be documented in Beeline Command Options, and 
*hive.reloadable.aux.jars.path* goes in the two wikidocs mentioned in a 
previous comment:

* [HiveServer2 Clients -- Beeline Command Options | 
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-BeelineCommandOptions]
* [_previous doc comment_ | 
https://issues.apache.org/jira/browse/HIVE-7553?focusedCommentId=14124772page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14124772]

One question:  What does this sentence mean?  In order to making the 
HiveServer2 as a service, it should not be restarted when the auxiliary jars 
changed.  (Maybe something like In order to make HiveServer2 function as a 
service, it should not need to be restarted ...?)

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
  Labels: TODOC14
 Fix For: 0.14.0

 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.4.patch, HIVE-7553.patch, HIVE-7553.pdf, SessionStateTest.jar.v1, 
 SessionStateTest.jar.v2


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-08 Thread Ferdinand Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126447#comment-14126447
 ] 

Ferdinand Xu commented on HIVE-7553:


[~leftylev] release notes are as below:
The newly added parameter hive.reloadable.aux.jars.path is serving for the 
complement of the previous configuration hive.aux.jars.path. In order to 
making the HiveServer2 as a service, it should not be restarted when the 
auxiliary jars changed. In this JIRA, a new command reload with a parameter 
hive.reloadable.aux.jars.path can be used for avoid the scheduling 
maintenance window for every jar change. User can execute the reload command 
via the beeline to make the HS2 aware of the changes made in the path specified 
by the parameter hive.reloadable.aux.jars.path. And the supported changes 
including adding, removing or updating jar files. 

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
  Labels: TODOC14
 Fix For: 0.14.0

 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.4.patch, HIVE-7553.patch, HIVE-7553.pdf, SessionStateTest.jar.v1, 
 SessionStateTest.jar.v2


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-06 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124772#comment-14124772
 ] 

Lefty Leverenz commented on HIVE-7553:
--

Doc note:  This adds configuration parameter *hive.reloadable.aux.jars.path* to 
HiveConf.java and adds reload to the default list for 
*hive.security.command.whitelist*, so two wikidocs need to be updated with 
links back to this JIRA ticket:

* [AdminManual -- Configuration -- Hive Configuration Variables | 
https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration#AdminManualConfiguration-HiveConfigurationVariables]
* [Configuration Properties -- hive.security.command.whitelist | 
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.security.command.whitelist]

Does the reload command need to be documented too?

A release note would also be helpful.

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
  Labels: TODOC14
 Fix For: 0.14.0

 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.4.patch, HIVE-7553.patch, HIVE-7553.pdf, SessionStateTest.jar.v1, 
 SessionStateTest.jar.v2


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-06 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124779#comment-14124779
 ] 

Brock Noland commented on HIVE-7553:


[~ferd] can you develop a release note?

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
  Labels: TODOC14
 Fix For: 0.14.0

 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.4.patch, HIVE-7553.patch, HIVE-7553.pdf, SessionStateTest.jar.v1, 
 SessionStateTest.jar.v2


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-05 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123435#comment-14123435
 ] 

Hive QA commented on HIVE-7553:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/1271/HIVE-7553.4.patch

{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 6171 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.session.TestSessionState.testReloadAuxJars2[0]
org.apache.hadoop.hive.ql.session.TestSessionState.testReloadAuxJars2[1]
org.apache.hadoop.hive.ql.session.TestSessionState.testReloadExistingAuxJars2[0]
org.apache.hadoop.hive.ql.session.TestSessionState.testReloadExistingAuxJars2[1]
org.apache.hive.jdbc.miniHS2.TestHiveServer2.testConnection
org.apache.hive.service.TestHS2ImpersonationWithRemoteMS.testImpersonation
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/656/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/656/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-656/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 1271

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.4.patch, HIVE-7553.patch, HIVE-7553.pdf, SessionStateTest.jar.v1, 
 SessionStateTest.jar.v2


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-05 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123603#comment-14123603
 ] 

Brock Noland commented on HIVE-7553:


I ran TestSessionState manually and it passed. +1

Thank you [~Ferd]!!

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.4.patch, HIVE-7553.patch, HIVE-7553.pdf, SessionStateTest.jar.v1, 
 SessionStateTest.jar.v2


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-03 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120093#comment-14120093
 ] 

Brock Noland commented on HIVE-7553:


[~Ferd] the jar cannot be included in the patch itself. Can you upload them to 
the JIRA for manual testing?

Thanks!!

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.patch, HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-02 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117938#comment-14117938
 ] 

Hive QA commented on HIVE-7553:
---



{color:red}Overall{color}: -1 at least one tests failed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665837/HIVE-7553.3.patch

{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 6139 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.session.TestSessionState.testReloadAuxJars[0]
org.apache.hadoop.hive.ql.session.TestSessionState.testReloadAuxJars[1]
org.apache.hadoop.hive.ql.session.TestSessionState.testReloadExistingAuxJars[0]
org.apache.hadoop.hive.ql.session.TestSessionState.testReloadExistingAuxJars[1]
{noformat}

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/596/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/596/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-596/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12665837

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.patch, HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-02 Thread Ferdinand Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117952#comment-14117952
 ] 

Ferdinand Xu commented on HIVE-7553:


Failed cases are caused by not founding jar files used by the newly added ones.
Error Messages:
Exception in thread Thread-7 java.lang.AssertionError: Could not find 
SessionStateTest.jar.v1

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.3.patch, 
 HIVE-7553.patch, HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-01 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117105#comment-14117105
 ] 

Lefty Leverenz commented on HIVE-7553:
--

+1 for doc issues

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.patch, 
 HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-09-01 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117247#comment-14117247
 ] 

Hive QA commented on HIVE-7553:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665715/HIVE-7553.2.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/587/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/587/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-587/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-587/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 
'shims/0.20/src/main/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java'
Reverted 
'shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java'
Reverted 
'shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java'
Reverted 
'shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java'
Reverted 'common/src/java/org/apache/hadoop/hive/common/FileUtils.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/core/target hcatalog/streaming/target 
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target 
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target 
accumulo-handler/target hwi/target common/target common/src/gen contrib/target 
service/target serde/target beeline/target odbc/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1621718.

At revision 1621718.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12665715

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.2.patch, HIVE-7553.patch, 
 HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be 

[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-28 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113962#comment-14113962
 ] 

Hive QA commented on HIVE-7553:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12665004/HIVE-7553.1.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/544/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/544/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-544/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-544/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 
'ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out'
Reverted 
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/Vectorizer.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/core/target hcatalog/streaming/target 
hcatalog/server-extensions/target hcatalog/hcatalog-pig-adapter/target 
hcatalog/webhcat/svr/target hcatalog/webhcat/java-client/target 
accumulo-handler/target hwi/target common/target common/src/gen service/target 
contrib/target serde/target beeline/target odbc/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1621165.

At revision 1621165.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12665004

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.patch, HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-28 Thread Lefty Leverenz (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114245#comment-14114245
 ] 

Lefty Leverenz commented on HIVE-7553:
--

Doc question (also posted on the review board):  one parameter description says 
renewed by executing reload command but the other adds refresh to its 
default values.  Which is correct, reload or refresh?

Answering myself:  The first patch had refresh for both, but patch 1 changes 
hive.refresh.aux.jars.path to hive.reloadable.aux.jars.path so I guess my 
review board question was backwards -- reload is correct and you need to change 
refresh to reload in the default values for hive.security.command.whitelist.

Otherwise the parameters are good to go.  Thanks for the fixes, [~Ferd].

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.1.patch, HIVE-7553.patch, HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-24 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108665#comment-14108665
 ] 

Hive QA commented on HIVE-7553:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664069/HIVE-7553.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/480/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/480/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-480/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit 
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ 
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.6.0_34/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost 
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-480/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java'
Reverted 
'metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java'
Reverted 'common/src/java/org/apache/hadoop/hive/conf/HiveConf.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/thrift/ThriftHttpCLIService.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/thrift/ThriftBinaryCLIService.java'
Reverted 'service/src/java/org/apache/hive/service/cli/CLIService.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/session/SessionManager.java'
Reverted 
'service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20/target 
shims/0.20S/target shims/0.23/target shims/aggregator/target 
shims/common/target shims/common-secure/target packaging/target 
hbase-handler/target testutils/target jdbc/target metastore/target 
itests/target itests/hcatalog-unit/target itests/test-serde/target 
itests/qtest/target itests/hive-unit-hadoop2/target itests/hive-minikdc/target 
itests/hive-unit/target itests/custom-serde/target itests/util/target 
hcatalog/target hcatalog/core/target hcatalog/streaming/target 
hcatalog/server-extensions/target hcatalog/webhcat/svr/target 
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target 
accumulo-handler/target hwi/target common/target common/src/gen service/target 
service/src/java/org/apache/hive/service/server/ThreadWithGarbageCleanup.java 
service/src/java/org/apache/hive/service/server/ThreadFactoryWithGarbageCleanup.java
 contrib/target serde/target beeline/target odbc/target cli/target 
ql/dependency-reduced-pom.xml ql/target
+ svn update

Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1620249.

At revision 1620249.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh 
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12664069

 avoid the scheduling maintenance window 

[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-24 Thread Hive QA (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14108694#comment-14108694
 ] 

Hive QA commented on HIVE-7553:
---



{color:red}Overall{color}: -1 no tests executed

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12664073/HIVE-7553.patch

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/482/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/482/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-482/

Messages:
{noformat}
 This message was trimmed, see log for full details 
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[580,17]
 cannot find symbol
[ERROR] symbol:   class Path
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[584,35]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[584,67]
 cannot find symbol
[ERROR] symbol:   class Path
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[584,18]
 cannot find symbol
[ERROR] symbol:   class Path
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[634,35]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[634,18]
 cannot find symbol
[ERROR] symbol:   class Path
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[642,35]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[642,55]
 cannot find symbol
[ERROR] symbol:   class Path
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[653,34]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[653,17]
 cannot find symbol
[ERROR] symbol:   class Path
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[806,52]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[814,68]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[836,67]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[840,76]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[865,75]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[1015,34]
 cannot find symbol
[ERROR] symbol:   class Configuration
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities
[ERROR] 
/data/hive-ptest/working/apache-svn-trunk-source/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java:[1224,53]
 cannot find symbol
[ERROR] symbol:   class JobConf
[ERROR] location: class org.apache.hadoop.hive.ql.exec.Utilities

[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-12 Thread Ferdinand Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093859#comment-14093859
 ] 

Ferdinand Xu commented on HIVE-7553:


As attached is just lines of the initial code, there are more things unfinished 
left like the changes on hive script  under HIVE_HOME/bin, logics to handle 
added jars and the session manager's control logic to initialize the Jar files.

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-12 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094649#comment-14094649
 ] 

Brock Noland commented on HIVE-7553:


Just an FYI that HIVE-6262 does some interning of classes. Not sure if that 
matters at this point.

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-12 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14094849#comment-14094849
 ] 

Brock Noland commented on HIVE-7553:


Hi [~Ferd],

Thank you very much for uploading the POC code!!

I think with this approach, we'll have to be sure we don't hold on to 
references to the class as that will keep the class and classloader around, 
causing a memory leak. As such, I think we'll have to change some of the work 
done in HIVE-6262 so we don't keep references around. 

Additionally, I think we'll want to do something similar to the UDF case when 
we load input and output formats, storage handlers, and serdes as these can all 
be specified as custom classes on the command. 

What are your thoughts on these topics?

Cheers!
Brock

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-11 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093251#comment-14093251
 ] 

Brock Noland commented on HIVE-7553:


Hi [~Ferd],

Thank you very much for creating the design document! Please do not worry about 
uploading to the wiki today, we'll do that once we've iterated on the design.

Could you add a few diagrams of user sessions and classloaders? I think that 
would help clarify things.

bq. When finding a class, it will check the parent class loader first 

I believe that something similar has been in for servlet containers in the 
past. Can you add an overview what was done and how it works to Class loader 
Features?

bq. . HiveConf will hold the custom class loader. 

What is the scope of the HiveConf object without a user session? When is it 
created and destroyed? Any user can execute add jar, will each user have their 
own instance of the class loader.

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu
 Attachments: HIVE-7553.pdf


 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-07 Thread Ferdinand Xu (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089008#comment-14089008
 ] 

Ferdinand Xu commented on HIVE-7553:


I am now working on this issue, but before putting in a patch I want to present 
the approach so that I could get some feedback.
To my understand, this issue is attempting to resolve the hot swap jar files 
for HIVE_AUX_JARS_PATH. I did some POCs locally. 
The main workflow for jars loader is as below:
1. Read env HIVE_AUX_JARS_PATH from the system and parse it through adding jar 
files under the directory one by one 
2. The system classloader loads the jar files in step 1. 
3. When trying to create a UDF based on the added aux jars, FunctionTask will 
try to get a class from loaded classes by calling getUdfClass method. From my 
view, the key factor to solve this problem is mainly about the classloader. 
For class loader, it has some limitations which should be some designs: 
a) when finding a class, it will check the parent classloader first to see 
whether the class is loaded and then current classloader. 
b) Classloader did not have the mechanism for us to reload a cached class. 
Based on this, I have come up with the following solutions in three catalogs. 
1. change the order of loading classes
 As mentioned in section 1, auxilary class path is parsed and loaded when hive 
server2 booting up. Can we postpone the loading phase until needed which means 
loading it on the go, namely creating UDF? In addition, reloading cached jars 
is handful issue. To resolve it, we should create a new classloader each time 
followed by calling the method Thread.setContextClassloader(refreshedCL) and 
HiveConf.setClassLoader(refreshedCL) 
2. override the standard classloader
Still keep the current loading order  make the classloader loading child 
classloader first and then the parent(still need to create a new classloader on 
the go) 
3. Others
use OSGi? JRebel? 
If I have anything incorrect, please feel free to figure me out. Thanks!

 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu

 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Commented] (HIVE-7553) avoid the scheduling maintenance window for every jar change

2014-08-07 Thread Brock Noland (JIRA)

[ 
https://issues.apache.org/jira/browse/HIVE-7553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14089396#comment-14089396
 ] 

Brock Noland commented on HIVE-7553:


Hi [~Ferd],

Thank you so much for looking into this!! I think you have a good direction on 
some of the possible solutions. I do think this is a very big item with many 
different aspects. 

Would you be interested in creating a design document on this? There are many 
examples out there: https://cwiki.apache.org/confluence/display/Hive/DesignDocs 
e.g: https://cwiki.apache.org/confluence/display/Hive/Theta+Join

If you create a design doc, I think the big aspect of this design doc would be 
evaluate all the pros/cons of each possible solution.

Thank you again for looking at this!!

Cheers,
Brock


 avoid the scheduling maintenance window for every jar change
 

 Key: HIVE-7553
 URL: https://issues.apache.org/jira/browse/HIVE-7553
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Reporter: Ferdinand Xu
Assignee: Ferdinand Xu

 When user needs to refresh existing or add a new jar to HS2, it needs to 
 restart it. As HS2 is service exposed to clients, this requires scheduling 
 maintenance window for every jar change. It would be great if we could avoid 
 that.



--
This message was sent by Atlassian JIRA
(v6.2#6252)