[jira] [Commented] (HIVE-13665) HS2 memory leak When multiple queries are running with get_json_object
[
https://issues.apache.org/jira/browse/HIVE-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263929#comment-16263929
]
Jürgen Thomann commented on HIVE-13665:
---
Hi, I had this problem some time ago as well, but they should be fixed with
https://issues.apache.org/jira/browse/HIVE-16196 and
https://issues.apache.org/jira/browse/HIVE-16060
> HS2 memory leak When multiple queries are running with get_json_object
> --
>
> Key: HIVE-13665
> URL: https://issues.apache.org/jira/browse/HIVE-13665
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0
>Reporter: JinsuKim
> Attachments: patch.lst.txt
>
>
> The extractObjectCache in UDFJson is increased over limitation(CACHE_SIZE =
> 16). When multiple queries are running concurrently on HS2 local(not mr/tez)
> with get_json_object or get_json_tuple
> {code:java|title=HS2 heap_dump}
> Object at 0x515ab18f8
> instance of org.apache.hadoop.hive.ql.udf.UDFJson$HashCache@0x515ab18f8 (77
> bytes)
> Class:
> class org.apache.hadoop.hive.ql.udf.UDFJson$HashCache
> Instance data members:
> accessOrder (Z) : false
> entrySet (L) :
> hashSeed (I) : 0
> header (L) : java.util.LinkedHashMap$Entry@0x515a577d0 (60 bytes)
> keySet (L) :
> loadFactor (F) : 0.6
> modCount (I) : 4741146
> size (I) : 2733158 <== here!!
> table (L) : [Ljava.util.HashMap$Entry;@0x7163d8b70 (67108880 bytes)
> threshold (I) : 5033165
> values (L) :
> References to this object:
> {code}
> I think that this problem be caused by the LinkedHashMap object is not
> thread-safe
> {code}
> * Note that this implementation is not synchronized.
> * If multiple threads access a linked hash map concurrently, and at least
> * one of the threads modifies the map structurally, it must be
> * synchronized externally. This is typically accomplished by
> * synchronizing on some object that naturally encapsulates the map.
> {code}
> Reproduce :
> # Multiple queries are running with get_json_object and small input data(for
> execution on hs2 local mode)
> # jvm heap dump & analyze
> {code:title=test scenario}
> Multiple queries are running with get_json_object and small input data(for
> execute on hs2 local mode)
> 1.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040105'
> 2.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040106'
> 3.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040107'
> 4.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040108'
>
> run.sh :
> t_cnt=0
> while true
> do
> echo "query executing..."
> for i in 1 2 3 4
> do
> beeline -u jdbc:hive2://localhost:1 -n hive --silent=true -f
> $i.hql > $i.log 2>&1 &
> done
> wait
> t_cnt=`expr $t_cnt + 1`
> echo "query count : $t_cnt"
> sleep 2
> done
> jvm heap dump & analyze :
> jmap -dump:format=b,file=hive.dmp $PID
> jhat -J-mx48000m -port 8080 hive.dmp &
> {code}
> Finally I have attached our patch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[
https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263925#comment-16263925
]
Hive QA commented on HIVE-16708:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898990/HIVE-16708.01-branch-2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10661 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=38)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=139)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[table_nonprintable]
(batchId=140)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=153)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[merge_negative_5]
(batchId=88)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[explaindenpendencydiffengs]
(batchId=115)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=125)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=176)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7983/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7983/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7983/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898990 - PreCommit-HIVE-Build
> Exception while renewing a Delegation Token
> ---
>
> Key: HIVE-16708
> URL: https://issues.apache.org/jira/browse/HIVE-16708
> Project: Hive
> Issue Type: Bug
>Reporter: Manikumar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-16708.01-branch-2.patch
>
>
> I am adding support to Storm Hive Bolt to access Hive meta-store using
> delegation tokens. I am able to create and cancel delegation tokens using
> HCatClient. I am getting below exception while renewing a delegation token.
> Exception:
> org.apache.hive.hcatalog.common.HCatException:
> org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred
> while processing HCat request : MetaException while renewing delegation
> token.. Cause : MetaException(message:hive/test.cluster@example.com tries
> to renew a token with renewer hive)
> at
> org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643)
> ~[hive-webhcat-java-client-0.14.0.jar:0.14.0]
>
> Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short
> name but HCatCleint.renewDelegationToken uses the fullName. This causes the
> renewal to fail.
> Relevant parts of code:
> https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105
> Let me know If I am missing something.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-13665) HS2 memory leak When multiple queries are running with get_json_object
[
https://issues.apache.org/jira/browse/HIVE-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263918#comment-16263918
]
靳峥 commented on HIVE-13665:
---
Besides, all the five HashCache Object have the same problem.
It must be beautiful when all the threads are fighting for these five locks : )
> HS2 memory leak When multiple queries are running with get_json_object
> --
>
> Key: HIVE-13665
> URL: https://issues.apache.org/jira/browse/HIVE-13665
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0
>Reporter: JinsuKim
> Attachments: patch.lst.txt
>
>
> The extractObjectCache in UDFJson is increased over limitation(CACHE_SIZE =
> 16). When multiple queries are running concurrently on HS2 local(not mr/tez)
> with get_json_object or get_json_tuple
> {code:java|title=HS2 heap_dump}
> Object at 0x515ab18f8
> instance of org.apache.hadoop.hive.ql.udf.UDFJson$HashCache@0x515ab18f8 (77
> bytes)
> Class:
> class org.apache.hadoop.hive.ql.udf.UDFJson$HashCache
> Instance data members:
> accessOrder (Z) : false
> entrySet (L) :
> hashSeed (I) : 0
> header (L) : java.util.LinkedHashMap$Entry@0x515a577d0 (60 bytes)
> keySet (L) :
> loadFactor (F) : 0.6
> modCount (I) : 4741146
> size (I) : 2733158 <== here!!
> table (L) : [Ljava.util.HashMap$Entry;@0x7163d8b70 (67108880 bytes)
> threshold (I) : 5033165
> values (L) :
> References to this object:
> {code}
> I think that this problem be caused by the LinkedHashMap object is not
> thread-safe
> {code}
> * Note that this implementation is not synchronized.
> * If multiple threads access a linked hash map concurrently, and at least
> * one of the threads modifies the map structurally, it must be
> * synchronized externally. This is typically accomplished by
> * synchronizing on some object that naturally encapsulates the map.
> {code}
> Reproduce :
> # Multiple queries are running with get_json_object and small input data(for
> execution on hs2 local mode)
> # jvm heap dump & analyze
> {code:title=test scenario}
> Multiple queries are running with get_json_object and small input data(for
> execute on hs2 local mode)
> 1.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040105'
> 2.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040106'
> 3.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040107'
> 4.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040108'
>
> run.sh :
> t_cnt=0
> while true
> do
> echo "query executing..."
> for i in 1 2 3 4
> do
> beeline -u jdbc:hive2://localhost:1 -n hive --silent=true -f
> $i.hql > $i.log 2>&1 &
> done
> wait
> t_cnt=`expr $t_cnt + 1`
> echo "query count : $t_cnt"
> sleep 2
> done
> jvm heap dump & analyze :
> jmap -dump:format=b,file=hive.dmp $PID
> jhat -J-mx48000m -port 8080 hive.dmp &
> {code}
> Finally I have attached our patch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17994: Status: Patch Available (was: In Progress) > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17994: Attachment: HIVE-17994.02.patch Classes don't serialize well. Just keep primitiveCategory enum. > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, HIVE-17994.02.patch, > vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17994: Status: In Progress (was: Patch Available) > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263890#comment-16263890
]
Hive QA commented on HIVE-17994:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898986/HIVE-17994.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 32 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[date_timestamp_prec]
(batchId=87)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing
(batchId=282)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestBigintSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestBooleanSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestCharSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDateSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDecimalSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestDoubleSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestFloatSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestTimestampSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.TestVarcharSarg
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression1
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression10
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression2
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression3
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression4
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression5
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression7
(batchId=264)
org.apache.hadoop.hive.ql.io.sarg.TestConvertAstToSearchArg.testExpression9
(batchId=264)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testMultipleConditions_noTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testMultipleConditions_withTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_noTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withDateType
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withTranslation
(batchId=205)
org.apache.hive.storage.jdbc.TestQueryConditionBuilder.testSimpleCondition_withVariedCaseMappings
(batchId=205)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7982/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7982/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7982/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 32 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898986 - PreCommit-HIVE-Build
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row
[jira] [Commented] (HIVE-18080) Performance degradation on VectorizedLogicBench#IfExprLongColumnLongColumnBench when AVX512 is enabled
[
https://issues.apache.org/jira/browse/HIVE-18080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263873#comment-16263873
]
liyunzhang commented on HIVE-18080:
---
It seems after If expression is vectorized by AVX2 instructions. It only
consumes 2.8% of all total instructions and consumes very little cpu time(23ms)
when the warm iter number is 5000 and iter number is 5. So the variation of
this test is too big. Later will try to enlarge iterations of jmh test to see
the degradation exists or not.
{code}
To specify different parameters, use:
* - This command will use 10 warm-up iterations, 5 test iterations, and 2
forks. And it will
* display the Average Time (avgt) in Microseconds (us)
* - Benchmark mode. Available modes are:
* [Throughput/thrpt, AverageTime/avgt, SampleTime/sample, SingleShotTime/ss,
All/all]
* - Output time unit. Available time units are: [m, s, ms, us, ns].
*
* $ java -jar target/benchmarks.jar
org.apache.hive.benchmark.vectorization.VectorizedLogicBench
* -wi 10 -i 5 -f 2 -bm avgt -tu us
*/
{code}
> Performance degradation on
> VectorizedLogicBench#IfExprLongColumnLongColumnBench when AVX512 is enabled
> --
>
> Key: HIVE-18080
> URL: https://issues.apache.org/jira/browse/HIVE-18080
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
> Attachments: IFExpression_AVX2_Instruction.png,
> log.logic.avx1.single.0, log_logic.avx1.part
>
>
> Use Xeon(R) Platinum 8180 CPU to test the performance of
> [AVX512|https://en.wikipedia.org/wiki/AVX-512].
> {code}
> #cat /proc/cpuinfo |grep "model name"|head -n 1
> model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
> {code}
> Before that I have compiled hive with JDK9 as JDK9 enables AVX512
> Use hive microbenchmark(HIVE-10189) to evaluate the performance improvement.
> It seems performance(20%+) in cases in
> {{VectorizedArithmeticBench}},{{VectorizedComparisonBench}},{{VectorizedLikeBench}},{{VectorizedLogicBench}}
> execpt
> {{VectorizedLogicBench#IfExprLongColumnLongColumnBench}},{{VectorizedLogicBench#IfExprRepeatingLongColumnLongColumnBench}}
> and
> {{VectorizedLogicBench#IfExprLongColumnRepeatingLongColumnBench}}.The data is
> like following
> When i use Skylake CPU to evaluate the performance improvement of AVX512.
> I found the performance in VectorizedLogicBench is like following
> || ||AVX2 us/op||AVX512 us/op || (AVX2-AVX512)/AVX2||
> |ColAndColBench|122510| 87014| 28.9%|
> |IfExprLongColumnLongColumnBench | 1325759| 1436073| -8.3% |
> |IfExprLongColumnRepeatingLongColumnBench|1397447|1480450| -5.9%|
> |IfExprRepeatingLongColumnLongColumnBench|1401164|1483062| -5.9% |
> |NotColBench|77042.83|51513.28| 33%|
> There are degradation in
> IfExprLongColumnLongColumnBench,IfExprLongColumnRepeatingLongColumnBench,
> IfExprRepeatingLongColumnLongColumnBench, very confused why there is
> degradation on IfExprLongColumnLongColumnBench cases.
> Here we use {{taskset -cp 1 $pid}} to run the benchmark on single core to
> avoid the impact of dynamic CPU frequency scaling.
> my script
> {code}
> export JAVA_HOME=/home/zly/jdk-9.0.1/
> export PATH=$JAVA_HOME/bin:$PATH
> export LD_LIBRARY_PATH=/home/zly/jdk-9.0.1/mylib
> for i in 0 1 2; do
> java -server -XX:UseAVX=3 -jar benchmarks.jar
> org.apache.hive.benchmark.vectorization.VectorizedLogicBench * -wi 10 -i 20
> -f 1 -bm avgt -tu us >log.logic.avx3.single.$i & export pid=$!
> taskset -cp 1 $pid
> wait $pid
> done
> for i in 0 1 2; do
> java -server -XX:UseAVX=2 -jar benchmarks.jar
> org.apache.hive.benchmark.vectorization.VectorizedLogicBench * -wi 10 -i 20
> -f 1 -bm avgt -tu us >log.logic.avx2.single.$i & export pid=$!
> taskset -cp 1 $pid
> wait $pid
> done
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18080) Performance degradation on VectorizedLogicBench#IfExprLongColumnLongColumnBench when AVX512 is enabled
[
https://issues.apache.org/jira/browse/HIVE-18080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang updated HIVE-18080:
--
Attachment: IFExpression_AVX2_Instruction.png
> Performance degradation on
> VectorizedLogicBench#IfExprLongColumnLongColumnBench when AVX512 is enabled
> --
>
> Key: HIVE-18080
> URL: https://issues.apache.org/jira/browse/HIVE-18080
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
> Attachments: IFExpression_AVX2_Instruction.png,
> log.logic.avx1.single.0, log_logic.avx1.part
>
>
> Use Xeon(R) Platinum 8180 CPU to test the performance of
> [AVX512|https://en.wikipedia.org/wiki/AVX-512].
> {code}
> #cat /proc/cpuinfo |grep "model name"|head -n 1
> model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
> {code}
> Before that I have compiled hive with JDK9 as JDK9 enables AVX512
> Use hive microbenchmark(HIVE-10189) to evaluate the performance improvement.
> It seems performance(20%+) in cases in
> {{VectorizedArithmeticBench}},{{VectorizedComparisonBench}},{{VectorizedLikeBench}},{{VectorizedLogicBench}}
> execpt
> {{VectorizedLogicBench#IfExprLongColumnLongColumnBench}},{{VectorizedLogicBench#IfExprRepeatingLongColumnLongColumnBench}}
> and
> {{VectorizedLogicBench#IfExprLongColumnRepeatingLongColumnBench}}.The data is
> like following
> When i use Skylake CPU to evaluate the performance improvement of AVX512.
> I found the performance in VectorizedLogicBench is like following
> || ||AVX2 us/op||AVX512 us/op || (AVX2-AVX512)/AVX2||
> |ColAndColBench|122510| 87014| 28.9%|
> |IfExprLongColumnLongColumnBench | 1325759| 1436073| -8.3% |
> |IfExprLongColumnRepeatingLongColumnBench|1397447|1480450| -5.9%|
> |IfExprRepeatingLongColumnLongColumnBench|1401164|1483062| -5.9% |
> |NotColBench|77042.83|51513.28| 33%|
> There are degradation in
> IfExprLongColumnLongColumnBench,IfExprLongColumnRepeatingLongColumnBench,
> IfExprRepeatingLongColumnLongColumnBench, very confused why there is
> degradation on IfExprLongColumnLongColumnBench cases.
> Here we use {{taskset -cp 1 $pid}} to run the benchmark on single core to
> avoid the impact of dynamic CPU frequency scaling.
> my script
> {code}
> export JAVA_HOME=/home/zly/jdk-9.0.1/
> export PATH=$JAVA_HOME/bin:$PATH
> export LD_LIBRARY_PATH=/home/zly/jdk-9.0.1/mylib
> for i in 0 1 2; do
> java -server -XX:UseAVX=3 -jar benchmarks.jar
> org.apache.hive.benchmark.vectorization.VectorizedLogicBench * -wi 10 -i 20
> -f 1 -bm avgt -tu us >log.logic.avx3.single.$i & export pid=$!
> taskset -cp 1 $pid
> wait $pid
> done
> for i in 0 1 2; do
> java -server -XX:UseAVX=2 -jar benchmarks.jar
> org.apache.hive.benchmark.vectorization.VectorizedLogicBench * -wi 10 -i 20
> -f 1 -bm avgt -tu us >log.logic.avx2.single.$i & export pid=$!
> taskset -cp 1 $pid
> wait $pid
> done
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18080) Performance degradation on VectorizedLogicBench#IfExprLongColumnLongColumnBench when AVX512 is enabled
[
https://issues.apache.org/jira/browse/HIVE-18080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263861#comment-16263861
]
liyunzhang commented on HIVE-18080:
---
I use vtune to see the assembly code of following code(If expression)
{code}
import java.util.Random;
/**
* * Created by lzhang66 on 11/13/2017.
* */
public class If {
static long[] in1, in2,in3,out;
public static void main(String[] args){
if( args.length == 3){
long warmIter=Long.parseLong(args[0]);
System.out.println("warmIter num:"+warmIter);
long iter=Long.parseLong(args[1]);
System.out.println("iter num:"+iter);
boolean enableHive10238= Boolean.parseBoolean(args[2]);
long startTime = System.currentTimeMillis();
calc(warmIter, iter,enableHive10238);
long endTime = System.currentTimeMillis();
long totalTime = endTime - startTime;
System.out.println("Total time:"+totalTime);
}else{
System.out.println("2 parameter need. Like java ReductionInt [warmIter]
[iter] [enable");
System.exit(0);
}
}
public static void calc(long warmIter, long iter, boolean enableHive10238)
{
in1 = new long [1026];
in2 = new long [1026];
in3 = new long [1026];
Random rand = new Random(435437646);
for(int i=0; i0? in2[i1]: in3[i1];
}
}
}
}
{code}
run {{java If 5000 5 true}} to enable the patch of HIVE-10238 and run
{{java If 5000 5 false}} to disable the patch of HIVE-10238. Here there is
three parameters, para1 means warmIter number, para2 means Iter number, para3
means with/wot HIVE-10238's patch. In the attached picture, I saw AVX2
instructions of if expression( {code}(in1[i1] - 1L)>0? in2[i1]: in3[i1]{code}).
> Performance degradation on
> VectorizedLogicBench#IfExprLongColumnLongColumnBench when AVX512 is enabled
> --
>
> Key: HIVE-18080
> URL: https://issues.apache.org/jira/browse/HIVE-18080
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
> Attachments: log.logic.avx1.single.0, log_logic.avx1.part
>
>
> Use Xeon(R) Platinum 8180 CPU to test the performance of
> [AVX512|https://en.wikipedia.org/wiki/AVX-512].
> {code}
> #cat /proc/cpuinfo |grep "model name"|head -n 1
> model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
> {code}
> Before that I have compiled hive with JDK9 as JDK9 enables AVX512
> Use hive microbenchmark(HIVE-10189) to evaluate the performance improvement.
> It seems performance(20%+) in cases in
> {{VectorizedArithmeticBench}},{{VectorizedComparisonBench}},{{VectorizedLikeBench}},{{VectorizedLogicBench}}
> execpt
> {{VectorizedLogicBench#IfExprLongColumnLongColumnBench}},{{VectorizedLogicBench#IfExprRepeatingLongColumnLongColumnBench}}
> and
> {{VectorizedLogicBench#IfExprLongColumnRepeatingLongColumnBench}}.The data is
> like following
> When i use Skylake CPU to evaluate the performance improvement of AVX512.
> I found the performance in VectorizedLogicBench is like following
> || ||AVX2 us/op||AVX512 us/op || (AVX2-AVX512)/AVX2||
> |ColAndColBench|122510| 87014| 28.9%|
> |IfExprLongColumnLongColumnBench | 1325759| 1436073| -8.3% |
> |IfExprLongColumnRepeatingLongColumnBench|1397447|1480450| -5.9%|
> |IfExprRepeatingLongColumnLongColumnBench|1401164|1483062| -5.9% |
> |NotColBench|77042.83|51513.28| 33%|
> There are degradation in
> IfExprLongColumnLongColumnBench,IfExprLongColumnRepeatingLongColumnBench,
> IfExprRepeatingLongColumnLongColumnBench, very confused why there is
> degradation on IfExprLongColumnLongColumnBench cases.
> Here we use {{taskset -cp 1 $pid}} to run the benchmark on single core to
> avoid the impact of dynamic CPU frequency scaling.
> my script
> {code}
> export JAVA_HOME=/home/zly/jdk-9.0.1/
> export PATH=$JAVA_HOME/bin:$PATH
> export
[jira] [Commented] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[
https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263860#comment-16263860
]
Hive QA commented on HIVE-17994:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} serde: The patch generated 2 new + 4 unchanged - 1
fixed = 6 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c5c2986 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7982/yetus/diff-checkstyle-serde.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7982/yetus/patch-asflicense-problems.txt
|
| modules | C: serde ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7982/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
> --
>
> Key: HIVE-17994
> URL: https://issues.apache.org/jira/browse/HIVE-17994
> Project: Hive
> Issue Type: Bug
>Reporter: Gopal V
>Assignee: Matt McCline
>Priority: Minor
> Attachments: HIVE-17994.01.patch, vec-serialize-hashmap.png
>
>
> On machines with slower NUMA, the hashmap lookup for
> TypeInfo::getPrimitiveCategory is the slowest part of the vectorized
> serialization loops. The static object references run hot with the NUMA
> access speeds penalizing half the threads.
> This lookup is done for every column, for every row - though vectorization
> enforces that this type cannot change at all.
> !vec-serialize-hashmap.png!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263851#comment-16263851 ] Manikumar commented on HIVE-16708: -- [~vihangk1] Thanks for working on this. > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16708.01-branch-2.patch > > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17528) Add more q-tests for Hive-on-Spark with Parquet vectorized reader
[
https://issues.apache.org/jira/browse/HIVE-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263846#comment-16263846
]
Hive QA commented on HIVE-17528:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898408/HIVE-17528.5.patch
{color:green}SUCCESS:{color} +1 due to 30 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11466 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=227)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=233)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=233)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7981/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7981/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7981/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898408 - PreCommit-HIVE-Build
> Add more q-tests for Hive-on-Spark with Parquet vectorized reader
> -
>
> Key: HIVE-17528
> URL: https://issues.apache.org/jira/browse/HIVE-17528
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
> Attachments: HIVE-17528.1.patch, HIVE-17528.2.patch,
> HIVE-17528.3.patch, HIVE-17528.4.patch, HIVE-17528.5.patch, HIVE-17528.patch
>
>
> Most of the vectorization related q-tests operate on ORC tables using Tez. It
> would be good to add more coverage on a different combination of engine and
> file-format. We can model existing q-tests using parquet tables and run it
> using TestSparkCliDriver
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17528) Add more q-tests for Hive-on-Spark with Parquet vectorized reader
[
https://issues.apache.org/jira/browse/HIVE-17528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263841#comment-16263841
]
Hive QA commented on HIVE-17528:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
2s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
26s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
28s{color} | {color:red} root: The patch generated 1 new + 191 unchanged - 1
fixed = 192 total (was 192) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} itests/util: The patch generated 1 new + 191 unchanged
- 1 fixed = 192 total (was 192) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 29 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
2s{color} | {color:red} The patch 298 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c5c2986 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7981/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7981/yetus/diff-checkstyle-itests_util.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7981/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7981/yetus/whitespace-tabs.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7981/yetus/patch-asflicense-problems.txt
|
| modules | C: ql . itests itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7981/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add more q-tests for Hive-on-Spark with Parquet vectorized reader
> -
>
> Key: HIVE-17528
> URL: https://issues.apache.org/jira/browse/HIVE-17528
> Project: Hive
> Issue Type: Sub-task
>Reporter: Vihang Karajgaonkar
>Assignee: Ferdinand Xu
> Attachments: HIVE-17528.1.patch, HIVE-17528.2.patch,
>
[jira] [Comment Edited] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263833#comment-16263833 ] Harish Jaiprakash edited comment on HIVE-18134 at 11/23/17 5:26 AM: Thanks Sergey. I've to make similar changes for stripping quotes at other places like policy. Just one comment, Should we use identifier to restrict to top level or poolPath to enable any path to be parsed, instead of String? I've changed alter resource plan in HIVE-17954, to check for disabled state for not status changes at the API end. was (Author: harishjp): Thanks Sergey. I've to make similar changes for stripping quotes at other places like policy. Just one comment, Should we use identifier to restrict to top level or poolPath to enable any path to be parsed, instead of String. I've changed alter resource plan in HIVE-17954, to check for disabled state for not status changes and made it bit more intuitive. > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263833#comment-16263833 ] Harish Jaiprakash commented on HIVE-18134: -- Thanks Sergey. I've to make similar changes for stripping quotes at other places like policy. Just one comment, Should we use identifier to restrict to top level or poolPath to enable any path to be parsed, instead of String. I've changed alter resource plan in HIVE-17954, to check for disabled state for not status changes and made it bit more intuitive. > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-13665) HS2 memory leak When multiple queries are running with get_json_object
[
https://issues.apache.org/jira/browse/HIVE-13665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263831#comment-16263831
]
靳峥 commented on HIVE-13665:
---
Hi, JinsuKim, I have met the same problem these days, and I'm also trying to
figure out the best solution.
I am afraid the performance of `get_json_object` will be slowed down heavily
with this patch, when running in multi-thread mode.
I have even written a ConcurrentLRUHashMap to fulfill a distributed cache, but
it does not behave as efficiently as I expected.
Perhaps making HashCache not static is an appropriate way.
> HS2 memory leak When multiple queries are running with get_json_object
> --
>
> Key: HIVE-13665
> URL: https://issues.apache.org/jira/browse/HIVE-13665
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0
>Reporter: JinsuKim
> Attachments: patch.lst.txt
>
>
> The extractObjectCache in UDFJson is increased over limitation(CACHE_SIZE =
> 16). When multiple queries are running concurrently on HS2 local(not mr/tez)
> with get_json_object or get_json_tuple
> {code:java|title=HS2 heap_dump}
> Object at 0x515ab18f8
> instance of org.apache.hadoop.hive.ql.udf.UDFJson$HashCache@0x515ab18f8 (77
> bytes)
> Class:
> class org.apache.hadoop.hive.ql.udf.UDFJson$HashCache
> Instance data members:
> accessOrder (Z) : false
> entrySet (L) :
> hashSeed (I) : 0
> header (L) : java.util.LinkedHashMap$Entry@0x515a577d0 (60 bytes)
> keySet (L) :
> loadFactor (F) : 0.6
> modCount (I) : 4741146
> size (I) : 2733158 <== here!!
> table (L) : [Ljava.util.HashMap$Entry;@0x7163d8b70 (67108880 bytes)
> threshold (I) : 5033165
> values (L) :
> References to this object:
> {code}
> I think that this problem be caused by the LinkedHashMap object is not
> thread-safe
> {code}
> * Note that this implementation is not synchronized.
> * If multiple threads access a linked hash map concurrently, and at least
> * one of the threads modifies the map structurally, it must be
> * synchronized externally. This is typically accomplished by
> * synchronizing on some object that naturally encapsulates the map.
> {code}
> Reproduce :
> # Multiple queries are running with get_json_object and small input data(for
> execution on hs2 local mode)
> # jvm heap dump & analyze
> {code:title=test scenario}
> Multiple queries are running with get_json_object and small input data(for
> execute on hs2 local mode)
> 1.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040105'
> 2.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040106'
> 3.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040107'
> 4.hql :
> SELECT get_json_object(body, '$.fileSize'), get_json_object(body,
> '$.ps_totalTimeSeconds'), get_json_object(body, '$.totalTimeSeconds') FROM
> xxx. WHERE part_hour='2016040108'
>
> run.sh :
> t_cnt=0
> while true
> do
> echo "query executing..."
> for i in 1 2 3 4
> do
> beeline -u jdbc:hive2://localhost:1 -n hive --silent=true -f
> $i.hql > $i.log 2>&1 &
> done
> wait
> t_cnt=`expr $t_cnt + 1`
> echo "query count : $t_cnt"
> sleep 2
> done
> jvm heap dump & analyze :
> jmap -dump:format=b,file=hive.dmp $PID
> jhat -J-mx48000m -port 8080 hive.dmp &
> {code}
> Finally I have attached our patch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-6058) Creating Table with column name exchange gives exception
[ https://issues.apache.org/jira/browse/HIVE-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263824#comment-16263824 ] Sumit commented on HIVE-6058: - Hi , We want to create column name as exchange. What is the alternative for this ? Please do not suggest change of column name > Creating Table with column nameexchange gives exception > > > Key: HIVE-6058 > URL: https://issues.apache.org/jira/browse/HIVE-6058 > Project: Hive > Issue Type: Bug > Components: Query Processor >Affects Versions: 0.12.0 > Environment: Hadoop 2.0.1-beta, Hive 0.12 , running hiveserver , > Installed on CentOS6.4 64-bit OS in Multinode configuration >Reporter: Anilkumar Kalshetti > Attachments: table_columnname_exchange_exception.png > > > 1. Create Table using below script > CREATE TABLE test_column_name2 ( c1 string) ROW FORMAT DELIMITED STORED AS > SEQUENCEFILE; > 2. Table is created successfully. > Now Create another Table with column name exchange > by using below script > CREATE TABLE test_column_name ( exchange string) ROW FORMAT DELIMITED > STORED AS SEQUENCEFILE; > 3. This gives exception for column name exchange, for Hive 0.12 > 4. The script properly works for Hive 0.11 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18029) beeline - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263811#comment-16263811
]
Hive QA commented on HIVE-18029:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898973/HIVE-18029.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7980/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7980/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7980/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898973 - PreCommit-HIVE-Build
> beeline - support proper usernames based on URL
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18029.patch
>
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-16708: --- Status: Patch Available (was: Open) > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16708.01-branch-2.patch > > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-16708: --- Attachment: HIVE-16708.01-branch-2.patch > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16708.01-branch-2.patch > > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263802#comment-16263802 ] Vihang Karajgaonkar commented on HIVE-16708: I think is caused due to inconsistency in the renewToken method in hadoop-common. I have created a JIRA to get this fixed in hadoop. Meanwhile I will provide a patch for Hive to workaround this problem. > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-16708.01-branch-2.patch > > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263785#comment-16263785 ] Eugene Koifman commented on HIVE-17361: --- patch 23 - all failures have age > 1 Still needs some clean up/refactoring > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Wei Zheng >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, > HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, > HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, > HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, > HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch, > HIVE-17361.23.patch, HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. > Current Documentation is under [DML > Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] > and [Loading files into > tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: > \\ > * Load Data performs very limited validations of the data, in particular it > uses the input file name which may not be in 0_0 which can break some > read logic. (Certainly will for Acid). > * It does not check the schema of the file. This may be a non issue for Acid > which requires ORC which is self describing so Schema Evolution may handle > this seamlessly. (Assuming Schema is not too different). > * It does check that _InputFormat_S are compatible. > * Bucketed (and thus sorted) tables don't support Load Data (but only if > hive.strict.checks.bucketing=true (default)). Will keep this restriction for > Acid. > * Load Data supports OVERWRITE clause > * What happens to file permissions/ownership: rename vs copy differences > \\ > The implementation will follow the same idea as in HIVE-14988 and use a > base_N/ dir for OVERWRITE clause. > \\ > How is minor compaction going to handle delta/base with original files? > Since delta_8_8/_meta_data is created before files are moved, delta_8_8 > becomes visible before it's populated. Is that an issue? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18029) beeline - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263772#comment-16263772
]
Hive QA commented on HIVE-18029:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 7m 42s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c5c2986 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7980/yetus/patch-asflicense-problems.txt
|
| modules | C: beeline U: beeline |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7980/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> beeline - support proper usernames based on URL
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18029.patch
>
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18026) Hive webhcat principal configuration optimization
[ https://issues.apache.org/jira/browse/HIVE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263771#comment-16263771 ] ZhangBing Lin commented on HIVE-18026: -- Thank you,[~lirui]. > Hive webhcat principal configuration optimization > - > > Key: HIVE-18026 > URL: https://issues.apache.org/jira/browse/HIVE-18026 > Project: Hive > Issue Type: Bug > Components: WebHCat >Affects Versions: 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin > Fix For: 3.0.0 > > Attachments: HIVE-18026.1.patch, HIVE-18026.2.patch > > > Hive webhcat principal configuration optimization，when you configure: > > templeton.kerberos.principal > HTTP/_HOST@ > > The '_HOST' should be replaced by specific host name. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263762#comment-16263762
]
Hive QA commented on HIVE-17361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898970/HIVE-17361.23.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 11418 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7979/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7979/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7979/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898970 - PreCommit-HIVE-Build
> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch,
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch,
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch,
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch,
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch,
> HIVE-17361.23.patch, HIVE-17361.3.patch, HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap
> between ACID table and regular hive table.
> Current Documentation is under [DML
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
> and [Loading files into
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it
> uses the input file name which may not be in 0_0 which can break some
> read logic. (Certainly will for Acid).
> * It does not check the schema of the file. This may be a non issue for Acid
> which requires ORC which is self describing so Schema Evolution may handle
> this seamlessly. (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible.
> * Bucketed (and thus sorted) tables don't support Load Data (but only if
> hive.strict.checks.bucketing=true (default)). Will keep this restriction for
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created before files are moved, delta_8_8
> becomes visible before it's populated. Is that an issue?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18026) Hive webhcat principal configuration optimization
[ https://issues.apache.org/jira/browse/HIVE-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-18026: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks [~linzhangbing] for the contribution. > Hive webhcat principal configuration optimization > - > > Key: HIVE-18026 > URL: https://issues.apache.org/jira/browse/HIVE-18026 > Project: Hive > Issue Type: Bug > Components: WebHCat >Affects Versions: 3.0.0 >Reporter: ZhangBing Lin >Assignee: ZhangBing Lin > Fix For: 3.0.0 > > Attachments: HIVE-18026.1.patch, HIVE-18026.2.patch > > > Hive webhcat principal configuration optimization，when you configure: > > templeton.kerberos.principal > HTTP/_HOST@ > > The '_HOST' should be replaced by specific host name. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Work started] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-17994 started by Matt McCline. --- > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17994: Attachment: HIVE-17994.01.patch > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-17994: Status: Patch Available (was: In Progress) > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: HIVE-17994.01.patch, vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263732#comment-16263732
]
Hive QA commented on HIVE-17361:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
5s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 116 new + 1404 unchanged - 24
fixed = 1520 total (was 1428) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a988c15 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7979/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7979/yetus/patch-asflicense-problems.txt
|
| modules | C: common standalone-metastore ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7979/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch,
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch,
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch,
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch,
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch,
> HIVE-17361.23.patch, HIVE-17361.3.patch, HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap
> between ACID table and regular hive table.
> Current Documentation is under [DML
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
> and [Loading files into
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it
> uses the input file name
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[
https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263708#comment-16263708
]
Hive QA commented on HIVE-18134:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898958/HIVE-18134.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7978/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7978/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7978/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898958 - PreCommit-HIVE-Build
> some alter resource plan fixes
> --
>
> Key: HIVE-18134
> URL: https://issues.apache.org/jira/browse/HIVE-18134
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18134.patch
>
>
> Part of HIVE-18075
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated
[ https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263694#comment-16263694 ] Sergey Shelukhin commented on HIVE-18076: - Looks like there's some bug with reopen that is triggered during the test (not sure why). Will check/fix based on logs; doesn't happen for me locally. > killquery doesn't actually work for non-trigger WM kills, or the error > message is not propagated > > > Key: HIVE-18076 > URL: https://issues.apache.org/jira/browse/HIVE-18076 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18076.01.patch, HIVE-18076.patch > > > Not sure what's wrong with it, need to take a look. > It dumps a lot of info about everything being cancelled, instead of a nice > message like triggers do. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18134) some alter resource plan fixes
[
https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263671#comment-16263671
]
Hive QA commented on HIVE-18134:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
11s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a988c15 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7978/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7978/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> some alter resource plan fixes
> --
>
> Key: HIVE-18134
> URL: https://issues.apache.org/jira/browse/HIVE-18134
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18134.patch
>
>
> Part of HIVE-18075
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Resolved] (HIVE-17734) Add trigger type to WM_TRIGGER table
[ https://issues.apache.org/jira/browse/HIVE-17734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin resolved HIVE-17734. - Resolution: Duplicate > Add trigger type to WM_TRIGGER table > > > Key: HIVE-17734 > URL: https://issues.apache.org/jira/browse/HIVE-17734 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran > > Following up with HIVE-17566, it will be useful to add a 'type' to WM_TRIGGER > table to represent the type of the trigger. HIVE-17508 added 'execution' > triggers but there could be other triggers that can be added later. This will > avoid fetching all triggers and filtering at client side. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18020) print per query workload management trace after query execution
[ https://issues.apache.org/jira/browse/HIVE-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263665#comment-16263665 ] Sergey Shelukhin commented on HIVE-18020: - Is this a duplicate of HIVE-18088? > print per query workload management trace after query execution > --- > > Key: HIVE-18020 > URL: https://issues.apache.org/jira/browse/HIVE-18020 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran > > For debugging and testing purposes, print some summary (pool, cluster > fraction traces) post query execution to console. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18078) WM getSession needs some retry logic
[ https://issues.apache.org/jira/browse/HIVE-18078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18078: --- Assignee: Sergey Shelukhin > WM getSession needs some retry logic > > > Key: HIVE-18078 > URL: https://issues.apache.org/jira/browse/HIVE-18078 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > > When we get a bad session (e.g. no registry info because AM has gone > catatonic), the failure by the timeout future fails the getSession call. > The retry model in TezTask is that it would get a session (which in original > model can be completely unusable, but we still get the object), and then > retry (reopen) if it's a lemon. If the reopen fails, we fail. > getSession is not covered by this retry scheme, and should thus do its own > retries (or the retry logic needs to be changed) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18029) beeline - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18029:
Status: Patch Available (was: Open)
[~thejas] can you take a look?
A small patch.
We might want to also make it consistent w.r.t. prompts (when you connect to an
url w/o username or password there's a prompt, but when you use the "u"
argument w/o username or password it just adds the blank values), but this can
be done in a separate jira if desired.
> beeline - support proper usernames based on URL
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18029.patch
>
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18029) user mapping - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18029:
Attachment: HIVE-18029.patch
The method that builds the command seems to override the url stuff with blank
values. Extracting the values to avoid that.
> user mapping - support proper usernames based on URL
>
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18029.patch
>
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18029) beeline - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18029:
Summary: beeline - support proper usernames based on URL (was: user
mapping - support proper usernames based on URL)
> beeline - support proper usernames based on URL
> ---
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18029.patch
>
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver
[ https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263657#comment-16263657 ] Andrew Sherman commented on HIVE-18127: --- Test failures are not caused by this change. [~stakiar] please review when you have time > Do not strip '--' comments from shell commands issued from CliDriver > > > Key: HIVE-18127 > URL: https://issues.apache.org/jira/browse/HIVE-18127 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-18127.1.patch, HIVE-18127.2.patch > > > CLiDriver has the ability to run shell commands by prefixing them with '!". > This behavior is not widely used (there are only 3 examples in .q files). > Since HIVE-16935 started stripping comments starting with '\-\-', a shell > command containing '--' will not work correctly. > Fix this by using the unstripped command for shell commands. > Note that it would be a security hole for HS2 to allow execution of arbitrary > shell commands from a client command. > Add tests to nail down correct behavior with '--' comments: > * CliDriver should not strip strings starting with '--' in a shell command > (FIXED in this change). > * HiveCli should strip '--' comments. > * A Jdbc program should allow commands starting with "!" but these will fail > in the sql parser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver
[ https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263656#comment-16263656 ] Andrew Sherman commented on HIVE-18127: --- logged [HIVE-18136] for asflicense issue > Do not strip '--' comments from shell commands issued from CliDriver > > > Key: HIVE-18127 > URL: https://issues.apache.org/jira/browse/HIVE-18127 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-18127.1.patch, HIVE-18127.2.patch > > > CLiDriver has the ability to run shell commands by prefixing them with '!". > This behavior is not widely used (there are only 3 examples in .q files). > Since HIVE-16935 started stripping comments starting with '\-\-', a shell > command containing '--' will not work correctly. > Fix this by using the unstripped command for shell commands. > Note that it would be a security hole for HS2 to allow execution of arbitrary > shell commands from a client command. > Add tests to nail down correct behavior with '--' comments: > * CliDriver should not strip strings starting with '--' in a shell command > (FIXED in this change). > * HiveCli should strip '--' comments. > * A Jdbc program should allow commands starting with "!" but these will fail > in the sql parser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18136) WorkloadManagerMxBean is missing the Apache license header
[
https://issues.apache.org/jira/browse/HIVE-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263654#comment-16263654
]
Hive QA commented on HIVE-18136:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898953/HIVE-18136.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7977/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7977/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7977/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898953 - PreCommit-HIVE-Build
> WorkloadManagerMxBean is missing the Apache license header
> --
>
> Key: HIVE-18136
> URL: https://issues.apache.org/jira/browse/HIVE-18136
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18136.1.patch
>
>
> This causes warnings in the yetus check:
> {quote}Lines that start with ? in the ASF License report indicate files
> that do not have an Apache license header:
> !?
> /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerMxBean.java{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Attachment: HIVE-17361.23.patch > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Wei Zheng >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, > HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, > HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, > HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, > HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch, > HIVE-17361.23.patch, HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. > Current Documentation is under [DML > Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] > and [Loading files into > tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: > \\ > * Load Data performs very limited validations of the data, in particular it > uses the input file name which may not be in 0_0 which can break some > read logic. (Certainly will for Acid). > * It does not check the schema of the file. This may be a non issue for Acid > which requires ORC which is self describing so Schema Evolution may handle > this seamlessly. (Assuming Schema is not too different). > * It does check that _InputFormat_S are compatible. > * Bucketed (and thus sorted) tables don't support Load Data (but only if > hive.strict.checks.bucketing=true (default)). Will keep this restriction for > Acid. > * Load Data supports OVERWRITE clause > * What happens to file permissions/ownership: rename vs copy differences > \\ > The implementation will follow the same idea as in HIVE-14988 and use a > base_N/ dir for OVERWRITE clause. > \\ > How is minor compaction going to handle delta/base with original files? > Since delta_8_8/_meta_data is created before files are moved, delta_8_8 > becomes visible before it's populated. Is that an issue? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18136) WorkloadManagerMxBean is missing the Apache license header
[
https://issues.apache.org/jira/browse/HIVE-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263625#comment-16263625
]
Hive QA commented on HIVE-18136:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
35s{color} | {color:red} ql: The patch generated 1 new + 1 unchanged - 1 fixed
= 2 total (was 2) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a988c15 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7977/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7977/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> WorkloadManagerMxBean is missing the Apache license header
> --
>
> Key: HIVE-18136
> URL: https://issues.apache.org/jira/browse/HIVE-18136
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18136.1.patch
>
>
> This causes warnings in the yetus check:
> {quote}Lines that start with ? in the ASF License report indicate files
> that do not have an Apache license header:
> !?
> /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerMxBean.java{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated
[
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263615#comment-16263615
]
Hive QA commented on HIVE-18076:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898949/HIVE-18076.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers1
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsUnionAll
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedFiles
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomNonExistent
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighBytesRead
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerHighShuffleBytes
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryElapsedTime
(batchId=231)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerSlowQueryExecutionTime
(batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7976/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7976/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7976/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898949 - PreCommit-HIVE-Build
> killquery doesn't actually work for non-trigger WM kills, or the error
> message is not propagated
>
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.01.patch, HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice
> message like triggers do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Issue Comment Deleted] (HIVE-18029) user mapping - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18029:
Comment: was deleted
(was: Actually even better, it works when the password is not specified... sigh)
> user mapping - support proper usernames based on URL
>
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18029) user mapping - support proper usernames with doAs = false
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263607#comment-16263607
]
Sergey Shelukhin commented on HIVE-18029:
-
Actually even better, it works when the password is not specified... sigh
> user mapping - support proper usernames with doAs = false
> -
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18029) user mapping - support proper usernames based on URL
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-18029:
Summary: user mapping - support proper usernames based on URL (was: user
mapping - support proper usernames with doAs = false)
> user mapping - support proper usernames based on URL
>
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18029) user mapping - support proper usernames with doAs = false
[
https://issues.apache.org/jira/browse/HIVE-18029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263598#comment-16263598
]
Sergey Shelukhin commented on HIVE-18029:
-
Ok, actually the behavior is more subtle; looks like the argument connection
URL is not handled consistently with the connect command one, and the latter
passes on user name that's entered in the prompt correctly.
So,
{noformat}
!connect (url);user=foo => foo
!connect (url) => whatever is entered in the prompt
beeline -n foo -u (url with or without the user) => foo
beeline -u (url);user=foo => anonymous
beeline -u (url) => anonymous (no prompt)
{noformat}
I'm going to add support for extracting the user from the -u argument, similar
to connect argument
> user mapping - support proper usernames with doAs = false
> -
>
> Key: HIVE-18029
> URL: https://issues.apache.org/jira/browse/HIVE-18029
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>
> Right now what happens on unsecure cluster with doAs=false (not sure which
> one is to blame - didn't look into it, maybe both) is {noformat}
> 2017-11-08T21:39:49,404 INFO [HiveServer2-Background-Pool: Thread-205]
> tez.WorkloadManagerFederation: Getting a WM session for anonymous
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18076) killquery doesn't actually work for non-trigger WM kills, or the error message is not propagated
[
https://issues.apache.org/jira/browse/HIVE-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263552#comment-16263552
]
Hive QA commented on HIVE-18076:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} ql: The patch generated 25 new + 142 unchanged - 21
fixed = 167 total (was 163) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a988c15 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7976/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7976/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7976/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> killquery doesn't actually work for non-trigger WM kills, or the error
> message is not propagated
>
>
> Key: HIVE-18076
> URL: https://issues.apache.org/jira/browse/HIVE-18076
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-18076.01.patch, HIVE-18076.patch
>
>
> Not sure what's wrong with it, need to take a look.
> It dumps a lot of info about everything being cancelled, instead of a nice
> message like triggers do.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18075) verify commands on a cluster
[ https://issues.apache.org/jira/browse/HIVE-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263452#comment-16263452 ] Sergey Shelukhin edited comment on HIVE-18075 at 11/22/17 11:21 PM: HIVE-17954 probably fixes some/all of these issues. We'll verify the commands after that patch is committed. Fixed 4-5 in a separate jira was (Author: sershe): HIVE-17954 probably fixes some/all of these issues. We'll verify the commands after that patch is committed. > verify commands on a cluster > > > Key: HIVE-18075 > URL: https://issues.apache.org/jira/browse/HIVE-18075 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > > I was running the commands in the cluster, with potentially a slightly > outdated version of the DB on mysql (yet, the DB was created with a script > from master and I didn't hit any schema issues, so it's fairly recent); > however, recent master code + the pools commands patch. I've hit the > following issues. > # Cannot drop pool or RP with a mapping (see also 3). > # Cannot drop pool that is set as default (probably correct, but the error > message is bad). > # When I dropped an RP with a mapping, and then created it again with the > same name, the pool creation in that RP would fail with an error that a > unique query returned multiple results. In the DB, there were actually 2 RPs > with the same name. Not sure how exactly that happened, there might have been > intermediate states, but I didn't mess with mysql. I think the name > uniqueness is either missing from some script or doesn't work. > # -Setting RP default pool no longer works. I think I might have broken it > with one of the rebases in that area, but it could also be something else (or > like other things, it works in q tests but not on cluster for whatever > reason-). > # -Resource plan rename doesn't check the disable state. It probably should. > Also need to see for other commands-. > Need to figure out which are real issues and which aren't and fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18134: Status: Patch Available (was: Open) > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18134: Attachment: HIVE-18134.patch [~prasanth_j] [~harishjp] can you take a look? > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18134: Description: Part of HIVE-18075 > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-18134.patch > > > Part of HIVE-18075 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver
[
https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263534#comment-16263534
]
Hive QA commented on HIVE-18127:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898930/HIVE-18127.2.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11413 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=246)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7975/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7975/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7975/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898930 - PreCommit-HIVE-Build
> Do not strip '--' comments from shell commands issued from CliDriver
>
>
> Key: HIVE-18127
> URL: https://issues.apache.org/jira/browse/HIVE-18127
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18127.1.patch, HIVE-18127.2.patch
>
>
> CLiDriver has the ability to run shell commands by prefixing them with '!".
> This behavior is not widely used (there are only 3 examples in .q files).
> Since HIVE-16935 started stripping comments starting with '\-\-', a shell
> command containing '--' will not work correctly.
> Fix this by using the unstripped command for shell commands.
> Note that it would be a security hole for HS2 to allow execution of arbitrary
> shell commands from a client command.
> Add tests to nail down correct behavior with '--' comments:
> * CliDriver should not strip strings starting with '--' in a shell command
> (FIXED in this change).
> * HiveCli should strip '--' comments.
> * A Jdbc program should allow commands starting with "!" but these will fail
> in the sql parser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18136) WorkloadManagerMxBean is missing the Apache license header
[
https://issues.apache.org/jira/browse/HIVE-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263530#comment-16263530
]
Andrew Sherman commented on HIVE-18136:
---
Hi [~sershe] can you review and push if you approve? Thanks
> WorkloadManagerMxBean is missing the Apache license header
> --
>
> Key: HIVE-18136
> URL: https://issues.apache.org/jira/browse/HIVE-18136
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18136.1.patch
>
>
> This causes warnings in the yetus check:
> {quote}Lines that start with ? in the ASF License report indicate files
> that do not have an Apache license header:
> !?
> /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerMxBean.java{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline reassigned HIVE-17994: --- Assignee: Matt McCline > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Assignee: Matt McCline >Priority: Minor > Attachments: vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18136) WorkloadManagerMxBean is missing the Apache license header
[
https://issues.apache.org/jira/browse/HIVE-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18136:
--
Status: Patch Available (was: Open)
> WorkloadManagerMxBean is missing the Apache license header
> --
>
> Key: HIVE-18136
> URL: https://issues.apache.org/jira/browse/HIVE-18136
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18136.1.patch
>
>
> This causes warnings in the yetus check:
> {quote}Lines that start with ? in the ASF License report indicate files
> that do not have an Apache license header:
> !?
> /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerMxBean.java{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18136) WorkloadManagerMxBean is missing the Apache license header
[
https://issues.apache.org/jira/browse/HIVE-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman updated HIVE-18136:
--
Attachment: HIVE-18136.1.patch
> WorkloadManagerMxBean is missing the Apache license header
> --
>
> Key: HIVE-18136
> URL: https://issues.apache.org/jira/browse/HIVE-18136
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18136.1.patch
>
>
> This causes warnings in the yetus check:
> {quote}Lines that start with ? in the ASF License report indicate files
> that do not have an Apache license header:
> !?
> /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerMxBean.java{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18136) WorkloadManagerMxBean is missing the Apache license header
[
https://issues.apache.org/jira/browse/HIVE-18136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Sherman reassigned HIVE-18136:
-
> WorkloadManagerMxBean is missing the Apache license header
> --
>
> Key: HIVE-18136
> URL: https://issues.apache.org/jira/browse/HIVE-18136
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
>
> This causes warnings in the yetus check:
> {quote}Lines that start with ? in the ASF License report indicate files
> that do not have an Apache license header:
> !?
> /data/hiveptest/working/yetus/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManagerMxBean.java{quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-18134) some alter resource plan fixes
[ https://issues.apache.org/jira/browse/HIVE-18134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-18134: --- > some alter resource plan fixes > -- > > Key: HIVE-18134 > URL: https://issues.apache.org/jira/browse/HIVE-18134 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18075) verify commands on a cluster
[ https://issues.apache.org/jira/browse/HIVE-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18075: Description: I was running the commands in the cluster, with potentially a slightly outdated version of the DB on mysql (yet, the DB was created with a script from master and I didn't hit any schema issues, so it's fairly recent); however, recent master code + the pools commands patch. I've hit the following issues. # Cannot drop pool or RP with a mapping (see also 3). # Cannot drop pool that is set as default (probably correct, but the error message is bad). # When I dropped an RP with a mapping, and then created it again with the same name, the pool creation in that RP would fail with an error that a unique query returned multiple results. In the DB, there were actually 2 RPs with the same name. Not sure how exactly that happened, there might have been intermediate states, but I didn't mess with mysql. I think the name uniqueness is either missing from some script or doesn't work. # -Setting RP default pool no longer works. I think I might have broken it with one of the rebases in that area, but it could also be something else (or like other things, it works in q tests but not on cluster for whatever reason-). # Resource plan rename doesn't check the disable state. It probably should. Also need to see for other commands. Need to figure out which are real issues and which aren't and fix. was: I was running the commands in the cluster, with potentially a slightly outdated version of the DB on mysql (yet, the DB was created with a script from master and I didn't hit any schema issues, so it's fairly recent); however, recent master code + the pools commands patch. I've hit the following issues. # Cannot drop pool or RP with a mapping (see also 3). # Cannot drop pool that is set as default (probably correct, but the error message is bad). # When I dropped an RP with a mapping, and then created it again with the same name, the pool creation in that RP would fail with an error that a unique query returned multiple results. In the DB, there were actually 2 RPs with the same name. Not sure how exactly that happened, there might have been intermediate states, but I didn't mess with mysql. I think the name uniqueness is either missing from some script or doesn't work. # Setting RP default pool no longer works. I think I might have broken it with one of the rebases in that area, but it could also be something else (or like other things, it works in q tests but not on cluster for whatever reason). # Resource plan rename doesn't check the disable state. It probably should. Also need to see for other commands. Need to figure out which are real issues and which aren't and fix. > verify commands on a cluster > > > Key: HIVE-18075 > URL: https://issues.apache.org/jira/browse/HIVE-18075 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > > I was running the commands in the cluster, with potentially a slightly > outdated version of the DB on mysql (yet, the DB was created with a script > from master and I didn't hit any schema issues, so it's fairly recent); > however, recent master code + the pools commands patch. I've hit the > following issues. > # Cannot drop pool or RP with a mapping (see also 3). > # Cannot drop pool that is set as default (probably correct, but the error > message is bad). > # When I dropped an RP with a mapping, and then created it again with the > same name, the pool creation in that RP would fail with an error that a > unique query returned multiple results. In the DB, there were actually 2 RPs > with the same name. Not sure how exactly that happened, there might have been > intermediate states, but I didn't mess with mysql. I think the name > uniqueness is either missing from some script or doesn't work. > # -Setting RP default pool no longer works. I think I might have broken it > with one of the rebases in that area, but it could also be something else (or > like other things, it works in q tests but not on cluster for whatever > reason-). > # Resource plan rename doesn't check the disable state. It probably should. > Also need to see for other commands. > Need to figure out which are real issues and which aren't and fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18075) verify commands on a cluster
[ https://issues.apache.org/jira/browse/HIVE-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18075: Description: I was running the commands in the cluster, with potentially a slightly outdated version of the DB on mysql (yet, the DB was created with a script from master and I didn't hit any schema issues, so it's fairly recent); however, recent master code + the pools commands patch. I've hit the following issues. # Cannot drop pool or RP with a mapping (see also 3). # Cannot drop pool that is set as default (probably correct, but the error message is bad). # When I dropped an RP with a mapping, and then created it again with the same name, the pool creation in that RP would fail with an error that a unique query returned multiple results. In the DB, there were actually 2 RPs with the same name. Not sure how exactly that happened, there might have been intermediate states, but I didn't mess with mysql. I think the name uniqueness is either missing from some script or doesn't work. # -Setting RP default pool no longer works. I think I might have broken it with one of the rebases in that area, but it could also be something else (or like other things, it works in q tests but not on cluster for whatever reason-). # -Resource plan rename doesn't check the disable state. It probably should. Also need to see for other commands-. Need to figure out which are real issues and which aren't and fix. was: I was running the commands in the cluster, with potentially a slightly outdated version of the DB on mysql (yet, the DB was created with a script from master and I didn't hit any schema issues, so it's fairly recent); however, recent master code + the pools commands patch. I've hit the following issues. # Cannot drop pool or RP with a mapping (see also 3). # Cannot drop pool that is set as default (probably correct, but the error message is bad). # When I dropped an RP with a mapping, and then created it again with the same name, the pool creation in that RP would fail with an error that a unique query returned multiple results. In the DB, there were actually 2 RPs with the same name. Not sure how exactly that happened, there might have been intermediate states, but I didn't mess with mysql. I think the name uniqueness is either missing from some script or doesn't work. # -Setting RP default pool no longer works. I think I might have broken it with one of the rebases in that area, but it could also be something else (or like other things, it works in q tests but not on cluster for whatever reason-). # Resource plan rename doesn't check the disable state. It probably should. Also need to see for other commands. Need to figure out which are real issues and which aren't and fix. > verify commands on a cluster > > > Key: HIVE-18075 > URL: https://issues.apache.org/jira/browse/HIVE-18075 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > > I was running the commands in the cluster, with potentially a slightly > outdated version of the DB on mysql (yet, the DB was created with a script > from master and I didn't hit any schema issues, so it's fairly recent); > however, recent master code + the pools commands patch. I've hit the > following issues. > # Cannot drop pool or RP with a mapping (see also 3). > # Cannot drop pool that is set as default (probably correct, but the error > message is bad). > # When I dropped an RP with a mapping, and then created it again with the > same name, the pool creation in that RP would fail with an error that a > unique query returned multiple results. In the DB, there were actually 2 RPs > with the same name. Not sure how exactly that happened, there might have been > intermediate states, but I didn't mess with mysql. I think the name > uniqueness is either missing from some script or doesn't work. > # -Setting RP default pool no longer works. I think I might have broken it > with one of the rebases in that area, but it could also be something else (or > like other things, it works in q tests but not on cluster for whatever > reason-). > # -Resource plan rename doesn't check the disable state. It probably should. > Also need to see for other commands-. > Need to figure out which are real issues and which aren't and fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver
[
https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263467#comment-16263467
]
Hive QA commented on HIVE-18127:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} cli: The patch generated 1 new + 37 unchanged - 1
fixed = 38 total (was 38) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a988c15 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7975/yetus/diff-checkstyle-cli.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7975/yetus/patch-asflicense-problems.txt
|
| modules | C: beeline cli itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7975/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Do not strip '--' comments from shell commands issued from CliDriver
>
>
> Key: HIVE-18127
> URL: https://issues.apache.org/jira/browse/HIVE-18127
> Project: Hive
> Issue Type: Bug
>Reporter: Andrew Sherman
>Assignee: Andrew Sherman
> Attachments: HIVE-18127.1.patch, HIVE-18127.2.patch
>
>
> CLiDriver has the ability to run shell commands by prefixing them with '!".
> This behavior is not widely used (there are only 3 examples in .q files).
> Since HIVE-16935 started stripping comments starting with '\-\-', a shell
> command containing '--' will not work correctly.
> Fix this by using the unstripped command for shell commands.
> Note that it would be a security hole for HS2 to allow execution of arbitrary
> shell commands from a client command.
> Add tests to nail down correct behavior with '--' comments:
> * CliDriver should not strip strings starting with '--' in a shell command
> (FIXED in this change).
> * HiveCli should strip '--' comments.
> * A Jdbc program should allow commands starting with "!" but these will fail
> in the sql parser.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18075) verify commands on a cluster
[ https://issues.apache.org/jira/browse/HIVE-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263452#comment-16263452 ] Sergey Shelukhin commented on HIVE-18075: - HIVE-17954 probably fixes some/all of these issues. We'll verify the commands after that patch is committed. > verify commands on a cluster > > > Key: HIVE-18075 > URL: https://issues.apache.org/jira/browse/HIVE-18075 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin > > I was running the commands in the cluster, with potentially a slightly > outdated version of the DB on mysql (yet, the DB was created with a script > from master and I didn't hit any schema issues, so it's fairly recent); > however, recent master code + the pools commands patch. I've hit the > following issues. > # Cannot drop pool or RP with a mapping (see also 3). > # Cannot drop pool that is set as default (probably correct, but the error > message is bad). > # When I dropped an RP with a mapping, and then created it again with the > same name, the pool creation in that RP would fail with an error that a > unique query returned multiple results. In the DB, there were actually 2 RPs > with the same name. Not sure how exactly that happened, there might have been > intermediate states, but I didn't mess with mysql. I think the name > uniqueness is either missing from some script or doesn't work. > # Setting RP default pool no longer works. I think I might have broken it > with one of the rebases in that area, but it could also be something else (or > like other things, it works in q tests but not on cluster for whatever > reason). > # Resource plan rename doesn't check the disable state. It probably should. > Also need to see for other commands. > Need to figure out which are real issues and which aren't and fix. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18133) Parametrize TestTxnNoBuckets wrt Vectorization
[
https://issues.apache.org/jira/browse/HIVE-18133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman reassigned HIVE-18133:
-
> Parametrize TestTxnNoBuckets wrt Vectorization
> --
>
> Key: HIVE-18133
> URL: https://issues.apache.org/jira/browse/HIVE-18133
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>
> it currently runs in Vector mode only
> {noformat}
> public void setUp() throws Exception {
> setUpInternal();
> hiveConf.setBoolVar(HiveConf.ConfVars.HIVE_VECTORIZATION_ENABLED, true);
> }
> {noformat}
> would be good to run both modes
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263425#comment-16263425 ] Vihang Karajgaonkar commented on HIVE-16708: I am hitting this same issue and I think we should fix this. I will take a look at this. > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18068) Replace LocalInterval by Interval in Druid storage handler
[
https://issues.apache.org/jira/browse/HIVE-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263406#comment-16263406
]
Hive QA commented on HIVE-18068:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} ql: The patch generated 0 new + 415 unchanged - 2
fixed = 415 total (was 417) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
32s{color} | {color:green} root: The patch generated 0 new + 415 unchanged - 2
fixed = 415 total (was 417) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a988c15 |
| Default Java | 1.8.0_111 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7974/yetus/patch-asflicense-problems.txt
|
| modules | C: ql . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7974/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Replace LocalInterval by Interval in Druid storage handler
> --
>
> Key: HIVE-18068
> URL: https://issues.apache.org/jira/browse/HIVE-18068
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-18068.2.patch, HIVE-18068.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16669) Fine tune Compaction to take advantage of Acid 2.0
[ https://issues.apache.org/jira/browse/HIVE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263404#comment-16263404 ] Eugene Koifman commented on HIVE-16669: --- If we have table based WriteIDs, can compactor compact up to the smallest open WriteID in a table rather that smallest open global txn? This would mean compactor is less likely to starve due to long running write transactions (multi statement especially) > Fine tune Compaction to take advantage of Acid 2.0 > -- > > Key: HIVE-16669 > URL: https://issues.apache.org/jira/browse/HIVE-16669 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-16669.wip.patch > > > * There is little point using 2.0 vectorized reader since there is no > operator pipeline in compaction > * If minor compaction just concats delete_delta files together, then the 2 > stage compaction should always ensure that we have a limited number of Orc > readers to do the merging and current OrcRawRecordMerger should be fine > * ... -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-16708) Exception while renewing a Delegation Token
[ https://issues.apache.org/jira/browse/HIVE-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar reassigned HIVE-16708: -- Assignee: Vihang Karajgaonkar > Exception while renewing a Delegation Token > --- > > Key: HIVE-16708 > URL: https://issues.apache.org/jira/browse/HIVE-16708 > Project: Hive > Issue Type: Bug >Reporter: Manikumar >Assignee: Vihang Karajgaonkar > > I am adding support to Storm Hive Bolt to access Hive meta-store using > delegation tokens. I am able to create and cancel delegation tokens using > HCatClient. I am getting below exception while renewing a delegation token. > Exception: > org.apache.hive.hcatalog.common.HCatException: > org.apache.hive.hcatalog.common.HCatException : 9001 : Exception occurred > while processing HCat request : MetaException while renewing delegation > token.. Cause : MetaException(message:hive/test.cluster@example.com tries > to renew a token with renewer hive) > at > org.apache.hive.hcatalog.api.HCatClientHMSImpl.renewDelegationToken(HCatClientHMSImpl.java:643) > ~[hive-webhcat-java-client-0.14.0.jar:0.14.0] > > Hadoop's AbstractDelegationTokenIdentifier sets the renewer to Kerberos short > name but HCatCleint.renewDelegationToken uses the fullName. This causes the > renewal to fail. > Relevant parts of code: > https://github.com/apache/hive/blob/master/shims/common/src/main/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java#L96 > https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenIdentifier.java#L105 > Let me know If I am missing something. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263345#comment-16263345
]
Hive QA commented on HIVE-17361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898805/HIVE-17361.21.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11418 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=12)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7973/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7973/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7973/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898805 - PreCommit-HIVE-Build
> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch,
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch,
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch,
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch,
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch,
> HIVE-17361.3.patch, HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap
> between ACID table and regular hive table.
> Current Documentation is under [DML
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
> and [Loading files into
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it
> uses the input file name which may not be in 0_0 which can break some
> read logic. (Certainly will for Acid).
> * It does not check the schema of the file. This may be a non issue for Acid
> which requires ORC which is self describing so Schema Evolution may handle
> this seamlessly. (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible.
> * Bucketed (and thus sorted) tables don't support Load Data (but only if
> hive.strict.checks.bucketing=true (default)). Will keep this restriction for
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created before files are moved, delta_8_8
> becomes visible before it's populated. Is that an issue?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263293#comment-16263293
]
Hive QA commented on HIVE-17361:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore: The patch generated 1 new + 28
unchanged - 0 fixed = 29 total (was 28) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
45s{color} | {color:red} ql: The patch generated 114 new + 1402 unchanged - 24
fixed = 1516 total (was 1426) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
53s{color} | {color:red} ql generated 1 new + 99 unchanged - 1 fixed = 100
total (was 100) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 2 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / a988c15 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7973/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7973/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7973/yetus/whitespace-eol.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7973/yetus/diff-javadoc-javadoc-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7973/yetus/patch-asflicense-problems.txt
|
| modules | C: common standalone-metastore ql itests itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7973/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch,
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch,
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch,
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch,
>
[jira] [Commented] (HIVE-18091) Failing tests of itests/qtest-spark and itests/hive-unit on branch-1
[
https://issues.apache.org/jira/browse/HIVE-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263269#comment-16263269
]
Daniel Voros commented on HIVE-18091:
-
It seems like it did the trick. There were ~160 failures before (see results on
HIVE-17947) and now there's only 62. All failures were there before the patch
but the following 10 failing test cases didn't get a chance to run before:
{code}
org.apache.hadoop.hive.ql.TestMTQueries#testMTQueries1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver#testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver#testCliDriver_vector_outer_join4
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver#testCliDriver_vector_outer_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver#testCliDriver_dynamic_rdd_cache
org.apache.hadoop.hive.cli.TestSparkCliDriver#testCliDriver_groupby1
org.apache.hadoop.hive.cli.TestSparkCliDriver#testCliDriver_stats9
org.apache.hadoop.hive.cli.TestSparkCliDriver#testCliDriver_stats_only_null
org.apache.hadoop.hive.cli.TestSparkCliDriver#testCliDriver_vectorization_13
org.apache.hadoop.hive.cli.TestSparkCliDriver#testCliDriver_vectorization_short_regress
{code}
I'll take a look at them to see if they were caused or only uncovered by this
fix.
> Failing tests of itests/qtest-spark and itests/hive-unit on branch-1
>
>
> Key: HIVE-18091
> URL: https://issues.apache.org/jira/browse/HIVE-18091
> Project: Hive
> Issue Type: Bug
> Components: Test, Testing Infrastructure
>Reporter: Daniel Voros
>Assignee: Daniel Voros
> Attachments: HIVE-18091.1-branch-1.patch, HIVE-18091.2-branch-1.patch
>
>
> Seen this while looking at ptest results for HIVE-17947 but is probably an
> older issue.
> Tests under itests/qtest-spark and itests/hive-unit fail when trying to
> execute the download-spark plugin with:
> {code}
> [INFO]
> [INFO]
>
> [INFO] Building Hive Integration - Unit Tests 1.3.0-SNAPSHOT
> [INFO]
>
> [INFO]
> [INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
> hive-it-unit ---
> [INFO]
> [INFO] --- maven-antrun-plugin:1.7:run (download-spark) @ hive-it-unit ---
> [INFO] Executing tasks
> main:
> [exec] + /bin/pwd
> [exec] + BASE_DIR=./target
> [exec] + HIVE_ROOT=./target/../../../
> [exec] + DOWNLOAD_DIR=./../thirdparty
> [exec] + mkdir -p ./../thirdparty
> [exec]
> /home/hiveptest/35.192.99.254-hiveptest-0/apache-github-branch-1-source/itests/hive-unit
> [exec] + download
> http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
> spark
> [exec] +
> url=http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
> [exec] + finalName=spark
> [exec] ++ basename
> http://d3jw87u4immizc.cloudfront.net/spark-tarball/spark-1.5.0-bin-hadoop2-without-hive.tgz
> [exec] + tarName=spark-1.5.0-bin-hadoop2-without-hive.tgz
> [exec] + rm -rf ./target/spark
> [exec] + [[ ! -f
> ./../thirdparty/spark-1.5.0-bin-hadoop2-without-hive.tgz ]]
> [exec] + tar -zxf
> ./../thirdparty/spark-1.5.0-bin-hadoop2-without-hive.tgz -C ./target
> [exec] + mv ./target/spark-1.5.0-bin-hadoop2-without-hive ./target/spark
> [exec] + cp -f ./target/../../..//data/conf/spark/log4j.properties
> ./target/spark/conf/
> [exec] + sed '/package /d'
> /data/hiveptest/working/apache-github-branch-1-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java
> [exec] sed: can't read
> /data/hiveptest/working/apache-github-branch-1-source/itests/../contrib/src/java/org/apache/hadoop/hive/contrib/udf/example/UDFExampleAdd.java:
> No such file or directory
> [exec] + javac -cp
> /data/hiveptest/working/maven/org/apache/hive/hive-exec/1.3.0-SNAPSHOT/hive-exec-1.3.0-SNAPSHOT.jar
> /tmp/UDFExampleAdd.java -d /tmp
> [exec] + jar -cf /tmp/udfexampleadd-1.0.jar -C /tmp UDFExampleAdd.class
> [exec] /tmp/UDFExampleAdd.class : no such file or directory
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 8.376s
> [INFO] Finished at: Mon Nov 06 22:29:39 UTC 2017
> [INFO] Final Memory: 18M/241M
> [INFO]
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:1.7:run (download-spark) on
> project hive-it-unit: An Ant BuildException has occured: exec returned: 1
> [ERROR] around Ant part
[jira] [Commented] (HIVE-16950) Dropping hive database/table which was created explicitly in default database location, deletes all databases data from default database location
[
https://issues.apache.org/jira/browse/HIVE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263253#comment-16263253
]
Hive QA commented on HIVE-16950:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898872/HIVE-16950.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 82 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_change_db_location]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabaseLocation
(batchId=212)
org.apache.hadoop.hive.metastore.TestFilterHooks.testDefaultFilter (batchId=211)
org.apache.hadoop.hive.metastore.TestFilterHooks.testDummyFilterForPartition
(batchId=211)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testDatabaseLocation
(batchId=214)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabaseLocation
(batchId=210)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testDatabaseLocation
(batchId=209)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testDatabaseLocation
(batchId=219)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testAlters
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBasic (batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBasicWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentRename
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCMConflict
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenateTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDrops (batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDropsWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDumpLimit
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testExchangePartition
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIdempotentMoveTaskForInsertFiles
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalAdds
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertDropUnpartitionedTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInserts
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoad
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadWithVariableLengthEventId
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalRepeatEventOnExistingObject
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalRepeatEventOnMissingObject
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testInsertOverwriteOnUnpartitionedTableWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testInsertToMultiKeyPartition
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRemoveStats
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenamePartitionedTableAcrossDatabases
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenameTableAcrossDatabases
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenameTableWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTruncatePartitionedTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTruncateTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTruncateWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testViewsReplication
(batchId=224)
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation (batchId=185)
[jira] [Updated] (HIVE-18127) Do not strip '--' comments from shell commands issued from CliDriver
[ https://issues.apache.org/jira/browse/HIVE-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18127: -- Attachment: HIVE-18127.2.patch > Do not strip '--' comments from shell commands issued from CliDriver > > > Key: HIVE-18127 > URL: https://issues.apache.org/jira/browse/HIVE-18127 > Project: Hive > Issue Type: Bug >Reporter: Andrew Sherman >Assignee: Andrew Sherman > Attachments: HIVE-18127.1.patch, HIVE-18127.2.patch > > > CLiDriver has the ability to run shell commands by prefixing them with '!". > This behavior is not widely used (there are only 3 examples in .q files). > Since HIVE-16935 started stripping comments starting with '\-\-', a shell > command containing '--' will not work correctly. > Fix this by using the unstripped command for shell commands. > Note that it would be a security hole for HS2 to allow execution of arbitrary > shell commands from a client command. > Add tests to nail down correct behavior with '--' comments: > * CliDriver should not strip strings starting with '--' in a shell command > (FIXED in this change). > * HiveCli should strip '--' comments. > * A Jdbc program should allow commands starting with "!" but these will fail > in the sql parser. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18068) Replace LocalInterval by Interval in Druid storage handler
[ https://issues.apache.org/jira/browse/HIVE-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18068: Attachment: HIVE-18068.2.patch > Replace LocalInterval by Interval in Druid storage handler > -- > > Key: HIVE-18068 > URL: https://issues.apache.org/jira/browse/HIVE-18068 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-18068.2.patch, HIVE-18068.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18068) Replace LocalInterval by Interval in Druid storage handler
[ https://issues.apache.org/jira/browse/HIVE-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18068: Status: Patch Available (was: Open) > Replace LocalInterval by Interval in Druid storage handler > -- > > Key: HIVE-18068 > URL: https://issues.apache.org/jira/browse/HIVE-18068 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-18068.2.patch, HIVE-18068.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18068) Replace LocalInterval by Interval in Druid storage handler
[ https://issues.apache.org/jira/browse/HIVE-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18068: Status: Open (was: Patch Available) > Replace LocalInterval by Interval in Druid storage handler > -- > > Key: HIVE-18068 > URL: https://issues.apache.org/jira/browse/HIVE-18068 > Project: Hive > Issue Type: Bug > Components: Druid integration >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Attachments: HIVE-18068.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18115) Fix schema version info for Hive-2.3.2
[
https://issues.apache.org/jira/browse/HIVE-18115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263203#comment-16263203
]
Vihang Karajgaonkar commented on HIVE-18115:
I looked into the schema tool tests in the hope of finding that a test that is
broken but I found surprisingly, that this error should not happen in the first
place in 2.3.2 (and 2.3.1 as well).
{{MetaStoreSchemaInfo.getHiveSchemaVersion()}} which is used to find the
version of Hive from the binaries should return 2.3.0 and it should match with
the schema version in the database which is also 2.3.0 from the init files.
{{MetaStoreSchemaInfo.getHiveSchemaVersion()}} gets the shortVersion from the
{{HiveVersionAnnotation}} which is generated in the build time using
{{saveVersion.sh}} and it injects the hive version from pom.xml's property
which is {{2.3.0}} for the
2.3.2 release.
[~osayankin] did you build the source code from the release-2.3.2 or you just
downloaded the binaries? If its the former can you paste the contents of
package-info.java from the hive-common project?
> Fix schema version info for Hive-2.3.2
> --
>
> Key: HIVE-18115
> URL: https://issues.apache.org/jira/browse/HIVE-18115
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.1, 2.3.2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Fix For: 2.3.3
>
> Attachments: HIVE-18115.02-branch-2.patch, HIVE-18115.1.patch
>
>
> Error while starting HiveMeta
> {code}
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema
> version 2.3.2 does not match metastore's schema version 2.3.0 Metastore is
> not upgraded or corrupt
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7600)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7563)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at com.sun.proxy.$Proxy23.verifySchema(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
> ~[hive-exec-2.3.2.jar:2.3.2]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16950) Dropping hive database/table which was created explicitly in default database location, deletes all databases data from default database location
[
https://issues.apache.org/jira/browse/HIVE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263184#comment-16263184
]
Hive QA commented on HIVE-16950:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore: The patch generated 1 new + 684
unchanged - 0 fixed = 685 total (was 684) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b5ba827 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7972/yetus/diff-checkstyle-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7972/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7972/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dropping hive database/table which was created explicitly in default database
> location, deletes all databases data from default database location
> -
>
> Key: HIVE-16950
> URL: https://issues.apache.org/jira/browse/HIVE-16950
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Rahul Kalgunde
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Attachments: HIVE-16950.1.patch, HIVE-16950.2.patch
>
>
> When database/table is created explicitly pointing to the default location,
> dropping the database/table deletes all the data associated with the all
> databases/tables.
> Steps to replicate:
> in below e.g. dropping table test_db2 also deletes data of test_db1 where as
> metastore still contains test_db1
> hive> create database test_db1;
> OK
> Time taken: 4.858 seconds
> hive> describe database test_db1;
> OK
> test_db1
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_db1.db root
> USER
> Time taken: 0.599 seconds, Fetched: 1 row(s)
> hive> create database test_db2 location '/apps/hive/warehouse' ;
> OK
> Time taken: 1.457 seconds
> hive> describe database test_db2;
> OK
> test_db2
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse rootUSER
> Time taken: 0.582 seconds, Fetched: 1 row(s)
> hive> drop database test_db2;
>
[jira] [Updated] (HIVE-18123) Explain formatted improve column expression map display
[ https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18123: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~ashutoshc] > Explain formatted improve column expression map display > --- > > Key: HIVE-18123 > URL: https://issues.apache.org/jira/browse/HIVE-18123 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-18123.1.patch, HIVE-18123.2.patch > > > HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that > map was a little off. This jira is to improve the formatting. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18109) fix identifier usage in parser
[ https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18109: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Sergey! > fix identifier usage in parser > -- > > Key: HIVE-18109 > URL: https://issues.apache.org/jira/browse/HIVE-18109 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-18109.patch > > > HIVE-17902 -broke- exposed this ;) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18123) Explain formatted improve column expression map display
[ https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263164#comment-16263164 ] Ashutosh Chauhan commented on HIVE-18123: - +1 > Explain formatted improve column expression map display > --- > > Key: HIVE-18123 > URL: https://issues.apache.org/jira/browse/HIVE-18123 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-18123.1.patch, HIVE-18123.2.patch > > > HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that > map was a little off. This jira is to improve the formatting. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16950) Dropping hive database/table which was created explicitly in default database location, deletes all databases data from default database location
[
https://issues.apache.org/jira/browse/HIVE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263153#comment-16263153
]
Hive QA commented on HIVE-16950:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898872/HIVE-16950.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 82 failed/errored test(s), 11410 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[alter_change_db_location]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testDatabaseLocation
(batchId=212)
org.apache.hadoop.hive.metastore.TestFilterHooks.testDefaultFilter (batchId=211)
org.apache.hadoop.hive.metastore.TestFilterHooks.testDummyFilterForPartition
(batchId=211)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testDatabaseLocation
(batchId=214)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testDatabaseLocation
(batchId=210)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testDatabaseLocation
(batchId=209)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testDatabaseLocation
(batchId=219)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testAlters
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBasic (batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBasicWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentDropTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testBootstrapWithConcurrentRename
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testCMConflict
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenatePartitionedTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConcatenateTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDrops (batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDropsWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testDumpLimit
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testExchangePartition
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIdempotentMoveTaskForInsertFiles
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalAdds
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInsertDropUnpartitionedTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalInserts
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoad
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalLoadWithVariableLengthEventId
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalRepeatEventOnExistingObject
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testIncrementalRepeatEventOnMissingObject
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testInsertOverwriteOnUnpartitionedTableWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testInsertToMultiKeyPartition
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRemoveStats
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenamePartitionedTableAcrossDatabases
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenameTableAcrossDatabases
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testRenameTableWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTruncatePartitionedTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTruncateTable
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testTruncateWithCM
(batchId=224)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testViewsReplication
(batchId=224)
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation (batchId=185)
[jira] [Commented] (HIVE-18109) fix identifier usage in parser
[ https://issues.apache.org/jira/browse/HIVE-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263150#comment-16263150 ] Ashutosh Chauhan commented on HIVE-18109: - +1 I assume TestTriggersWorkloadManager failures are unrelated. > fix identifier usage in parser > -- > > Key: HIVE-18109 > URL: https://issues.apache.org/jira/browse/HIVE-18109 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Ashutosh Chauhan >Assignee: Sergey Shelukhin > Attachments: HIVE-18109.patch > > > HIVE-17902 -broke- exposed this ;) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16950) Dropping hive database/table which was created explicitly in default database location, deletes all databases data from default database location
[
https://issues.apache.org/jira/browse/HIVE-16950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263056#comment-16263056
]
Hive QA commented on HIVE-16950:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore: The patch generated 1 new + 684
unchanged - 0 fixed = 685 total (was 684) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
13s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b5ba827 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7971/yetus/diff-checkstyle-standalone-metastore.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7971/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7971/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dropping hive database/table which was created explicitly in default database
> location, deletes all databases data from default database location
> -
>
> Key: HIVE-16950
> URL: https://issues.apache.org/jira/browse/HIVE-16950
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 1.2.1
>Reporter: Rahul Kalgunde
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Attachments: HIVE-16950.1.patch, HIVE-16950.2.patch
>
>
> When database/table is created explicitly pointing to the default location,
> dropping the database/table deletes all the data associated with the all
> databases/tables.
> Steps to replicate:
> in below e.g. dropping table test_db2 also deletes data of test_db1 where as
> metastore still contains test_db1
> hive> create database test_db1;
> OK
> Time taken: 4.858 seconds
> hive> describe database test_db1;
> OK
> test_db1
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse/test_db1.db root
> USER
> Time taken: 0.599 seconds, Fetched: 1 row(s)
> hive> create database test_db2 location '/apps/hive/warehouse' ;
> OK
> Time taken: 1.457 seconds
> hive> describe database test_db2;
> OK
> test_db2
> hdfs://sandbox.hortonworks.com:8020/apps/hive/warehouse rootUSER
> Time taken: 0.582 seconds, Fetched: 1 row(s)
> hive> drop database test_db2;
>
[jira] [Commented] (HIVE-18115) Fix schema version info for Hive-2.3.2
[
https://issues.apache.org/jira/browse/HIVE-18115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263023#comment-16263023
]
Vihang Karajgaonkar commented on HIVE-18115:
Yes, based on the current implementation it looks like it needs to be done for
every release. We should definitely update the wiki and/or add a automated test
to make sure this is not broken. The problem with adding a test to cover this
scenario is that by default our unit tests have
{{hive.metastore.schema.verification}} turned off because of which this was not
caught. I wonder why the schematool tests did not catch this though.
> Fix schema version info for Hive-2.3.2
> --
>
> Key: HIVE-18115
> URL: https://issues.apache.org/jira/browse/HIVE-18115
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.1, 2.3.2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Fix For: 2.3.3
>
> Attachments: HIVE-18115.02-branch-2.patch, HIVE-18115.1.patch
>
>
> Error while starting HiveMeta
> {code}
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema
> version 2.3.2 does not match metastore's schema version 2.3.0 Metastore is
> not upgraded or corrupt
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7600)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7563)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at com.sun.proxy.$Proxy23.verifySchema(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
> ~[hive-exec-2.3.2.jar:2.3.2]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17972) Implement Parquet vectorization reader for Map type
[
https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16263018#comment-16263018
]
Hive QA commented on HIVE-17972:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898810/HIVE-17972.001.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11414 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7970/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7970/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7970/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898810 - PreCommit-HIVE-Build
> Implement Parquet vectorization reader for Map type
> ---
>
> Key: HIVE-17972
> URL: https://issues.apache.org/jira/browse/HIVE-17972
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-17972.001.patch
>
>
> Parquet vectorized reader can't support map type, it should be supported to
> improve the performance when the query with map type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18115) Fix schema version info for Hive-2.3.2
[
https://issues.apache.org/jira/browse/HIVE-18115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262988#comment-16262988
]
Sahil Takiar commented on HIVE-18115:
-
[~vihangk1] is this something that will need to be done for every release? If
thats the case maybe we can make this a bit easier. By default we can assume
all 2.3.x versions are equivalent to 2.3.0 unless otherwise specified? Or more
generally, all x.y.z versions are equivalent to x.y.0 versions.
Maybe we need to add a snippet in the How to Release section of the Apache Hive
Wiki?
> Fix schema version info for Hive-2.3.2
> --
>
> Key: HIVE-18115
> URL: https://issues.apache.org/jira/browse/HIVE-18115
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.1, 2.3.2
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Minor
> Fix For: 2.3.3
>
> Attachments: HIVE-18115.02-branch-2.patch, HIVE-18115.1.patch
>
>
> Error while starting HiveMeta
> {code}
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Hive Schema
> version 2.3.2 does not match metastore's schema version 2.3.0 Metastore is
> not upgraded or corrupt
> at
> org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7600)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7563)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:101)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at com.sun.proxy.$Proxy23.verifySchema(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:591)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:584)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:651)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:427)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_141]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_141]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_141]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_141]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> ~[hive-exec-2.3.2.jar:2.3.2]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
> ~[hive-exec-2.3.2.jar:2.3.2]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17972) Implement Parquet vectorization reader for Map type
[
https://issues.apache.org/jira/browse/HIVE-17972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262934#comment-16262934
]
Hive QA commented on HIVE-17972:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
32s{color} | {color:red} ql: The patch generated 4 new + 42 unchanged - 0 fixed
= 46 total (was 42) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / b5ba827 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7970/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7970/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7970/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement Parquet vectorization reader for Map type
> ---
>
> Key: HIVE-17972
> URL: https://issues.apache.org/jira/browse/HIVE-17972
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-17972.001.patch
>
>
> Parquet vectorized reader can't support map type, it should be supported to
> improve the performance when the query with map type.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262910#comment-16262910
]
Hive QA commented on HIVE-17856:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898808/HIVE-17856.14.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 11420 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=77)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dp_counter_mm]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[0]
(batchId=188)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[5]
(batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7969/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7969/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7969/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898808 - PreCommit-HIVE-Build
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.2.patch, HIVE-17856.3.patch,
> HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch,
> HIVE-17856.7.patch, HIVE-17856.8.patch, HIVE-17856.9.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from
[jira] [Updated] (HIVE-18131) Truncate table for Acid tables
[ https://issues.apache.org/jira/browse/HIVE-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18131: -- Description: How should this work? Should it work like Insert Overwrite T select * from T where 1=2? This should create a new empty base_x/ and thus operate w/o violating Snapshot Isolation semantics. This makes sense for specific partition or unpartitioned table. What about "Truncate T" where T is partitioned? Is the expectation to wipe out all partition info or to make each partition empty? was: How should this work? Should it work like Insert Overwrite T select * from T where 1=2? This should create a new empty base_x/ and thus operate w/o violating Snapshot Isolation semantics. > Truncate table for Acid tables > -- > > Key: HIVE-18131 > URL: https://issues.apache.org/jira/browse/HIVE-18131 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > How should this work? Should it work like Insert Overwrite T select * from T > where 1=2? > This should create a new empty base_x/ and thus operate w/o violating > Snapshot Isolation semantics. > This makes sense for specific partition or unpartitioned table. What about > "Truncate T" where T is partitioned? Is the expectation to wipe out all > partition info or to make each partition empty? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18131) Truncate table for Acid tables
[ https://issues.apache.org/jira/browse/HIVE-18131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18131: - > Truncate table for Acid tables > -- > > Key: HIVE-18131 > URL: https://issues.apache.org/jira/browse/HIVE-18131 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > How should this work? Should it work like Insert Overwrite T select * from T > where 1=2? > This should create a new empty base_x/ and thus operate w/o violating > Snapshot Isolation semantics. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18112) show create for view having special char in where clause is not showing properly
[ https://issues.apache.org/jira/browse/HIVE-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262886#comment-16262886 ] Ashutosh Chauhan commented on HIVE-18112: - I see. Then you need to name the patch as per https://cwiki.apache.org/confluence/display/Hive/Hive+PreCommit+Patch+Testing so that tests may run on it. > show create for view having special char in where clause is not showing > properly > > > Key: HIVE-18112 > URL: https://issues.apache.org/jira/browse/HIVE-18112 > Project: Hive > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Minor > Fix For: 2.1.0 > > Attachments: HIVE-18112.patch > > > e.g., > CREATE VIEW `v2` AS select `evil_byte1`.`a` from `default`.`EVIL_BYTE1` where > `evil_byte1`.`a` = 'abcÖdefÖgh'; > Output: > == > 0: jdbc:hive2://172.26.122.227:1> show create table v2; > ++--+ > | createtab_stmt >| > ++--+ > | CREATE VIEW `v2` AS select `evil_byte1`.`a` from `default`.`EVIL_BYTE1` > where `evil_byte1`.`a` = 'abc�def�gh' | > ++--+ > Only show create output is having invalid characters, actual source table > content is displayed properly in the console. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17361) Support LOAD DATA for transactional tables
[ https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17361: -- Attachment: (was: HIVE-17361.22.patch) > Support LOAD DATA for transactional tables > -- > > Key: HIVE-17361 > URL: https://issues.apache.org/jira/browse/HIVE-17361 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Wei Zheng >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch, > HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch, > HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch, > HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch, > HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch, > HIVE-17361.3.patch, HIVE-17361.4.patch > > > LOAD DATA was not supported since ACID was introduced. Need to fill this gap > between ACID table and regular hive table. > Current Documentation is under [DML > Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations] > and [Loading files into > tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]: > \\ > * Load Data performs very limited validations of the data, in particular it > uses the input file name which may not be in 0_0 which can break some > read logic. (Certainly will for Acid). > * It does not check the schema of the file. This may be a non issue for Acid > which requires ORC which is self describing so Schema Evolution may handle > this seamlessly. (Assuming Schema is not too different). > * It does check that _InputFormat_S are compatible. > * Bucketed (and thus sorted) tables don't support Load Data (but only if > hive.strict.checks.bucketing=true (default)). Will keep this restriction for > Acid. > * Load Data supports OVERWRITE clause > * What happens to file permissions/ownership: rename vs copy differences > \\ > The implementation will follow the same idea as in HIVE-14988 and use a > base_N/ dir for OVERWRITE clause. > \\ > How is minor compaction going to handle delta/base with original files? > Since delta_8_8/_meta_data is created before files are moved, delta_8_8 > becomes visible before it's populated. Is that an issue? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18100) Some tests time out
[ https://issues.apache.org/jira/browse/HIVE-18100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18100: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. > Some tests time out > --- > > Key: HIVE-18100 > URL: https://issues.apache.org/jira/browse/HIVE-18100 > Project: Hive > Issue Type: Test > Components: Test >Reporter: Ashutosh Chauhan >Assignee: Ashutosh Chauhan > Fix For: 3.0.0 > > Attachments: HIVE-18100.1.patch, HIVE-18100.2.patch, > HIVE-18100.3.patch, HIVE-18100.patch > > > Some tests had 100s of queries in a single query which times out resulting in > Hive QA failures. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262854#comment-16262854
]
Hive QA commented on HIVE-17856:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
1s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
9s{color} | {color:red} common: The patch generated 2 new + 8 unchanged - 1
fixed = 10 total (was 9) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 41 new + 1755 unchanged - 7
fixed = 1796 total (was 1762) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 9 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch 6 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / c7aa7b6 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/whitespace-eol.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/whitespace-tabs.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-7969/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch, HIVE-17856.10.patch,
> HIVE-17856.11.patch, HIVE-17856.12.patch, HIVE-17856.13.patch,
> HIVE-17856.14.patch, HIVE-17856.2.patch, HIVE-17856.3.patch,
> HIVE-17856.4.patch, HIVE-17856.5.patch, HIVE-17856.6.patch,
>
[jira] [Commented] (HIVE-18123) Explain formatted improve column expression map display
[ https://issues.apache.org/jira/browse/HIVE-18123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262823#comment-16262823 ] Vineet Garg commented on HIVE-18123: [~ashutoshc] Can you take a look? > Explain formatted improve column expression map display > --- > > Key: HIVE-18123 > URL: https://issues.apache.org/jira/browse/HIVE-18123 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-18123.1.patch, HIVE-18123.2.patch > > > HIVE-17898 introduced columnExprMap in explain formatted. Formatting of that > map was a little off. This jira is to improve the formatting. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17361) Support LOAD DATA for transactional tables
[
https://issues.apache.org/jira/browse/HIVE-17361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16262816#comment-16262816
]
Hive QA commented on HIVE-17361:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12898805/HIVE-17361.21.patch
{color:green}SUCCESS:{color} +1 due to 12 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 11418 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dbtxnmgr_showlocks]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort]
(batchId=81)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[vector_inner_join]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=224)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testTupleInBagInTupleInBag[2]
(batchId=188)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteChar (batchId=188)
org.apache.hive.hcatalog.pig.TestTextFileHCatStorer.testWriteSmallint
(batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=230)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=230)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=230)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7968/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7968/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7968/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12898805 - PreCommit-HIVE-Build
> Support LOAD DATA for transactional tables
> --
>
> Key: HIVE-17361
> URL: https://issues.apache.org/jira/browse/HIVE-17361
> Project: Hive
> Issue Type: New Feature
> Components: Transactions
>Reporter: Wei Zheng
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-17361.07.patch, HIVE-17361.08.patch,
> HIVE-17361.09.patch, HIVE-17361.1.patch, HIVE-17361.10.patch,
> HIVE-17361.11.patch, HIVE-17361.12.patch, HIVE-17361.14.patch,
> HIVE-17361.16.patch, HIVE-17361.17.patch, HIVE-17361.19.patch,
> HIVE-17361.2.patch, HIVE-17361.20.patch, HIVE-17361.21.patch,
> HIVE-17361.22.patch, HIVE-17361.3.patch, HIVE-17361.4.patch
>
>
> LOAD DATA was not supported since ACID was introduced. Need to fill this gap
> between ACID table and regular hive table.
> Current Documentation is under [DML
> Operations|https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-DMLOperations]
> and [Loading files into
> tables|https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Loadingfilesintotables]:
> \\
> * Load Data performs very limited validations of the data, in particular it
> uses the input file name which may not be in 0_0 which can break some
> read logic. (Certainly will for Acid).
> * It does not check the schema of the file. This may be a non issue for Acid
> which requires ORC which is self describing so Schema Evolution may handle
> this seamlessly. (Assuming Schema is not too different).
> * It does check that _InputFormat_S are compatible.
> * Bucketed (and thus sorted) tables don't support Load Data (but only if
> hive.strict.checks.bucketing=true (default)). Will keep this restriction for
> Acid.
> * Load Data supports OVERWRITE clause
> * What happens to file permissions/ownership: rename vs copy differences
> \\
> The implementation will follow the same idea as in HIVE-14988 and use a
> base_N/ dir for OVERWRITE clause.
> \\
> How is minor compaction going to handle delta/base with original files?
> Since delta_8_8/_meta_data is created
