[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518829#comment-14518829
]
Sushanth Sowmyan commented on HIVE-9674:
Will go ahead and commit this to master and to 1.2.
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch, HIVE-9674.3.patch, HIVE-9674.4.patch,
> HIVE-9674.5.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518827#comment-14518827
]
Sushanth Sowmyan commented on HIVE-9674:
All the failures listed here have nothing to do with the patch itself. I
wondered about the
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
failure, since it's in HCat's package. However, that is not related to this
patch, and on retesting locally, it seems to pass. Also, that's likely to be
related to ACID rather than this, since its error log looks like the following:
{noformat}
java.sql.SQLException: Table/View 'TXNS' already exists in Schema 'APP'.
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at org.apache.derby.iapi.error.StandardException.newException(Unknown
Source)
at
org.apache.derby.impl.sql.catalog.DataDictionaryImpl.duplicateDescriptorException(Unknown
Source)
at
org.apache.derby.impl.sql.catalog.DataDictionaryImpl.addDescriptor(Unknown
Source)
at
org.apache.derby.impl.sql.execute.CreateTableConstantAction.executeConstantAction(Unknown
Source)
at org.apache.derby.impl.sql.execute.MiscResultSet.open(Unknown Source)
at
org.apache.derby.impl.sql.GenericPreparedStatement.executeStmt(Unknown Source)
at org.apache.derby.impl.sql.GenericPreparedStatement.execute(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.executeStatement(Unknown
Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at org.apache.derby.impl.jdbc.EmbedStatement.execute(Unknown Source)
at
org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:72)
at
org.apache.hadoop.hive.metastore.txn.TxnDbUtil.prepDb(TxnDbUtil.java:131)
at
org.apache.hive.hcatalog.streaming.TestStreaming.(TestStreaming.java:157)
{noformat}
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch, HIVE-9674.3.patch, HIVE-9674.4.patch,
> HIVE-9674.5.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518624#comment-14518624
]
Hive QA commented on HIVE-9674:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12728832/HIVE-9674.5.patch
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 8825 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3635/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3635/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3635/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12728832 - PreCommit-HIVE-TRUNK-Build
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch, HIVE-9674.3.patch, HIVE-9674.4.patch,
> HIVE-9674.5.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516425#comment-14516425
]
Sushanth Sowmyan commented on HIVE-9674:
Since this jira seems to not be picked up by precommit tests automatically, I
manually queued one over at
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/
, build#3626
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch, HIVE-9674.3.patch, HIVE-9674.4.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14508407#comment-14508407
]
Hive QA commented on HIVE-9674:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12726535/HIVE-9674.4.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3535/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3535/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3535/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-3535/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/test/results/clientpositive/windowing_navfn.q.out'
Reverted 'ql/src/test/queries/clientpositive/windowing_navfn.q'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java'
++ awk '{print $2}'
++ egrep -v '^X|^Performing status on external'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target itests/target itests/thirdparty
itests/hcatalog-unit/target itests/test-serde/target itests/qtest/target
itests/hive-unit-hadoop2/target itests/hive-minikdc/target
itests/hive-jmh/target itests/hive-unit/target itests/custom-serde/target
itests/util/target itests/qtest-spark/target hcatalog/target
hcatalog/core/target hcatalog/streaming/target
hcatalog/server-extensions/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target hcatalog/hcatalog-pig-adapter/target
accumulo-handler/target hwi/target common/target common/src/gen
spark-client/target contrib/target service/target serde/target beeline/target
odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1675534.
At revision 1675534.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12726535 - PreCommit-HIVE-TRUNK-Build
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch, HIVE-9674.3.patch, HIVE-9674.4.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486152#comment-14486152
]
Sushanth Sowmyan commented on HIVE-9674:
HIVE-9609 should be good to go in another few hours.
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14486145#comment-14486145
]
Mithun Radhakrishnan commented on HIVE-9674:
Actually, [~sushanth], let's hold off for right now, on this one. I'll rebase
this under the assumption that HIVE-9609 is good to go.
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14380320#comment-14380320
]
Mithun Radhakrishnan commented on HIVE-9674:
Sush, could you please review this one? I'd like to avoid another rebase.
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch, HIVE-9736.3.patch, HIVE-9736.4.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9674) *DropPartitionEvent should handle partition-sets.
[
https://issues.apache.org/jira/browse/HIVE-9674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348331#comment-14348331
]
Hive QA commented on HIVE-9674:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12702724/HIVE-9736.4.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 7594 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables_compact
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2953/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2953/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2953/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12702724 - PreCommit-HIVE-TRUNK-Build
> *DropPartitionEvent should handle partition-sets.
> -
>
> Key: HIVE-9674
> URL: https://issues.apache.org/jira/browse/HIVE-9674
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 0.14.0
>Reporter: Mithun Radhakrishnan
>Assignee: Mithun Radhakrishnan
> Attachments: HIVE-9674.2.patch, HIVE-9736.3.patch, HIVE-9736.4.patch
>
>
> Dropping a set of N partitions from a table currently results in N
> DropPartitionEvents (and N PreDropPartitionEvents) being fired serially. This
> is wasteful, especially so for large N. It also makes it impossible to even
> try to run authorization-checks on all partitions in a batch.
> Taking the cue from HIVE-9609, we should compose an {{Iterable}}
> in the event, and expose them via an {{Iterator}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
