[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Attachment: HIVE-18988.04.patch > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19171) Persist runtime statistics in metastore
[ https://issues.apache.org/jira/browse/HIVE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19171: Attachment: HIVE-19171.01-branch-3.patch > Persist runtime statistics in metastore > --- > > Key: HIVE-19171 > URL: https://issues.apache.org/jira/browse/HIVE-19171 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19171.01-branch-3.patch, HIVE-19171.01.patch, > HIVE-19171.01wip01.patch, HIVE-19171.01wip02.patch, HIVE-19171.01wip03.patch, > HIVE-19171.02.patch, HIVE-19171.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19171) Persist runtime statistics in metastore
[ https://issues.apache.org/jira/browse/HIVE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19171: Status: Patch Available (was: Reopened) > Persist runtime statistics in metastore > --- > > Key: HIVE-19171 > URL: https://issues.apache.org/jira/browse/HIVE-19171 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19171.01-branch-3.patch, HIVE-19171.01.patch, > HIVE-19171.01wip01.patch, HIVE-19171.01wip02.patch, HIVE-19171.01wip03.patch, > HIVE-19171.02.patch, HIVE-19171.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-19171) Persist runtime statistics in metastore
[ https://issues.apache.org/jira/browse/HIVE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reopened HIVE-19171: - > Persist runtime statistics in metastore > --- > > Key: HIVE-19171 > URL: https://issues.apache.org/jira/browse/HIVE-19171 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19171.01.patch, HIVE-19171.01wip01.patch, > HIVE-19171.01wip02.patch, HIVE-19171.01wip03.patch, HIVE-19171.02.patch, > HIVE-19171.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19171) Persist runtime statistics in metastore
[ https://issues.apache.org/jira/browse/HIVE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19171: Attachment: (was: HIVE-19171.01-branch-3.patch) > Persist runtime statistics in metastore > --- > > Key: HIVE-19171 > URL: https://issues.apache.org/jira/browse/HIVE-19171 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19171.01.patch, HIVE-19171.01wip01.patch, > HIVE-19171.01wip02.patch, HIVE-19171.01wip03.patch, HIVE-19171.02.patch, > HIVE-19171.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19171) Persist runtime statistics in metastore
[ https://issues.apache.org/jira/browse/HIVE-19171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-19171: Attachment: HIVE-19171.01-branch-3.patch > Persist runtime statistics in metastore > --- > > Key: HIVE-19171 > URL: https://issues.apache.org/jira/browse/HIVE-19171 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-19171.01.patch, HIVE-19171.01wip01.patch, > HIVE-19171.01wip02.patch, HIVE-19171.01wip03.patch, HIVE-19171.02.patch, > HIVE-19171.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450393#comment-16450393 ] Sergey Shelukhin commented on HIVE-19124: - Frankly having to do this points at the need to get rid of all these classes and just have a single one; in a follow up patch. I'll take a look into the root canal thru the ear variant today, for now. > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Attachment: (was: HIVE-18988.04.patch) > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450388#comment-16450388
]
Gopal V commented on HIVE-19124:
You need to modify the TxnUtils to accept a ValidReaderWriteIdList and rewrite
the current impl as
{code}
return
createValidCompactWriteIdList(createValidReaderWriteIdList(tableWriteIds));
{code}
And then you don't need a new metastore object.
ValidReaderWriteIdList & ValidCompactorWriteIdList extends
ValidReaderWriteIdList
You need a ValidReaderWriteIdList.get to pass into compactionWriteIds.
> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Labels: mm-gap-2
> Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch,
> HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch,
> HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch,
> HIVE-19124.patch
>
>
> For now, it will run a query directly and only major compactions will be
> supported.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450387#comment-16450387 ] Sergey Shelukhin edited comment on HIVE-19124 at 4/24/18 6:56 PM: -- There's tons of code in AcidUtils that does similar things based on different inputs (e.g. overloads if isAcidTable)... why is this not acceptable? Do you want to modify the latest patch? I don't see a way to use this method without either copy-pasting the old method for different input structure, or making it so we do a root canal thru the ear... was (Author: sershe): There's tons of code in AcidUtils that does similar things based on different inputs (e.g. overloads if isAcidTable)... why is this not acceptable? Do you want to modify a patch? I don't see a way to use this method without either copy-pasting the old method for different input structure, or making it so we do a root canal thru the ear... > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450387#comment-16450387 ] Sergey Shelukhin commented on HIVE-19124: - There's tons of code in AcidUtils that does similar things based on different inputs (e.g. overloads if isAcidTable)... why is this not acceptable? Do you want to modify a patch? I don't see a way to use this method without either copy-pasting the old method for different input structure, or making it so we do a root canal thru the ear... > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450383#comment-16450383 ] Eugene Koifman commented on HIVE-19124: --- I saw the comment. That isn't acceptable. > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450379#comment-16450379 ] Gopal V commented on HIVE-19124: Yes, this is true even today, but it will move away with standalone-metastore. I just realized there's one dropped issue in the code from [~ekoifman] to address. TxnUtils.updateForCompactionQuery is yet another codepath, which we shouldn't need - the TxnUtils.createValidCompactWriteIdList goes further ahead than that. Basically if low-water mark is x and x+1, x+2 etc are aborted, your impl only compactors till x. > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450380#comment-16450380 ] Sergey Shelukhin commented on HIVE-19124: - See the comment on RB; we don't have the same input structures needed for this method. There are too many different classes that do not convert into each other... it's better to have an utility method that adjust one property than to convert existing structure back to thrift (or alternatively call metastore again) and then rebuild the new one from scratch from thrift. > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19215) JavaUtils.AnyIdDirFilter ignores base_n directories
[ https://issues.apache.org/jira/browse/HIVE-19215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450375#comment-16450375 ] Prasanth Jayachandran commented on HIVE-19215: -- lgtm, +1 > JavaUtils.AnyIdDirFilter ignores base_n directories > --- > > Key: HIVE-19215 > URL: https://issues.apache.org/jira/browse/HIVE-19215 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19215.01.patch, HIVE-19215.02.patch, > HIVE-19215.03.patch, HIVE-19215.04.patch, HIVE-19215.patch > > > cc [~sershe], [~steveyeom2017] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Status: Open (was: Patch Available) > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450373#comment-16450373 ] Eugene Koifman commented on HIVE-19124: --- Hold on. We already have a mechanism for compactor to make sure it doesn't compact above an open txn: TxnUtils.createValidCompactWriteIdList() . This reinvents the wheel and add a new one to do the same thing in the Reader list. Why is this good? > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19215) JavaUtils.AnyIdDirFilter ignores base_n directories
[ https://issues.apache.org/jira/browse/HIVE-19215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450368#comment-16450368 ] Sergey Shelukhin commented on HIVE-19215: - The only test that looks relevant actually failed due to ConcurrentModificationException in Hadoop config. [~prasanth_j] can you take a look again? thnx > JavaUtils.AnyIdDirFilter ignores base_n directories > --- > > Key: HIVE-19215 > URL: https://issues.apache.org/jira/browse/HIVE-19215 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19215.01.patch, HIVE-19215.02.patch, > HIVE-19215.03.patch, HIVE-19215.04.patch, HIVE-19215.patch > > > cc [~sershe], [~steveyeom2017] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Attachment: HIVE-19285.2.patch > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > Attachments: HIVE-19285.1.patch, HIVE-19285.2.patch > > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450365#comment-16450365 ] Sergey Shelukhin commented on HIVE-19124: - Well, it already needs to have access to the cluster to use the MR compactor job. When metastore is separate from Hive, I think compactor should move to HS2 cc [~alangates] > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19137) orcfiledump doesn't print hive.acid.version value
[
https://issues.apache.org/jira/browse/HIVE-19137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19137:
--
Resolution: Fixed
Fix Version/s: 3.0.0
Release Note: n/a
Status: Resolved (was: Patch Available)
committed to branch-3/master
thanks Igor for the contribution
> orcfiledump doesn't print hive.acid.version value
> -
>
> Key: HIVE-19137
> URL: https://issues.apache.org/jira/browse/HIVE-19137
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19137-branch-3.01.patch, HIVE-19137.01.patch,
> HIVE-19137.02-branch-3.patch, HIVE-19137.02.patch,
> HIVE-19137.03-branch-3.patch, HIVE-19137.03.patch, HIVE-19137.04.patch,
> HIVE-19137.05.patch
>
>
> HIVE-18659 added hive.acid.version in the file footer.
> orcfiledump prints something like
> {noformat}
> User Metadata:
> hive.acid.key.index=1,536870912,1;
> hive.acid.stats=2,0,0
> hive.acid.version=
> {noformat}
> probably because
> {noformat}
> public static void setAcidVersionInDataFile(Writer writer) {
> //so that we know which version wrote the file
> ByteBuffer bf = ByteBuffer.allocate(4).putInt(ORC_ACID_VERSION);
> bf.rewind(); //don't ask - some ByteBuffer weridness. w/o this, empty
> buffer is written
> writer.addUserMetadata(ACID_VERSION_KEY, bf);
> }
> {noformat}
> use
> {{UTF8.encode())}} instead
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names
[ https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450362#comment-16450362 ] Eugene Koifman commented on HIVE-19260: --- Thanks Prasanth for the review committed to master > Streaming Ingest API doesn't normalize db.table names > - > > Key: HIVE-19260 > URL: https://issues.apache.org/jira/browse/HIVE-19260 > Project: Hive > Issue Type: Bug > Components: HCatalog, Streaming, Transactions >Affects Versions: 0.13 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19260.01-branch-3.patch, HIVE-19260.01.patch > > > Hive is case insensitive wrt db/table names. These gets normalized to lower > case for SQL processing. > When HiveEndPoint is created it uses db.table strings as is, and they end up > propagated this way to transaction metadata tables in the metastore via lock > acquisition. This makes them look like different tables in Cleaner and lock > manager. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[ https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450358#comment-16450358 ] Gopal V commented on HIVE-19124: LGTM - +1 This ticket opens up an interesting question about the way metastore fires off compactor jobs (i.e the metastore has to have access to a YARN cluster & it needs to submit it to a specific queue etc). > implement a basic major compactor for MM tables > --- > > Key: HIVE-19124 > URL: https://issues.apache.org/jira/browse/HIVE-19124 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-19124.01.patch, HIVE-19124.02.patch, > HIVE-19124.03.patch, HIVE-19124.03.patch, HIVE-19124.04.patch, > HIVE-19124.05.patch, HIVE-19124.06.patch, HIVE-19124.07.patch, > HIVE-19124.patch > > > For now, it will run a query directly and only major compactions will be > supported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17657) export/import for MM tables is broken
[ https://issues.apache.org/jira/browse/HIVE-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450354#comment-16450354 ] Sergey Shelukhin commented on HIVE-17657: - Test failures are unrelated. [~ekoifman] can you take a look? > export/import for MM tables is broken > - > > Key: HIVE-17657 > URL: https://issues.apache.org/jira/browse/HIVE-17657 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Sergey Shelukhin >Priority: Major > Labels: mm-gap-2 > Attachments: HIVE-17657.01.patch, HIVE-17657.02.patch, > HIVE-17657.03.patch, HIVE-17657.04.patch, HIVE-17657.patch > > > there is mm_exim.q but it's not clear from the tests what file structure it > creates > On import the txnids in the directory names would have to be remapped if > importing to a different cluster. Perhaps export can be smart and export > highest base_x and accretive deltas (minus aborted ones). Then import can > ...? It would have to remap txn ids from the archive to new txn ids. This > would then mean that import is made up of several transactions rather than 1 > atomic op. (all locks must belong to a transaction) > One possibility is to open a new txn for each dir in the archive (where > start/end txn of file name is the same) and commit all of them at once (need > new TMgr API for that). This assumes using a shared lock (if any!) and thus > allows other inserts (not related to import) to occur. > What if you have delta_6_9, such as a result of concatenate? If we stipulate > that this must mean that there is no delta_6_6 or any other "obsolete" delta > in the archive we can map it to a new single txn delta_x_x. > Add read_only mode for tables (useful in general, may be needed for upgrade > etc) and use that to make the above atomic. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19291) Three underscores are in the CTAS example of the documentation
[
https://issues.apache.org/jira/browse/HIVE-19291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa reassigned HIVE-19291:
---
Assignee: Nishant Bangarwa
> Three underscores are in the CTAS example of the documentation
> ---
>
> Key: HIVE-19291
> URL: https://issues.apache.org/jira/browse/HIVE-19291
> Project: Hive
> Issue Type: Bug
> Components: Documentation
>Reporter: Wade Salazar
>Assignee: Nishant Bangarwa
>Priority: Trivial
>
> on the page https://cwiki.apache.org/confluence/display/Hive/Druid+Integration
>
> {{The following example is provided}}
> {{"}}
> {{CREATE TABLE druid_table_1}}
> {{STORED BY }}{{'org.apache.hadoop.hive.druid.DruidStorageHandler'}}
> {{AS}}
> {{ `metric2`>;}}
> {{"}}
>
> {{There are 3 underscores in front of the time dimension where the code only
> executes if 2 underscores are provided}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18037) Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x
[ https://issues.apache.org/jira/browse/HIVE-18037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450346#comment-16450346 ] Sergey Shelukhin commented on HIVE-18037: - +1 pending tests > Migrate Slider LLAP package to YARN Service framework for Hadoop 3.x > > > Key: HIVE-18037 > URL: https://issues.apache.org/jira/browse/HIVE-18037 > Project: Hive > Issue Type: Sub-task >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18037.001.patch, HIVE-18037.002.patch, > HIVE-18037.003.patch, HIVE-18037.004.patch > > > Apache Slider has been migrated to Hadoop-3.x and is referred to as YARN > Service (YARN-4692). Most of the classic Slider features are now going to be > supported in a first-class manner by core YARN. It includes several new > features like a RESTful API. Command line equivalents of classic Slider are > supported by YARN Service as well. > This jira will take care of all changes required to Slider LLAP packaging and > scripts to make it work against Hadoop 3.x. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18983) Add support for table properties inheritance in Create table like
[
https://issues.apache.org/jira/browse/HIVE-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Igor Kryvenko updated HIVE-18983:
-
Priority: Major (was: Minor)
> Add support for table properties inheritance in Create table like
> -
>
> Key: HIVE-18983
> URL: https://issues.apache.org/jira/browse/HIVE-18983
> Project: Hive
> Issue Type: Improvement
>Reporter: Igor Kryvenko
>Assignee: Igor Kryvenko
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18983.01.patch, HIVE-18983.02.patch,
> HIVE-18983.03.patch, HIVE-18983.04.patch, HIVE-18983.05.patch,
> HIVE-18983.06.patch, HIVE-18983.07.patch, HIVE-18983.08.patch,
> HIVE-18983.09.patch, HIVE-18983.10.patch, HIVE-18983.11.patch,
> HIVE-18983.12.patch
>
>
> Currently, Create table like support table properties.
> But it doesn't inherit table properties from original table.
> {code}
> create table T1(a int, b int) clustered by (a) into 2 buckets stored as orc
> TBLPROPERTIES ('comment'='comm');
> create table T like T1;
> show create table T;
> {code}
> *Output:*
> {code}
> CREATE TABLE `T`(
> `a` int,
> `b` int)
> CLUSTERED BY (
> a)
> INTO 2 BUCKETS
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> LOCATION
> 'maprfs:/user/hive/warehouse/t'
> TBLPROPERTIES (
> 'COLUMN_STATS_ACCURATE'='{\"BASIC_STATS\":\"true\"}',
> 'numFiles'='0',
> 'numRows'='0',
> 'rawDataSize'='0',
> 'totalSize'='0',
> 'transient_lastDdlTime'='1521230300')
> {code}
> It uses just default table properties and doesn't inherit properties from
> original table.
> It would be great if create table like will inherit origin table properties
> and will override them if they are specified in query .
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19271) TestMiniLlapLocalCliDriver default_constraint and check_constraint failing
[
https://issues.apache.org/jira/browse/HIVE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450343#comment-16450343
]
Hive QA commented on HIVE-19271:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10464/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10464/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestMiniLlapLocalCliDriver default_constraint and check_constraint failing
> --
>
> Key: HIVE-19271
> URL: https://issues.apache.org/jira/browse/HIVE-19271
> Project: Hive
> Issue Type: Test
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-19271.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19137) orcfiledump doesn't print hive.acid.version value
[
https://issues.apache.org/jira/browse/HIVE-19137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450339#comment-16450339
]
Eugene Koifman commented on HIVE-19137:
---
+1
> orcfiledump doesn't print hive.acid.version value
> -
>
> Key: HIVE-19137
> URL: https://issues.apache.org/jira/browse/HIVE-19137
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Igor Kryvenko
>Priority: Major
> Attachments: HIVE-19137-branch-3.01.patch, HIVE-19137.01.patch,
> HIVE-19137.02-branch-3.patch, HIVE-19137.02.patch,
> HIVE-19137.03-branch-3.patch, HIVE-19137.03.patch, HIVE-19137.04.patch,
> HIVE-19137.05.patch
>
>
> HIVE-18659 added hive.acid.version in the file footer.
> orcfiledump prints something like
> {noformat}
> User Metadata:
> hive.acid.key.index=1,536870912,1;
> hive.acid.stats=2,0,0
> hive.acid.version=
> {noformat}
> probably because
> {noformat}
> public static void setAcidVersionInDataFile(Writer writer) {
> //so that we know which version wrote the file
> ByteBuffer bf = ByteBuffer.allocate(4).putInt(ORC_ACID_VERSION);
> bf.rewind(); //don't ask - some ByteBuffer weridness. w/o this, empty
> buffer is written
> writer.addUserMetadata(ACID_VERSION_KEY, bf);
> }
> {noformat}
> use
> {{UTF8.encode())}} instead
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19198) Few flaky hcatalog tests
[
https://issues.apache.org/jira/browse/HIVE-19198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450333#comment-16450333
]
Hive QA commented on HIVE-19198:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920374/HIVE-19198.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 30 failed/errored test(s), 14296 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10463/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10463/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10463/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 30 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920374 - PreCommit-HIVE-Build
> Few flaky hcatalog tests
>
>
> Key: HIVE-19198
> URL: https://issues.apache.org/jira/browse/HIVE-19198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Chauhan
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19198.1.patch, HIVE-19198.2.patch
>
>
> TestPermsGrp : Consider removing this since hcat cli is not widely used.
> TestHCatPartitionPublish.testPartitionPublish
> TestHCatMultiOutputFormat.testOutputFormat
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-12414) ALTER TABLE UNSET SERDEPROPERTIES does not work
[
https://issues.apache.org/jira/browse/HIVE-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450272#comment-16450272
]
Anthony Hsu commented on HIVE-12414:
Currently, the only way to drop SERDEPROPERTIES is to use the Thrift API
directly (e.g.: using the MetaStoreClient via Java code).
> ALTER TABLE UNSET SERDEPROPERTIES does not work
> ---
>
> Key: HIVE-12414
> URL: https://issues.apache.org/jira/browse/HIVE-12414
> Project: Hive
> Issue Type: Bug
> Components: Metastore, SQL
>Affects Versions: 1.1.1
>Reporter: Lenni Kuff
>Assignee: Reuben Kuhnert
>Priority: Major
> Labels: newbie
>
> alter table tablename set tblproperties ('key'='value') => works as expected
> alter table tablename unset tblproperties ('key') => works as expected
> alter table tablename set serdeproperties ('key'='value') => works as
> expected
> alter table tablename unset serdeproperties ('key') => not supported
> FAILED: ParseException line 1:28 mismatched input 'serdeproperties' expecting
> TBLPROPERTIES near 'unset' in alter properties statement
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-12414) ALTER TABLE UNSET SERDEPROPERTIES does not work
[
https://issues.apache.org/jira/browse/HIVE-12414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Hsu updated HIVE-12414:
---
Summary: ALTER TABLE UNSET SERDEPROPERTIES does not work (was: ALTER TABLE
UNSET SERDEPROPERTY does not work)
> ALTER TABLE UNSET SERDEPROPERTIES does not work
> ---
>
> Key: HIVE-12414
> URL: https://issues.apache.org/jira/browse/HIVE-12414
> Project: Hive
> Issue Type: Bug
> Components: Metastore, SQL
>Affects Versions: 1.1.1
>Reporter: Lenni Kuff
>Assignee: Reuben Kuhnert
>Priority: Major
> Labels: newbie
>
> alter table tablename set tblproperties ('key'='value') => works as expected
> alter table tablename unset tblproperties ('key') => works as expected
> alter table tablename set serdeproperties ('key'='value') => works as
> expected
> alter table tablename unset serdeproperties ('key') => not supported
> FAILED: ParseException line 1:28 mismatched input 'serdeproperties' expecting
> TBLPROPERTIES near 'unset' in alter properties statement
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names
[ https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450265#comment-16450265 ] Prasanth Jayachandran commented on HIVE-19260: -- ok. lgtm, +1 > Streaming Ingest API doesn't normalize db.table names > - > > Key: HIVE-19260 > URL: https://issues.apache.org/jira/browse/HIVE-19260 > Project: Hive > Issue Type: Bug > Components: HCatalog, Streaming, Transactions >Affects Versions: 0.13 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19260.01-branch-3.patch, HIVE-19260.01.patch > > > Hive is case insensitive wrt db/table names. These gets normalized to lower > case for SQL processing. > When HiveEndPoint is created it uses db.table strings as is, and they end up > propagated this way to transaction metadata tables in the metastore via lock > acquisition. This makes them look like different tables in Cleaner and lock > manager. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450240#comment-16450240
]
Sahil Takiar commented on HIVE-19204:
-
+1 LGTM
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch, HIVE-19204.2.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result
> will have such exception set and Driver.java will get such details and
> display to the client. But some tasks don't set such exceptions so the client
> won't see such details unless you check the HS2 log.
>
> {noformat}
> public void runSequential() {
> int exitVal = -101;
> try {
> exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
> if (tsk.getException() == null) {
> tsk.setException(t);
> }
> LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
> result.setTaskError(tsk.getException());
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450238#comment-16450238
]
Aihua Xu commented on HIVE-19204:
-
[~stakiar] Can you take another look? Thanks.
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Attachments: HIVE-19204.1.patch, HIVE-19204.2.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result
> will have such exception set and Driver.java will get such details and
> display to the client. But some tasks don't set such exceptions so the client
> won't see such details unless you check the HS2 log.
>
> {noformat}
> public void runSequential() {
> int exitVal = -101;
> try {
> exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
> if (tsk.getException() == null) {
> tsk.setException(t);
> }
> LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
> result.setTaskError(tsk.getException());
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19198) Few flaky hcatalog tests
[
https://issues.apache.org/jira/browse/HIVE-19198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450220#comment-16450220
]
Hive QA commented on HIVE-19198:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
13s{color} | {color:red} hcatalog/core: The patch generated 3 new + 6 unchanged
- 1 fixed = 9 total (was 7) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 10m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10463/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10463/yetus/diff-checkstyle-hcatalog_core.txt
|
| modules | C: hcatalog/core U: hcatalog/core |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10463/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Few flaky hcatalog tests
>
>
> Key: HIVE-19198
> URL: https://issues.apache.org/jira/browse/HIVE-19198
> Project: Hive
> Issue Type: Sub-task
>Reporter: Ashutosh Chauhan
>Assignee: Daniel Dai
>Priority: Major
> Attachments: HIVE-19198.1.patch, HIVE-19198.2.patch
>
>
> TestPermsGrp : Consider removing this since hcat cli is not widely used.
> TestHCatPartitionPublish.testPartitionPublish
> TestHCatMultiOutputFormat.testOutputFormat
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[ https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450219#comment-16450219 ] Yongzhi Chen commented on HIVE-18986: - +1 for PATCH-4 > Table rename will run java.lang.StackOverflowError in dataNucleus if the > table contains large number of columns > --- > > Key: HIVE-18986 > URL: https://issues.apache.org/jira/browse/HIVE-18986 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch, > HIVE-18986.3.patch, HIVE-18986.4.patch > > > If the table contains a lot of columns e.g, 5k, simple table rename would > fail with the following stack trace. The issue is datanucleus can't handle > the query with lots of colName='c1' && colName='c2' && ... . > > 2018-03-13 17:19:52,770 INFO > org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: > ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: > db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 > 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-200]: java.lang.StackOverflowError at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19247) StatsOptimizer: Missing stats fast-path for Date
[
https://issues.apache.org/jira/browse/HIVE-19247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450218#comment-16450218
]
Gopal V commented on HIVE-19247:
Failures are unrelated, but two of failed the tests failed to startup the
CliDriver fully.
{code}
java.lang.AssertionError: Failed during initFromDatasets processLine with code=2
at org.junit.Assert.fail(Assert.java:88)
at org.apache.hadoop.hive.ql.QTestUtil.initDataset(QTestUtil.java:1227)
at
org.apache.hadoop.hive.ql.QTestUtil.initDataSetForTest(QTestUtil.java:1207)
at org.apache.hadoop.hive.ql.QTestUtil.cliInit(QTestUtil.java:1275)
at
org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:176)
at
org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver(TestCliDriver.java:5
{code}
> StatsOptimizer: Missing stats fast-path for Date
>
>
> Key: HIVE-19247
> URL: https://issues.apache.org/jira/browse/HIVE-19247
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 2.2.0, 3.0.0, 2.3.2
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-19247.1.patch
>
>
> {code}
> 2018-04-19T18:57:24,268 DEBUG [67259108-c184-4c92-9e18-9e296922
> HiveServer2-Handler-Pool: Thread-73]: optimizer.StatsOptimizer
> (StatsOptimizer.java:process(614)) - Unsupported type: date encountered in
> metadata optimizer for column : jour
> {code}
> {code}
> if (udaf instanceof GenericUDAFMin) {
> ExprNodeColumnDesc colDesc =
> (ExprNodeColumnDesc)exprMap.get(((ExprNodeColumnDesc)aggr.getParameters().get(0)).getColumn());
> String colName = colDesc.getColumn();
> StatType type = getType(colDesc.getTypeString());
> if (!tbl.isPartitioned()) {
> if
> (!StatsSetupConst.areColumnStatsUptoDate(tbl.getParameters(), colName)) {
> Logger.debug("Stats for table : " + tbl.getTableName() + "
> column " + colName
> + " are not up to date.");
> return null;
> }
> ColumnStatisticsData statData =
> hive.getMSC().getTableColumnStatistics(
> tbl.getDbName(), tbl.getTableName(),
> Lists.newArrayList(colName))
> .get(0).getStatsData();
> String name = colDesc.getTypeString().toUpperCase();
> switch (type) {
> case Integeral: {
> LongSubType subType = LongSubType.valueOf(name);
> LongColumnStatsData lstats = statData.getLongStats();
> if (lstats.isSetLowValue()) {
> oneRow.add(subType.cast(lstats.getLowValue()));
> } else {
> oneRow.add(null);
> }
> break;
> }
> case Double: {
> DoubleSubType subType = DoubleSubType.valueOf(name);
> DoubleColumnStatsData dstats = statData.getDoubleStats();
> if (dstats.isSetLowValue()) {
> oneRow.add(subType.cast(dstats.getLowValue()));
> } else {
> oneRow.add(null);
> }
> break;
> }
> default: // unsupported type
> Logger.debug("Unsupported type: " + colDesc.getTypeString()
> + " encountered in " +
> "metadata optimizer for column : " + colName);
> return null;
> }
> }
> {code}
> {code}
> enum StatType{
> Integeral,
> Double,
> String,
> Boolean,
> Binary,
> Unsupported
> }
> enum LongSubType {
> BIGINT { @Override
> Object cast(long longValue) { return longValue; } },
> INT { @Override
> Object cast(long longValue) { return (int)longValue; } },
> SMALLINT { @Override
> Object cast(long longValue) { return (short)longValue; } },
> TINYINT { @Override
> Object cast(long longValue) { return (byte)longValue; } };
> abstract Object cast(long longValue);
> }
> {code}
> Date is stored in stats (& also the typo there).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[ https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450207#comment-16450207 ] Aihua Xu commented on HIVE-18986: - The tests don't seem to be related. [~ychena] can you take a look at the patch? Thanks. > Table rename will run java.lang.StackOverflowError in dataNucleus if the > table contains large number of columns > --- > > Key: HIVE-18986 > URL: https://issues.apache.org/jira/browse/HIVE-18986 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch, > HIVE-18986.3.patch, HIVE-18986.4.patch > > > If the table contains a lot of columns e.g, 5k, simple table rename would > fail with the following stack trace. The issue is datanucleus can't handle > the query with lots of colName='c1' && colName='c2' && ... . > > 2018-03-13 17:19:52,770 INFO > org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: > ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: > db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 > 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-200]: java.lang.StackOverflowError at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19184) Hive 3.0.0 release branch preparation
[
https://issues.apache.org/jira/browse/HIVE-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450190#comment-16450190
]
Hive QA commented on HIVE-19184:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12919517/HIVE-19184.01-branch-3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 45 failed/errored test(s), 14136 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=253)
TestMiniDruidCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=253)
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=253)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_non_nullable_union]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[cachingprintstream]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[compute_stats_long]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dyn_part3]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dyn_part_max_per_node]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[dynamic_partitions_with_whitelist]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_broken_pipe2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_broken_pipe3]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[script_error]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[serde_regex2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_1]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_publisher_error_2]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_corr_in_agg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_in_implicit_gby]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notin_implicit_gby]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_corr_multi_rows]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_scalar_multi_rows]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true2]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_assert_true]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_reflect_neg]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error]
(batchId=96)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[udf_test_error_reduce]
(batchId=95)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[local_mapred_error_cache]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=242)
{noformat}
Test results:
[jira] [Updated] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names
[ https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19260: -- Attachment: HIVE-19260.01-branch-3.patch > Streaming Ingest API doesn't normalize db.table names > - > > Key: HIVE-19260 > URL: https://issues.apache.org/jira/browse/HIVE-19260 > Project: Hive > Issue Type: Bug > Components: HCatalog, Streaming, Transactions >Affects Versions: 0.13 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19260.01-branch-3.patch, HIVE-19260.01.patch > > > Hive is case insensitive wrt db/table names. These gets normalized to lower > case for SQL processing. > When HiveEndPoint is created it uses db.table strings as is, and they end up > propagated this way to transaction metadata tables in the metastore via lock > acquisition. This makes them look like different tables in Cleaner and lock > manager. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19228) Remove commons-httpclient 3.x usage
[ https://issues.apache.org/jira/browse/HIVE-19228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-19228: --- Attachment: HIVE-19228.2.patch > Remove commons-httpclient 3.x usage > --- > > Key: HIVE-19228 > URL: https://issues.apache.org/jira/browse/HIVE-19228 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-19228.1.patch, HIVE-19228.2.patch > > > Commons-httpclient is not supported well anymore. Remove dependency and move > to Apache HTTP client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names
[ https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450170#comment-16450170 ] Eugene Koifman commented on HIVE-19260: --- yes, it guaranteed to be not null > Streaming Ingest API doesn't normalize db.table names > - > > Key: HIVE-19260 > URL: https://issues.apache.org/jira/browse/HIVE-19260 > Project: Hive > Issue Type: Bug > Components: HCatalog, Streaming, Transactions >Affects Versions: 0.13 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-19260.01.patch > > > Hive is case insensitive wrt db/table names. These gets normalized to lower > case for SQL processing. > When HiveEndPoint is created it uses db.table strings as is, and they end up > propagated this way to transaction metadata tables in the metastore via lock > acquisition. This makes them look like different tables in Cleaner and lock > manager. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-19286:
--
Component/s: Transactions
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
>
[jira] [Commented] (HIVE-15767) Hive On Spark is not working on secure clusters from Oozie
[
https://issues.apache.org/jira/browse/HIVE-15767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450073#comment-16450073
]
Peter Cseh commented on HIVE-15767:
---
[~linwukang], we haven't seen this exception after the fix.
This might have to do something in regards the command you're executing or the
time frame where the job runs. Without knowing those it's hard to give
meaningful suggestions.
Can you check the expiry date for the HDFS token in the job?
> Hive On Spark is not working on secure clusters from Oozie
> --
>
> Key: HIVE-15767
> URL: https://issues.apache.org/jira/browse/HIVE-15767
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Affects Versions: 1.2.1, 2.1.1
>Reporter: Peter Cseh
>Assignee: Peter Cseh
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-15767-001.patch, HIVE-15767-002.patch,
> HIVE-15767.1.patch
>
>
> When a HiveAction is launched form Oozie with Hive On Spark enabled, we're
> getting errors:
> {noformat}
> Caused by: java.io.IOException: Exception reading
> file:/yarn/nm/usercache/yshi/appcache/application_1485271416004_0022/container_1485271416004_0022_01_02/container_tokens
> at
> org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:188)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:155)
> {noformat}
> This is caused by passing the {{mapreduce.job.credentials.binary}} property
> to the Spark configuration in RemoteHiveSparkClient.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19184) Hive 3.0.0 release branch preparation
[
https://issues.apache.org/jira/browse/HIVE-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450049#comment-16450049
]
Hive QA commented on HIVE-19184:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10462/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10462/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive 3.0.0 release branch preparation
> -
>
> Key: HIVE-19184
> URL: https://issues.apache.org/jira/browse/HIVE-19184
> Project: Hive
> Issue Type: Task
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19184.01-branch-3.patch
>
>
> Need to do bunch of things to prepare branch-3 for release e.g.
> * Update pom.xml to delete SNAPSHOT
> * Update .reviewboardrc
> * Remove storage-api module to build
> * Change storage-api depdency etc
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450045#comment-16450045
]
Oleksiy Sayankin commented on HIVE-19286:
-
PS : the above multiple {{INSERT}} query does not work even on
{{TextInputFormat}} tables with the same error:
{code}
FAILED: SemanticException [Error 10087]: The same output cannot be present
multiple times: customer_target
{code}
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
>
[jira] [Commented] (HIVE-19283) Select count(distinct()) a couple of times stuck in last reducer
[
https://issues.apache.org/jira/browse/HIVE-19283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450041#comment-16450041
]
Gopal V commented on HIVE-19283:
This was fixed sometime during hive-3.x
https://github.com/apache/hive/blob/master/ql/src/test/results/clientpositive/tez/multi_count_distinct.q.out
> Select count(distinct()) a couple of times stuck in last reducer
>
>
> Key: HIVE-19283
> URL: https://issues.apache.org/jira/browse/HIVE-19283
> Project: Hive
> Issue Type: Improvement
> Components: CBO, Logical Optimizer
>Affects Versions: 2.1.1
>Reporter: Goun Na
>Assignee: Ashutosh Chauhan
>Priority: Major
>
> Distinct count query performance is significantly improved due to
> HIVE-10568.
> {code:java}
> select count(distinct elevenst_id)
> from 11st.log_table
> where part_dt between '20180101' and '20180131'{code}
>
> However, some queries with several distinct counts are still slow. It starts
> with multiple mappers, but stuck in the last one reducer.
> {code:java}
> select
> count(distinct elevenst_id)
> , count(distinct member_id)
> , count(distinct user_id)
> , count(distinct action_id)
> , count(distinct other_id)
> from 11st.log_table
> where part_dt between '20180101' and '20180131'{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19247) StatsOptimizer: Missing stats fast-path for Date
[
https://issues.apache.org/jira/browse/HIVE-19247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450032#comment-16450032
]
Hive QA commented on HIVE-19247:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920372/HIVE-19247.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 35 failed/errored test(s), 14297 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_decimal_date]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_columnPruning]
(batchId=83)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.metastore.client.TestDefaultConstraint.addNoSuchTable[Remote]
(batchId=211)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testDropTableWithoutPurge[Embedded]
(batchId=209)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10461/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10461/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10461/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 35 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920372 - PreCommit-HIVE-Build
> StatsOptimizer: Missing stats fast-path for Date
>
>
> Key: HIVE-19247
> URL: https://issues.apache.org/jira/browse/HIVE-19247
> Project: Hive
>
[jira] [Commented] (HIVE-19178) TestMiniTezCliDriver.testCliDriver[explainanalyze_5] failure
[
https://issues.apache.org/jira/browse/HIVE-19178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449982#comment-16449982
]
Jesus Camacho Rodriguez commented on HIVE-19178:
[~ashutoshc], [~vgarg], I'm taking a look at it today.
> TestMiniTezCliDriver.testCliDriver[explainanalyze_5] failure
>
>
> Key: HIVE-19178
> URL: https://issues.apache.org/jira/browse/HIVE-19178
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Vineet Garg
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
>
> I have verified that this failure is due to HIVE-18825.
> Error stack:
> {code}
> java.lang.IllegalStateException: calling recordValidTxn() more than once in
> the same txnid:5
> at org.apache.hadoop.hive.ql.Driver.acquireLocks(Driver.java:1439)
> at org.apache.hadoop.hive.ql.Driver.lockAndRespond(Driver.java:1624)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1794)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1538)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1527)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:137)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:287)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:635)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1655)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1602)
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1597)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:200)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:335)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClientInternal(QTestUtil.java:1455)
> at
> org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:1429)
> at
> org.apache.hadoop.hive.cli.control.CoreCliDriver.runTest(CoreCliDriver.java:177)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter.runTest(CliAdapter.java:104)
> at
> org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver(TestMiniTezCliDriver.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$2$1.evaluate(CliAdapter.java:92)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
> at org.junit.runners.Suite.runChild(Suite.java:127)
> at org.junit.runners.Suite.runChild(Suite.java:26)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
> at
> org.apache.hadoop.hive.cli.control.CliAdapter$1$1.evaluate(CliAdapter.java:73)
> at
[jira] [Commented] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449980#comment-16449980
]
Oleksiy Sayankin commented on HIVE-19286:
-
I tried to find out was it the problem in {{{MERGE}} statement or in multiple
{{INSERT}}. Skipping cardinality check I have got after
{code}
FROM
`default`.`customer_target` `trg`
RIGHT OUTER JOIN
`default`.`customer_source` `src`
ON `src`.`id` = `trg`.`id`
INSERT INTO `default`.`customer_target`-- update clause
select `trg`.ROW__ID, `trg`.`id`, `src`.`first_name`, `src`.`last_name`,
`trg`.`age`
WHERE `src`.`id` = `trg`.`id`
sort by `trg`.ROW__ID
INSERT INTO `default`.`customer_target`-- insert clause
select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
WHERE `trg`.`id` IS NULL
{code}
this result
{code}
FAILED: NoMatchingMethodException No matching method for class
org.apache.hadoop.hive.ql.udf.UDFToString with
(struct). Possible choices:
_FUNC_(bigint) _FUNC_(binary) _FUNC_(boolean) _FUNC_(date)
_FUNC_(decimal(38,18)) _FUNC_(double) _FUNC_(float) _FUNC_(int)
_FUNC_(smallint) _FUNC_(string) _FUNC_(timestamp) _FUNC_(tinyint)
_FUNC_(void)
{code}
Then I added simple implementation into {{UDFToString}}:
{code}
public Text evaluate(Object object){
if (object == null){
return null;
}
return new Text(object.toString());
}
{code}
And I got this exception:
{code}
FAILED: SemanticException [Error 10087]: The same output cannot be present
multiple times: customer_target
{code}
Right the same exception I have obtained when removed {{ROW__ID}} from
multiple {{INSERT}} query:
{code}
FROM
`default`.`customer_target` `trg`
RIGHT OUTER JOIN
`default`.`customer_source` `src`
ON `src`.`id` = `trg`.`id`
INSERT INTO `default`.`customer_target`-- update clause
select `trg`.`id`, `src`.`first_name`, `src`.`last_name`, `trg`.`age`
WHERE `src`.`id` = `trg`.`id`
sort by `trg`.id
INSERT INTO `default`.`customer_target`-- insert clause
select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
WHERE `trg`.`id` IS NULL
{code}
Looks like multiple {{INSERT}} queries do not work for ORC on MR engine.
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
>
[jira] [Updated] (HIVE-19267) Create/Replicate ACID Write event
[ https://issues.apache.org/jira/browse/HIVE-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] mahesh kumar behera updated HIVE-19267: --- Attachment: HIVE-19267.02.patch > Create/Replicate ACID Write event > - > > Key: HIVE-19267 > URL: https://issues.apache.org/jira/browse/HIVE-19267 > Project: Hive > Issue Type: Sub-task > Components: repl, Transactions >Affects Versions: 3.0.0 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.0.0 > > Attachments: HIVE-19267.01.patch, HIVE-19267.02.patch > > > > h1. Replicate ACID write Events > * Create new EVENT_WRITE event with related message format to log the write > operations with in a txn along with data associated. > * Log this event when perform any writes (insert into, insert overwrite, > load table, delete, update, merge, truncate) on table/partition. > * If a single MERGE/UPDATE/INSERT/DELETE statement operates on multiple > partitions, then need to log one event per partition. > * DbNotificationListener should log this type of event to special metastore > table named "MTxnWriteNotificationLog". > * This table should maintain a map of txn ID against list of > tables/partitions written by given txn. > * The entry for a given txn should be removed by the cleaner thread that > removes the expired events from EventNotificationTable. > h1. Replicate Commit Txn operation (with writes) > Add new EVENT_COMMIT_TXN to log the metadata/data of all tables/partitions > modified within the txn. > *Source warehouse:* > * This event should read the EVENT_WRITEs from "MTxnWriteNotificationLog" > metastore table to consolidate the list of tables/partitions modified within > this txn scope. > * Based on the list of tables/partitions modified and table Write ID, need > to compute the list of delta files added by this txn. > * Repl dump should read this message and dump the metadata and delta files > list. > *Target warehouse:* > * Ensure snapshot isolation at target for on-going read txns which shouldn't > view the data replicated from committed txn. (Ensured with open and allocate > write ID events). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19287) parse error with semicolon in comment which starts with whitespace in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449974#comment-16449974 ] zhuwei commented on HIVE-19287: --- [~abstractdog] Thanks . I have revise the title. Actually, it's related with both semicolon and whitespace. If there is no semicolon, the whitespace will be handled later . > parse error with semicolon in comment which starts with whitespace in file > -- > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19288) Implement protobuf logging hive hook.
[ https://issues.apache.org/jira/browse/HIVE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19288: - Status: Patch Available (was: Open) Implemented proto based logger. The logger, reader and writer was copied from tez, I wanted to reuse them but realized that code is in unrelease tez version and hence a copy right now. > Implement protobuf logging hive hook. > - > > Key: HIVE-19288 > URL: https://issues.apache.org/jira/browse/HIVE-19288 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19288.01.patch > > > Implement a protobuf based logger which will log hive hook events into date > partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19289) Grouping function with several argument does not work
[
https://issues.apache.org/jira/browse/HIVE-19289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mykhailo Kysliuk updated HIVE-19289:
Description:
This steps are taken from [Apache wiki |
https://cwiki.apache.org/confluence/display/Hive/Enhanced+Aggregation%2C+Cube%2C+Grouping+and+Rollup]
*General Info*
Hive version : 2.3.3
{code:java}
commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
Author: Daniel Dai
Date: Wed Mar 28 16:46:29 2018 -0700
Preparing for 2.3.3 release
{code}
Hadoop version: 2.7.2.
Engine
{code:java}
hive> set hive.execution.engine;
hive.execution.engine=mr{code}
*STEP 1. Create test data*
{code:java}
DROP TABLE IF EXISTS T1;
CREATE TABLE T1 (key int, value int);
INSERT INTO T1 VALUES (1,NULL),(1,1),(2,2),(3,3),(3,NULL),(4,5);
SELECT * FROM T1;
{code}
{code:java}
1 NULL
1 1
2 2
3 3
3 NULL
4 5
{code}
*STEP 2. Execute group by query*
{code:java}
SELECT key, value, GROUPING__ID,
grouping(key, value), grouping(value, key), grouping(key), grouping(value),
count(*)
FROM T1
GROUP BY key, value WITH ROLLUP;
{code}
*ACTUAL RESULT*
{code:java}
FAILED: SemanticException [Error 10016]: Line 1:33 Argument type mismatch
'value': The second argument to grouping() must be a constant
{code}
Stacktrace:
{code:java}
2018-04-25T18:08:25,438 ERROR [6033d3ac-f243-4e54-8724-d9665fed531c main]
parse.CalcitePlanner: CBO failed, skipping CBO.
org.apache.hadoop.hive.ql.parse.SemanticException: Line 2:16 Argument type
mismatch 'value': The second argument to grouping() must be a constant
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1361)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:229)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:176)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11613)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11568)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genSelectLogicalPlan(CalcitePlanner.java:3706)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.genLogicalPlan(CalcitePlanner.java:3999)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1315)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1261)
~[hive-exec-2.3.3.jar:2.3.3]
at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
~[calcite-core-1.10.0.jar:1.10.0]
at
org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
~[calcite-core-1.10.0.jar:1.10.0]
at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
~[calcite-core-1.10.0.jar:1.10.0]
at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
~[calcite-core-1.10.0.jar:1.10.0]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
~[hive-exec-2.3.3.jar:2.3.3]
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
~[hive-exec-2.3.3.jar:2.3.3]
at
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment which starts with whitespace in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Summary: parse error with semicolon in comment which starts with whitespace in file (was: parse error with semicolon in comment with in file) > parse error with semicolon in comment which starts with whitespace in file > -- > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment with in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Summary: parse error with semicolon in comment with in file (was: parse error with semicolon in comment with in file) > parse error with semicolon in comment with in file > --- > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19288) Implement protobuf logging hive hook.
[ https://issues.apache.org/jira/browse/HIVE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19288: - Attachment: HIVE-19288.01.patch > Implement protobuf logging hive hook. > - > > Key: HIVE-19288 > URL: https://issues.apache.org/jira/browse/HIVE-19288 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19288.01.patch > > > Implement a protobuf based logger which will log hive hook events into date > partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment with in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Summary: parse error with semicolon in comment with in file (was: parse error with semicolon in comment in file) > parse error with semicolon in comment with in file > -- > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Status: Patch Available (was: Open) > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Attachment: HIVE-19202.2.patch > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19288) Implement protobuf logging hive hook.
[ https://issues.apache.org/jira/browse/HIVE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash reassigned HIVE-19288: > Implement protobuf logging hive hook. > - > > Key: HIVE-19288 > URL: https://issues.apache.org/jira/browse/HIVE-19288 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > > Implement a protobuf based logger which will log hive hook events into date > partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Attachment: HIVE-19287.2.patch > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Attachment: (was: HIVE-19202.2.patch) > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch, HIVE-19287.2.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17429) Hive JDBC doesn't return rows when querying Impala
[ https://issues.apache.org/jira/browse/HIVE-17429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449955#comment-16449955 ] Mark Smith commented on HIVE-17429: --- [~aihuaxu] thank you very much for backporting it, I'm sorry it's taken me some time to circle back and try upgrading again. Unfortunately, I'm still seeing the same problem with 2.2.1 and 2.3.3 with an exception being thrown from here: https://github.com/apache/hive/blob/rel/release-2.3.3/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L477 so I think there must be some other issue with how GetOperationStatus() is being built in the underlying TCLIService.Iface client: [https://github.com/apache/hive/blob/rel/release-2.3.3/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L254 https://github.com/apache/hive/blob/rel/release-2.3.3/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L367|https://github.com/apache/hive/blob/rel/release-2.3.3/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L367] Unless the underlying client's response is fixed, I think the 2.2.x and 2.3.x branches will need to do the same workaround/fix as master, also checking the `stmtHandle` for a resultSet: [https://github.com/apache/hive/blob/master/jdbc/src/java/org/apache/hive/jdbc/HiveStatement.java#L269] In case you want to test the same queries as I'm running against Impala via JDBC, I'm doing the following: * "SHOW TABLE STATS foo.oldTable" * "CREATE EXTERNAL TABLE foo.newTable LIKE foo.oldTable STORED AS PARQUET LOCATION 'hdfs://MyNameServer/location/newFooDir'" * "SHOW CREATE TABLE foo.TEMP_newTable" I admit that the two "SHOW" ones are a little unusual as 'queries', but they seem to work fine with the 2.1.1 driver. > Hive JDBC doesn't return rows when querying Impala > -- > > Key: HIVE-17429 > URL: https://issues.apache.org/jira/browse/HIVE-17429 > Project: Hive > Issue Type: Bug > Components: JDBC >Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.3.1, 2.3.2 >Reporter: Zach Amsden >Assignee: Zach Amsden >Priority: Major > Fix For: 2.1.0, 2.1.1, 3.0.0, 2.2.1, 2.3.3 > > Attachments: HIVE-17429.1.patch, HIVE-17429.2.patch > > > The Hive JDBC driver used to return a result set when querying Impala. Now, > instead, it gets data back but interprets the data as query logs instead of a > resultSet. This causes many issues (we see complaints about beeline as well > as test failures). > This appears to be a regression introduced with asynchronous operation > against Hive. > Ideally, we could make both behaviors work. I have a simple patch that > should fix the problem. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19247) StatsOptimizer: Missing stats fast-path for Date
[
https://issues.apache.org/jira/browse/HIVE-19247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449933#comment-16449933
]
Hive QA commented on HIVE-19247:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
0s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 93 new + 223 unchanged - 8
fixed = 316 total (was 231) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 8 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10461/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10461/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10461/yetus/whitespace-eol.txt
|
| modules | C: itests ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10461/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> StatsOptimizer: Missing stats fast-path for Date
>
>
> Key: HIVE-19247
> URL: https://issues.apache.org/jira/browse/HIVE-19247
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 2.2.0, 3.0.0, 2.3.2
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Attachments: HIVE-19247.1.patch
>
>
> {code}
> 2018-04-19T18:57:24,268 DEBUG [67259108-c184-4c92-9e18-9e296922
> HiveServer2-Handler-Pool: Thread-73]: optimizer.StatsOptimizer
> (StatsOptimizer.java:process(614)) - Unsupported type: date encountered in
> metadata optimizer for column : jour
> {code}
> {code}
> if (udaf instanceof GenericUDAFMin) {
> ExprNodeColumnDesc colDesc =
> (ExprNodeColumnDesc)exprMap.get(((ExprNodeColumnDesc)aggr.getParameters().get(0)).getColumn());
> String colName = colDesc.getColumn();
> StatType type = getType(colDesc.getTypeString());
> if (!tbl.isPartitioned()) {
> if
> (!StatsSetupConst.areColumnStatsUptoDate(tbl.getParameters(), colName)) {
> Logger.debug("Stats for table : " +
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Status: Open (was: Patch Available) > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449912#comment-16449912 ] Laszlo Bodor commented on HIVE-19287: - [~qunyan] : the issue title could be a bit misleading, as the patch fixes the case when commented line starts with whitespace, so it's not about the semicolon, isn't it? > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Status: Patch Available (was: Open) add line.trim() to fix the bug > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19281) incorrect protocol name for LLAP AM plugin
[
https://issues.apache.org/jira/browse/HIVE-19281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449879#comment-16449879
]
Hive QA commented on HIVE-19281:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920367/HIVE-19281.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 14296 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.metastore.TestStats.partitionedTableInHiveCatalog
(batchId=211)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testConcurrentLineage (batchId=242)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10460/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10460/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10460/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920367 - PreCommit-HIVE-Build
> incorrect protocol name for LLAP AM plugin
> --
>
> Key: HIVE-19281
> URL: https://issues.apache.org/jira/browse/HIVE-19281
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments:
[jira] [Updated] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei updated HIVE-19287: -- Attachment: HIVE-19287.1.patch > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > Attachments: HIVE-19287.1.patch > > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19287) parse error with semicolon in comment in file
[ https://issues.apache.org/jira/browse/HIVE-19287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuwei reassigned HIVE-19287: - > parse error with semicolon in comment in file > - > > Key: HIVE-19287 > URL: https://issues.apache.org/jira/browse/HIVE-19287 > Project: Hive > Issue Type: Bug > Environment: hive 2.2.1 >Reporter: zhuwei >Assignee: zhuwei >Priority: Minor > > It will get error when hive query written in file look like this: > select col > --this is; an example > from db.table > limit 1; > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449820#comment-16449820
]
Oleksiy Sayankin edited comment on HIVE-19286 at 4/24/18 1:15 PM:
--
My results after some debug. NPE happens because
{{inspector.getStructFieldRef(names[0]);}} returns {{null}}
{code}
@Override
public ObjectInspector initialize(ObjectInspector rowInspector) throws
HiveException {
// We need to support field names like KEY.0, VALUE.1 between
// map-reduce boundary.
String[] names = expr.getColumn().split("\\.");
String[] unionfields = names[0].split("\\:");
if (names.length == 1 && unionfields.length == 1) {
simpleCase = true;
inspector = (StructObjectInspector) rowInspector;
field = inspector.getStructFieldRef(names[0]);
return outputOI = field.getFieldObjectInspector();
}
{code}
in {{ExprNodeColumnEvaluator}}. Here {{names[0]}} == {{"ROW__ID"}}. Class
{{OrcStruct}} contains method
{code}
@Override
public StructField getStructFieldRef(String s) {
for(StructField field: fields) {
if (field.getFieldName().equalsIgnoreCase(s)) {
return field;
}
}
return null;
}
{code}
and array {{fields}} is initialized with {{StructField}} for only four columns:
{{id}}, {{first_name}}, {{last_name}}, {{age}}. So it returns {{null}} when
{{s}} == {{"ROW__ID"}} . Hive wants no insert {{"ROW__ID"}} because it
transforms {{MERGE}} into multiple {{INSERT}}:
{code}
FROM
`default`.`customer_target` `trg`
RIGHT OUTER JOIN
`default`.`customer_source` `src`
ON `src`.`id` = `trg`.`id`
INSERT INTO `default`.`customer_target`-- update clause
select `trg`.ROW__ID, `trg`.`id`, `src`.`first_name`, `src`.`last_name`,
`trg`.`age`
WHERE `src`.`id` = `trg`.`id`
sort by `trg`.ROW__ID
INSERT INTO `default`.`customer_target`-- insert clause
select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
WHERE `trg`.`id` IS NULL
INSERT INTO merge_tmp_table
SELECT cardinality_violation(`trg`.ROW__ID)
WHERE `src`.`id` = `trg`.`id` GROUP BY `trg`.ROW__ID HAVING count(*) > 1
{code}
was (Author: osayankin):
My results after some debug. NPE happens because
{{inspector.getStructFieldRef(names[0]);}} returns {{null}}
{code}
@Override
public ObjectInspector initialize(ObjectInspector rowInspector) throws
HiveException {
// We need to support field names like KEY.0, VALUE.1 between
// map-reduce boundary.
String[] names = expr.getColumn().split("\\.");
String[] unionfields = names[0].split("\\:");
if (names.length == 1 && unionfields.length == 1) {
simpleCase = true;
inspector = (StructObjectInspector) rowInspector;
field = inspector.getStructFieldRef(names[0]);
return outputOI = field.getFieldObjectInspector();
}
{code}
in {{ExprNodeColumnEvaluator}}. Here {{names[0] == "ROW__ID"}}. Class
{{OrcStruct}} contains method
{code}
@Override
public StructField getStructFieldRef(String s) {
for(StructField field: fields) {
if (field.getFieldName().equalsIgnoreCase(s)) {
return field;
}
}
return null;
}
{code}
and array {{fields}} is initialized with {{StructField}} for only four columns:
{{id}}, {{first_name}}, {{last_name}}, {{age}}. So it returns {{null}} when {{s
== "ROW__ID"}}. Hive wants no insert {{ROW__ID}} because it transforms
{{MERGE}} into multiple {{INSERT}}:
{code}
FROM
`default`.`customer_target` `trg`
RIGHT OUTER JOIN
`default`.`customer_source` `src`
ON `src`.`id` = `trg`.`id`
INSERT INTO `default`.`customer_target`-- update clause
select `trg`.ROW__ID, `trg`.`id`, `src`.`first_name`, `src`.`last_name`,
`trg`.`age`
WHERE `src`.`id` = `trg`.`id`
sort by `trg`.ROW__ID
INSERT INTO `default`.`customer_target`-- insert clause
select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
WHERE `trg`.`id` IS NULL
INSERT INTO merge_tmp_table
SELECT cardinality_violation(`trg`.ROW__ID)
WHERE `src`.`id` = `trg`.`id` GROUP BY `trg`.ROW__ID HAVING count(*) > 1
{code}
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1.
[jira] [Commented] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449820#comment-16449820
]
Oleksiy Sayankin commented on HIVE-19286:
-
My results after some debug. NPE happens because
{{inspector.getStructFieldRef(names[0]);}} returns {{null}}
{code}
@Override
public ObjectInspector initialize(ObjectInspector rowInspector) throws
HiveException {
// We need to support field names like KEY.0, VALUE.1 between
// map-reduce boundary.
String[] names = expr.getColumn().split("\\.");
String[] unionfields = names[0].split("\\:");
if (names.length == 1 && unionfields.length == 1) {
simpleCase = true;
inspector = (StructObjectInspector) rowInspector;
field = inspector.getStructFieldRef(names[0]);
return outputOI = field.getFieldObjectInspector();
}
{code}
in {{ExprNodeColumnEvaluator}}. Here {{names[0] == "ROW__ID"}}. Class
{{OrcStruct}} contains method
{code}
@Override
public StructField getStructFieldRef(String s) {
for(StructField field: fields) {
if (field.getFieldName().equalsIgnoreCase(s)) {
return field;
}
}
return null;
}
{code}
and array {{fields}} is initialized with {{StructField}} for only four columns:
{{id}}, {{first_name}}, {{last_name}}, {{age}}. So it returns {{null}} when {{s
== "ROW__ID"}}. Hive wants no insert {{ROW__ID}} because it transforms
{{MERGE}} into multiple {{INSERT}}:
{code}
FROM
`default`.`customer_target` `trg`
RIGHT OUTER JOIN
`default`.`customer_source` `src`
ON `src`.`id` = `trg`.`id`
INSERT INTO `default`.`customer_target`-- update clause
select `trg`.ROW__ID, `trg`.`id`, `src`.`first_name`, `src`.`last_name`,
`trg`.`age`
WHERE `src`.`id` = `trg`.`id`
sort by `trg`.ROW__ID
INSERT INTO `default`.`customer_target`-- insert clause
select `src`.`id`, `src`.`first_name`, `src`.`last_name`, `src`.`age`
WHERE `trg`.`id` IS NULL
INSERT INTO merge_tmp_table
SELECT cardinality_violation(`trg`.ROW__ID)
WHERE `src`.`id` = `trg`.`id` GROUP BY `trg`.ROW__ID HAVING count(*) > 1
{code}
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> Engine
> {code}
> hive> set hive.execution.engine;
> hive.execution.engine=mr
> {code}
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
>
[jira] [Commented] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs
[
https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449794#comment-16449794
]
Sahil Takiar commented on HIVE-17193:
-
+1
> HoS: don't combine map works that are targets of different DPPs
> ---
>
> Key: HIVE-17193
> URL: https://issues.apache.org/jira/browse/HIVE-17193
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-17193.1.patch, HIVE-17193.2.patch,
> HIVE-17193.3.patch, HIVE-17193.4.patch, HIVE-17193.5.patch
>
>
> Suppose {{srcpart}} is partitioned by {{ds}}. The following query can trigger
> the issue:
> {code}
> explain
> select * from
> (select srcpart.ds,srcpart.key from srcpart join src on srcpart.ds=src.key)
> a
> join
> (select srcpart.ds,srcpart.key from srcpart join src on
> srcpart.ds=src.value) b
> on a.key=b.key;
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter
[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16295:
Attachment: HIVE-16295.2.WIP.patch
> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a
> {{NullOutputCommitter}} and uses its own commit logic spread across
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with
> S3Guard and does a safe, coordinate commit of data on S3 inside individual
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}}
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from
> task retries or speculative execution) should not step on each other
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[
https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449738#comment-16449738
]
Hive QA commented on HIVE-19252:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920370/HIVE-19252.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 36 failed/errored test(s), 14296 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testAutoPurgeUnset (batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedNoAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateInvalidAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateNoAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncatePartitionedNoAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateUnsetAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveBackKill
(batchId=242)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10459/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10459/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10459/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 36 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12920370 - PreCommit-HIVE-Build
> TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
>
>
> Key: HIVE-19252
> URL:
[jira] [Commented] (HIVE-18423) Hive should support usage of external tables using jdbc
[ https://issues.apache.org/jira/browse/HIVE-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449716#comment-16449716 ] Jonathan Doron commented on HIVE-18423: --- [~jcamachorodriguez] I have fixed all failed test, please review. > Hive should support usage of external tables using jdbc > --- > > Key: HIVE-18423 > URL: https://issues.apache.org/jira/browse/HIVE-18423 > Project: Hive > Issue Type: Improvement >Reporter: Jonathan Doron >Assignee: Jonathan Doron >Priority: Major > Labels: pull-request-available > Fix For: 3.1.0 > > Attachments: HIVE-18423.1.patch, HIVE-18423.2.patch, > HIVE-18423.3.patch, HIVE-18423.4.patch, HIVE-18423.5.patch, HIVE-18423.6.patch > > > Hive should support the usage of external jdbc tables(and not only external > tables that hold queries), so an Hive user would be able to use the external > table as an hive internal table. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Status: Patch Available (was: Open) Added 04.patch with * Logic to timeout the open txns which are opened before triggering bootstrap. * Replicate the write ids state in target based on validWriteIdlist for each ACID/MM table getting replicated. > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Attachment: HIVE-18988.04.patch > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Description:
*General Info*
Hive version : 2.3.3
{code}
commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
Author: Daniel Dai
Date: Wed Mar 28 16:46:29 2018 -0700
Preparing for 2.3.3 release
{code}
Hadoop version: 2.7.2.
Engine
{code}
hive> set hive.execution.engine;
hive.execution.engine=mr
{code}
*STEP 1. Create test data*
{code}
DROP TABLE IF EXISTS customer_target;
DROP TABLE IF EXISTS customer_source;
{code}
{code}
CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
('transactional'='true');
{code}
{code}
insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
SELECT id, first_name, last_name, age FROM customer_target;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 002 | Michael | Watson | 27 |
| 001 | John| Smith | 45 |
| 003 | Den | Brown | 33 |
+--+-++--+
{code}
{code}
CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
age INT);
insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
SELECT id, first_name, last_name, age FROM customer_source;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 088 | Robert | Dowson | 25 |
| 001 | Dorothi | Hogward| 77 |
| 007 | Alex| Bowee | 1|
+--+-++--+
{code}
*STEP 2. Merge data*
{code}
merge into customer_target trg using customer_source src on src.id = trg.id
when matched then update set first_name = src.first_name, last_name =
src.last_name when not matched then insert values (src.id, src.first_name,
src.last_name, src.age);
{code}
*ACTUAL RESULT*
{code}
2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities: Deserializing
MapredLocalWork using kryo
2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
Starting to launch local task to process map join; maximum memory =
477626368
2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
trg, children = [HASHTABLESINK[37]]
2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for trg
created
2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
operator TS[0]
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
Done 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
initialized
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
children of 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
child 37 HASHTABLESINK
2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
operator HASHTABLESINK[37]
2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
JVM Max Heap Size: 477626368
2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
Map local work failed
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Description:
*General Info*
Hive version : 2.3.3
{code}
commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
Author: Daniel Dai
Date: Wed Mar 28 16:46:29 2018 -0700
Preparing for 2.3.3 release
{code}
Hadoop version: 2.7.2.
*STEP 1. Create test data*
{code}
DROP TABLE IF EXISTS customer_target;
DROP TABLE IF EXISTS customer_source;
{code}
{code}
CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
('transactional'='true');
{code}
{code}
insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
SELECT id, first_name, last_name, age FROM customer_target;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 002 | Michael | Watson | 27 |
| 001 | John| Smith | 45 |
| 003 | Den | Brown | 33 |
+--+-++--+
{code}
{code}
CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
age INT);
insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
SELECT id, first_name, last_name, age FROM customer_source;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 088 | Robert | Dowson | 25 |
| 001 | Dorothi | Hogward| 77 |
| 007 | Alex| Bowee | 1|
+--+-++--+
{code}
*STEP 2. Merge data*
{code}
merge into customer_target trg using customer_source src on src.id = trg.id
when matched then update set first_name = src.first_name, last_name =
src.last_name when not matched then insert values (src.id, src.first_name,
src.last_name, src.age);
{code}
*ACTUAL RESULT*
{code}
2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities: Deserializing
MapredLocalWork using kryo
2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
Starting to launch local task to process map join; maximum memory =
477626368
2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
trg, children = [HASHTABLESINK[37]]
2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for trg
created
2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
operator TS[0]
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
Done 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
initialized
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
children of 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
child 37 HASHTABLESINK
2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
operator HASHTABLESINK[37]
2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
JVM Max Heap Size: 477626368
2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
Map local work failed
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:411)
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Affects Version/s: 2.3.3
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
>
[jira] [Work started] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-19286 started by Oleksiy Sayankin.
---
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.3
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> *General Info*
> Hive version : 2.3.3
> {code}
> commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
> Author: Daniel Dai
> Date: Wed Mar 28 16:46:29 2018 -0700
> Preparing for 2.3.3 release
> {code}
> Hadoop version: 2.7.2.
> *STEP 1. Create test data*
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> *STEP 2. Merge data*
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> *ACTUAL RESULT*
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
>
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Description:
*General Info*
Hive version : 2.3.3
{code}
commit 3f7dde31aed44b5440563d3f9d8a8887beccf0be
Author: Daniel Dai
Date: Wed Mar 28 16:46:29 2018 -0700
Preparing for 2.3.3 release
{code}
Hadoop version: 2.7.2.
*STEP 1. Create test data*
{code}
DROP TABLE IF EXISTS customer_target;
DROP TABLE IF EXISTS customer_source;
{code}
{code}
CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
('transactional'='true');
{code}
{code}
insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
SELECT id, first_name, last_name, age FROM customer_target;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 002 | Michael | Watson | 27 |
| 001 | John| Smith | 45 |
| 003 | Den | Brown | 33 |
+--+-++--+
{code}
{code}
CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
age INT);
insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
SELECT id, first_name, last_name, age FROM customer_source;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 088 | Robert | Dowson | 25 |
| 001 | Dorothi | Hogward| 77 |
| 007 | Alex| Bowee | 1|
+--+-++--+
{code}
*STEP 2. Merge data*
{code}
merge into customer_target trg using customer_source src on src.id = trg.id
when matched then update set first_name = src.first_name, last_name =
src.last_name when not matched then insert values (src.id, src.first_name,
src.last_name, src.age);
{code}
*ACTUAL RESULT*
{code}
2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities: Deserializing
MapredLocalWork using kryo
2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
Starting to launch local task to process map join; maximum memory =
477626368
2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
trg, children = [HASHTABLESINK[37]]
2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for trg
created
2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
operator TS[0]
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
Done 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
initialized
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
children of 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
child 37 HASHTABLESINK
2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
operator HASHTABLESINK[37]
2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
JVM Max Heap Size: 477626368
2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
Map local work failed
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:411)
[jira] [Commented] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[
https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449654#comment-16449654
]
Hive QA commented on HIVE-19252:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 38s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10459/dev-support/hive-personality.sh
|
| git revision | master / 56c3a95 |
| Default Java | 1.8.0_111 |
| modules | C: itests/hive-minikdc U: itests/hive-minikdc |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10459/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
>
>
> Key: HIVE-19252
> URL: https://issues.apache.org/jira/browse/HIVE-19252
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19252.1.patch, HIVE-19252.1.patch
>
>
> For last 8 builds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin updated HIVE-19286:
Description:
*STEP 1. Create test data*
{code}
DROP TABLE IF EXISTS customer_target;
DROP TABLE IF EXISTS customer_source;
{code}
{code}
CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
('transactional'='true');
{code}
{code}
insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
SELECT id, first_name, last_name, age FROM customer_target;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 002 | Michael | Watson | 27 |
| 001 | John| Smith | 45 |
| 003 | Den | Brown | 33 |
+--+-++--+
{code}
{code}
CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
age INT);
insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
SELECT id, first_name, last_name, age FROM customer_source;
{code}
{code}
+--+-++--+
| id | first_name | last_name | age |
+--+-++--+
| 088 | Robert | Dowson | 25 |
| 001 | Dorothi | Hogward| 77 |
| 007 | Alex| Bowee | 1|
+--+-++--+
{code}
*STEP 2. Merge data*
{code}
merge into customer_target trg using customer_source src on src.id = trg.id
when matched then update set first_name = src.first_name, last_name =
src.last_name when not matched then insert values (src.id, src.first_name,
src.last_name, src.age);
{code}
*ACTUAL RESULT*
{code}
2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities: Deserializing
MapredLocalWork using kryo
2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger:
2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
Starting to launch local task to process map join; maximum memory =
477626368
2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
trg, children = [HASHTABLESINK[37]]
2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
Session not initiated, use thread based class loader instead
2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for trg
created
2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
operator TS[0]
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
Done 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
initialized
2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
children of 0 TS
2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
child 37 HASHTABLESINK
2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
operator HASHTABLESINK[37]
2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
JVM Max Heap Size: 477626368
2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
Map local work failed
java.lang.NullPointerException
at
org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:376)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.initializeOperators(MapredLocalTask.java:508)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.startForward(MapredLocalTask.java:411)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask.executeInProcess(MapredLocalTask.java:391)
~[hive-exec-2.3.3.jar:2.3.3]
at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:764)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Attachment: (was: HIVE-18988.04.patch) > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Description: Bootstrapping of ACID tables, need special handling to replicate a stable state of data. - If ACID feature enables, then perform bootstrap dump for ACID tables with in read txn. -> Dump table/partition metadata. -> Get the list of valid data files for a table using same logic as read txn do. -> Dump latest ValidWriteIdList as per current read txn. - Set the valid last replication state such that it doesn't miss any open txn started after triggering bootstrap dump. - If any txns on-going which was opened before triggering bootstrap dump, then it is not guaranteed that if open_txn event captured for these txns. Also, if these txns are opened for streaming ingest case, then dumped ACID table data may include data of open txns which impact snapshot isolation at target. To avoid that, bootstrap dump should wait for timeout (new configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, just force abort those txns and continue. - If any txns force aborted belongs to a streaming ingest case, then dumped ACID table data may have aborted data too. So, it is necessary to replicate the aborted write ids to target to mark those data invalid for any readers. was: Bootstrapping of ACID tables, need special handling to replicate a stable state of data. - If ACID feature enables, then perform bootstrap dump for ACID tables with in read txn. -> Dump table/partition metadata. -> Get the list of valid data files for a table using same logic as read txn do. -> Dump latest ValidWriteIdList as per current read txn. - Find the valid last replication state such that it points to event ID of open_txn event of oldest on-going txn. > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Set the valid last replication state such that it doesn't miss any open > txn started after triggering bootstrap dump. > - If any txns on-going which was opened before triggering bootstrap dump, > then it is not guaranteed that if open_txn event captured for these txns. > Also, if these txns are opened for streaming ingest case, then dumped ACID > table data may include data of open txns which impact snapshot isolation at > target. To avoid that, bootstrap dump should wait for timeout (new > configuration: hive.repl.bootstrap.dump.open.txn.timeout). After timeout, > just force abort those txns and continue. > - If any txns force aborted belongs to a streaming ingest case, then dumped > ACID table data may have aborted data too. So, it is necessary to replicate > the aborted write ids to target to mark those data invalid for any readers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19286) NPE in MERGE operator on MR mode
[
https://issues.apache.org/jira/browse/HIVE-19286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksiy Sayankin reassigned HIVE-19286:
---
> NPE in MERGE operator on MR mode
>
>
> Key: HIVE-19286
> URL: https://issues.apache.org/jira/browse/HIVE-19286
> Project: Hive
> Issue Type: Bug
>Reporter: Oleksiy Sayankin
>Assignee: Oleksiy Sayankin
>Priority: Blocker
>
> {code}
> DROP TABLE IF EXISTS customer_target;
> DROP TABLE IF EXISTS customer_source;
> {code}
> {code}
> CREATE TABLE customer_target (id STRING, first_name STRING, last_name STRING,
> age INT) clustered by (id) into 2 buckets stored as ORC TBLPROPERTIES
> ('transactional'='true');
> {code}
> {code}
> insert into customer_target values ('001', 'John', 'Smith', 45), ('002',
> 'Michael', 'Watson', 27), ('003', 'Den', 'Brown', 33);
> SELECT id, first_name, last_name, age FROM customer_target;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 002 | Michael | Watson | 27 |
> | 001 | John| Smith | 45 |
> | 003 | Den | Brown | 33 |
> +--+-++--+
> {code}
> {code}
> CREATE TABLE customer_source (id STRING, first_name STRING, last_name STRING,
> age INT);
> insert into customer_source values ('001', 'Dorothi', 'Hogward', 77), ('007',
> 'Alex', 'Bowee', 1), ('088', 'Robert', 'Dowson', 25);
> SELECT id, first_name, last_name, age FROM customer_source;
> {code}
> {code}
> +--+-++--+
> | id | first_name | last_name | age |
> +--+-++--+
> | 088 | Robert | Dowson | 25 |
> | 001 | Dorothi | Hogward| 77 |
> | 007 | Alex| Bowee | 1|
> +--+-++--+
> {code}
> {code}
> merge into customer_target trg using customer_source src on src.id = trg.id
> when matched then update set first_name = src.first_name, last_name =
> src.last_name when not matched then insert values (src.id, src.first_name,
> src.last_name, src.age);
> {code}
> {code}
> 2018-04-24T07:11:44,448 DEBUG [main] log.PerfLogger: method=deserializePlan
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,448 INFO [main] exec.SerializationUtilities:
> Deserializing MapredLocalWork using kryo
> 2018-04-24T07:11:44,463 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,538 DEBUG [main] log.PerfLogger: method=deserializePlan start=1524568304448 end=1524568304538 duration=90
> from=org.apache.hadoop.hive.ql.exec.SerializationUtilities>
> 2018-04-24T07:11:44,545 INFO [main] mr.MapredLocalTask: 2018-04-24 07:11:44
> Starting to launch local task to process map join; maximum memory =
> 477626368
> 2018-04-24T07:11:44,545 DEBUG [main] mr.MapredLocalTask: initializeOperators:
> trg, children = [HASHTABLESINK[37]]
> 2018-04-24T07:11:44,656 DEBUG [main] exec.Utilities: Hive Conf not found or
> Session not initiated, use thread based class loader instead
> 2018-04-24T07:11:44,676 INFO [main] mr.MapredLocalTask: fetchoperator for
> trg created
> 2018-04-24T07:11:44,676 INFO [main] exec.TableScanOperator: Initializing
> operator TS[0]
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initialization
> Done 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Operator 0 TS
> initialized
> 2018-04-24T07:11:44,676 DEBUG [main] exec.TableScanOperator: Initializing
> children of 0 TS
> 2018-04-24T07:11:44,676 DEBUG [main] exec.HashTableSinkOperator: Initializing
> child 37 HASHTABLESINK
> 2018-04-24T07:11:44,676 INFO [main] exec.HashTableSinkOperator: Initializing
> operator HASHTABLESINK[37]
> 2018-04-24T07:11:44,677 INFO [main] mapjoin.MapJoinMemoryExhaustionHandler:
> JVM Max Heap Size: 477626368
> 2018-04-24T07:11:44,680 ERROR [main] mr.MapredLocalTask: Hive Runtime Error:
> Map local work failed
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator.initialize(ExprNodeColumnEvaluator.java:57)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.JoinUtil.getObjectInspectorsFromEvaluators(JoinUtil.java:91)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.HashTableSinkOperator.initializeOp(HashTableSinkOperator.java:153)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:366)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
> org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:556)
> ~[hive-exec-2.3.3.jar:2.3.3]
> at
>
[jira] [Commented] (HIVE-19054) Function replication shall use "hive.repl.replica.functions.root.dir" as root
[
https://issues.apache.org/jira/browse/HIVE-19054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449642#comment-16449642
]
Hive QA commented on HIVE-19054:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12917274/HIVE-19054.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 34 failed/errored test(s), 14296 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=39)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[windowing_windowspec4]
(batchId=68)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[results_cache_invalidation2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_1]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.parse.repl.load.message.TestPrimaryToReplicaResourceFunction.createDestinationPath
(batchId=280)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithMiniKdcCookie.testCookieNegative
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10458/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10458/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10458/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 34 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12917274 - PreCommit-HIVE-Build
> Function replication shall use "hive.repl.replica.functions.root.dir" as root
> -
>
> Key: HIVE-19054
> URL: https://issues.apache.org/jira/browse/HIVE-19054
> Project: Hive
> Issue Type: Bug
>
[jira] [Updated] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18988: Attachment: HIVE-18988.04.patch > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch, HIVE-18988.04.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Find the valid last replication state such that it points to event ID of > open_txn event of oldest on-going txn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work stopped] (HIVE-18988) Support bootstrap replication of ACID tables
[ https://issues.apache.org/jira/browse/HIVE-18988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-18988 stopped by Sankar Hariappan. --- > Support bootstrap replication of ACID tables > > > Key: HIVE-18988 > URL: https://issues.apache.org/jira/browse/HIVE-18988 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, repl >Affects Versions: 3.0.0 >Reporter: Sankar Hariappan >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available, replication > Fix For: 3.1.0 > > Attachments: HIVE-18988.01.patch, HIVE-18988.02.patch, > HIVE-18988.03.patch > > > Bootstrapping of ACID tables, need special handling to replicate a stable > state of data. > - If ACID feature enables, then perform bootstrap dump for ACID tables with > in read txn. > -> Dump table/partition metadata. > -> Get the list of valid data files for a table using same logic as read txn > do. > -> Dump latest ValidWriteIdList as per current read txn. > - Find the valid last replication state such that it points to event ID of > open_txn event of oldest on-going txn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19211) New streaming ingest API and support for dynamic partitioning
[ https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19211: - Attachment: HIVE-19211.6.patch > New streaming ingest API and support for dynamic partitioning > - > > Key: HIVE-19211 > URL: https://issues.apache.org/jira/browse/HIVE-19211 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19211.1.patch, HIVE-19211.2.patch, > HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch, HIVE-19211.6.patch > > > - New streaming API under new hive sub-module > - Dynamic partitioning support > - Auto-rollover transactions > - Automatic heartbeating -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Attachment: HIVE-19285.1.patch > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > Attachments: HIVE-19285.1.patch > > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Status: Patch Available (was: Open) > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > Attachments: HIVE-19285.1.patch > > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Affects Version/s: (was: 3.1.0) 3.0.0 > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.0.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Component/s: HiveServer2 > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-19285: - Affects Version/s: 3.1.0 > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Affects Versions: 3.1.0 >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19285) Add logs to the subclasses of MetaDataOperation
[ https://issues.apache.org/jira/browse/HIVE-19285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora reassigned HIVE-19285: > Add logs to the subclasses of MetaDataOperation > --- > > Key: HIVE-19285 > URL: https://issues.apache.org/jira/browse/HIVE-19285 > Project: Hive > Issue Type: Improvement >Reporter: Marta Kuczora >Assignee: Marta Kuczora >Priority: Minor > > Subclasses of MetaDataOperation are not writing anything to the logs. It > would be useful to have some INFO and DEBUG level logging in these classes. > The following classes are affected > * GetCatalogsOperation > * GetColumnsOperation > * GetFunctionsOperation > * GetSchemasOperation > * GetTablesOperation > * GetTypeInfoOperation > * GetTableTypesOperation > * GetCrossReferenceOperation > * GetPrimaryKeysOperation -- This message was sent by Atlassian JIRA (v7.6.3#76005)
