[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548051#comment-13548051 ] Hudson commented on HIVE-3443: -- Integrated in Hive-trunk-hadoop2 #54 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/54/]) HIVE-3443. Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key (Shreepadma Venugopalan via cws) (Revision 1388098) Backout HIVE-3443 (cws) (Revision 1386857) HIVE-3443. Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key (Shreepadma Venugopalan via cws) (Revision 1386782) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388098 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386857 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386782 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Fix For: 0.10.0 > > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt, > HIVE-3443.3.patch.txt, HIVE-3443.4.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13460241#comment-13460241 ] Hudson commented on HIVE-3443: -- Integrated in Hive-trunk-h0.21 #1687 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1687/]) HIVE-3443. Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key (Shreepadma Venugopalan via cws) (Revision 1388098) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1388098 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Fix For: 0.10.0 > > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt, > HIVE-3443.3.patch.txt, HIVE-3443.4.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458126#comment-13458126 ] Carl Steinbach commented on HIVE-3443: -- +1 > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt, > HIVE-3443.3.patch.txt, HIVE-3443.4.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13458029#comment-13458029 ] Shreepadma Venugopalan commented on HIVE-3443: -- Updated patch available on both JIRA and review board. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt, > HIVE-3443.3.patch.txt, HIVE-3443.4.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457583#comment-13457583 ] Hudson commented on HIVE-3443: -- Integrated in Hive-trunk-h0.21 #1678 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1678/]) Backout HIVE-3443 (cws) (Revision 1386857) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386857 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt, > HIVE-3443.3.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457328#comment-13457328 ] Hudson commented on HIVE-3443: -- Integrated in Hive-trunk-h0.21 #1676 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1676/]) HIVE-3443. Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key (Shreepadma Venugopalan via cws) (Revision 1386782) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1386782 Files : * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java * /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/tools/HiveMetaTool.java * /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaTool.java > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt, > HIVE-3443.3.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13457237#comment-13457237 ] Carl Steinbach commented on HIVE-3443: -- +1. Will commit if tests pass. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt, > HIVE-3443.3.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455004#comment-13455004 ] Shreepadma Venugopalan commented on HIVE-3443: -- Updated patch available for review at : https://reviews.apache.org/r/6962/ > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt, > HIVE-3443.3.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452616#comment-13452616 ] Carl Steinbach commented on HIVE-3443: -- Comments on reviewboard. Please change the status to 'patch available' when this ticket is ready for review. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452564#comment-13452564 ] Shreepadma Venugopalan commented on HIVE-3443: -- Updated patch available for review at : https://reviews.apache.org/r/6962/ > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt, HIVE-3443.2.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452376#comment-13452376 ] Shreepadma Venugopalan commented on HIVE-3443: -- @Carl, Jakob: Note that the code Jakob is pointing to is from the patch I uploaded prior to this discussion. I've not uploaded the latest patch yet, but it doesn't have anything that is Avro specific. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452372#comment-13452372 ] Shreepadma Venugopalan commented on HIVE-3443: -- I'm not adding any code that is specific to Haivvreo to Hive, nor is this patch supporting the change from Haivvreo to AvroSerde. Note that this tool is specifically intended for migration to HA and **not** to support change from Haivvreo to AvroSerde. This patch makes the metatool more flexible and generic to allow the users to update both Haivvreo and AvroSerde specific metadata that resides in the metastore. If its not Hive's responsibility to support non-ASF code such as Haivvreo, then Haivvreo specific property such as schema.url shouldn't be past of the metastore in the first place. Given that it is and users need a way to migrate to HA, there is not a better and a convenient way to do this migration other than make Metatool flexible enough to take in serdeParamKey from the user. This change is fixing what you are specifically objecting to i.e., Hive shouldn't know about Haivvreo. With this patch, Hive Metatool will be oblivious to whether its updating Haivvreo or AvroSerde metadata. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452370#comment-13452370 ] Carl Steinbach commented on HIVE-3443: -- Thanks for pointing that out. Shreepadma, please remove the AvroSerDe/Haivvreo specific stuff from this patch. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452367#comment-13452367 ] Jakob Homan commented on HIVE-3443: --- {code}+if (key != null) { + LOG.info("Looking for location in the value field of "+ key + " key in SERDES table..."); +} else { + LOG.info("Looking for location in the value field of schema.url/avro.schema.url key in " + + "SERDES table..."); +}{code} Seems pretty Haivvreo or AvroSerDe specific... > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452362#comment-13452362 ] Carl Steinbach commented on HIVE-3443: -- @Jakob: the code that we're trying to add isn't specific to Haivvreo or the AvroSerDe, but making the transition easier is one of the use cases we are trying to address. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452358#comment-13452358 ] Jakob Homan commented on HIVE-3443: --- You're attempting to introduce code designed to support the change from Haivvreo to AvroSerde. Once again, it is not Hive's responsibility to support non-ASF code like Haivvreo. If you want this functionality, it should be added to Haivvreo (and I'd be happy to review any pull requests there), not Hive. If users chose to use Haivvreo (or had it via vendor-supplied packages), it's their (or their vendor's) responsibility to update as necessary. There is no need to add code or complexity to Hive to support non-ASF code. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452350#comment-13452350 ] Shreepadma Venugopalan commented on HIVE-3443: -- @Jakob: This tool has been added to support migration to HA NN for Hive customers. I'm not sure I understand what you mean by change to the Sedere class since its name changed. Hive Metatool replies on the key "schema.url" to update avro schema url records that contain a reference to the NN. Given the recent change from schema.url to avro.schema.url, unless Hive Metatool allows the user to pass in the serdeParamKey, it can't reliably update all metastore records to refer the the right NN. Failing to update some of the metastore records will render some tables in the warehouse inaccessible. This is in no way a minor bug. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Minor > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13452326#comment-13452326 ] Jakob Homan commented on HIVE-3443: --- I don't believe Apache Hive is obligated to support automating this relatively simple task. Are you also going to change to the Serde class, since its name changed? Are you going to update input and outputformat at the same time? People writing tools on github (including me) that eventually get merged in some format to the main Hive branch do not obligate Hive to support the old github version in any way. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451729#comment-13451729 ] Shreepadma Venugopalan commented on HIVE-3443: -- I think this approach may be the safest, even though users have to be aware of the fact that avro serde's schema url property is stored as table prop and other serde props are stored as serde props. I can't think of an approach that is cleaner. If you have other suggestions, I'm open to hearing them. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451726#comment-13451726 ] Carl Steinbach commented on HIVE-3443: -- Yup, I think that makes sense. Is there another approach that might work better? > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451725#comment-13451725 ] Shreepadma Venugopalan commented on HIVE-3443: -- @Carl: Are you suggesting that metatool should take in the serdeParamKey in addition to tableProp to update potential serde location? Just so that I've understood you are suggesting the following, metatool -tableProp -serdeParamKey > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451724#comment-13451724 ] Carl Steinbach commented on HIVE-3443: -- Yup, thanks for clarifying. That makes sense. bq. I think its safe to attempt an update of serde prop schema.url irrespective of whether the user passes in the tableProp. If the table or partition record is using the Haivvreo SerDe and has a schema.url property defined, then I agree that it's probably safe to update this field even if the user didn't specify it. However, one of the points Jakob made in his comment was that the name "schema.url" is pretty generic, and there's the chance that other SerDes may use the same key for other purposes. I think it would be safer all around to force the user to manually specify the serdeproperties and tableproperties that they want to update. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451718#comment-13451718 ] Shreepadma Venugopalan commented on HIVE-3443: -- @Carl: StorageDescriptor has a param key/value pair. When I say propValue I mean "avro.schema.url" i.e., value of the field param key in storagedescriptor. I guess this is confusing. How about -tableProp propKey where a valid propKey is avro.schema.url? > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451706#comment-13451706 ] Carl Steinbach commented on HIVE-3443: -- Are users required to the specify the propValue field? If someone is trying to replace the NN address for a bunch of Avro tables, the actual propValue is going to be slightly different for each table since it includes the path as well as the host. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451695#comment-13451695 ] Shreepadma Venugopalan commented on HIVE-3443: -- @Carl: Good catch. Yup, looks like we have to take in tableProp instead of serdeProp. I need to look into this a little more carefully but what I have in mind at this time is, metatool -updateLocation -tableProp propKey=propValue I think in addition to addressing the avro use case, it will make the tool a lot more generic. However, we also have to consider users who are using schema.url as a serde property at this time. I think its safe to attempt an update of serde prop schema.url irrespective of whether the user passes in the tableProp. What are your thoughts on this? > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > Attachments: HIVE-3443.1.patch.txt > > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13451034#comment-13451034 ] Shreepadma Venugopalan commented on HIVE-3443: -- Review available on review board: https://reviews.apache.org/r/6962/ Thanks. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3443) Hive Metatool should take serde_param_key from the user to allow for changes to avro serde's schema url key
[ https://issues.apache.org/jira/browse/HIVE-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13450242#comment-13450242 ] Shreepadma Venugopalan commented on HIVE-3443: -- Support for Hive MetaTool was added in HIVE-3056. > Hive Metatool should take serde_param_key from the user to allow for changes > to avro serde's schema url key > --- > > Key: HIVE-3443 > URL: https://issues.apache.org/jira/browse/HIVE-3443 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 0.10.0 >Reporter: Shreepadma Venugopalan >Assignee: Shreepadma Venugopalan >Priority: Critical > > Hive Metatool should take serde_param_key from the user to allow for chanes > to avro serde's schema url key. In the past "avro.schema.url" key used to be > called "schema.url". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira