Thank you Rahul
From: [gmail.com] rahul patwari <[email protected]> Sent: Tuesday, April 28, 2020 4:21 PM To: user Subject: Re: HCatalogIO - Trying to read table metadata (columns names and indexes) Hi Noam, Currently, Beam doesn't support conversion of HCatRecords to Rows (or) in your case creating Beam Schema from Hive table schema, when the Hive table have parameterized types. We can use HCatFieldSchema[1] to create the Beam Schema from the Hive table Schema. I have created a JIRA ticket to track this issue: https://issues.apache.org/jira/browse/BEAM-9840<https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_BEAM-2D9840&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=UpRPq6_EtLYraBrDMcIghqO8oK1s_hVA83P0RHySIg4&e=> [1]: https://github.com/apache/hive/blob/f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb/hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/schema/HCatFieldSchema.java#L34<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hive_blob_f37c5de6c32b9395d1b34fa3c02ed06d1bfbf6eb_hcatalog_core_src_main_java_org_apache_hive_hcatalog_data_schema_HCatFieldSchema.java-23L34&d=DwMFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=kWQOeV5GJ_ztlIIab74XZdlzlC79Sv1LkwjyfchuvJQ&e=> PS: I am working on supporting this feature. This feature should be supported in the future releases of Apache Beam. Regards, Rahul On Mon, Apr 27, 2020 at 6:57 PM Gershi, Noam <[email protected]<mailto:[email protected]>> wrote: Hi Using HCatalogIO as a source - I am trying to read column tables. Code: PCollection<HCatRecord> hcatRecords = input .apply(HCatalogIO.read() .withConfigProperties(configProperties) .withDatabase(“db-name”) .withTable(“my-table-name”)); ... HCatalogBeamSchema hcatSchema = HCatalogBeamSchema.create(ImmutableMap.of("table", "my-table-name")); Schema schema = hcatSchema.getTableSchema("db-name", "my-table-name”).get(); List<Schema.Field> fields = schema.getFields(); I get: 20/04/27 09:12:16 INFO LineBufferedStream: Caused by: java.lang.UnsupportedOperationException: The type 'decimal(30,16)' of field 'amount' is not supported. 20/04/27 09:12:16 INFO LineBufferedStream: at org.apache.beam.sdk.io.hcatalog.SchemaUtils.toBeamField(SchemaUtils.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__SchemaUtils.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=76JA525sdVMYK1G-Vha1lBn2XUF2h_R_WwUNc1Q9-qc&e=>:60) 20/04/27 09:12:16 INFO LineBufferedStream: at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__ReferencePipeline.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=RcLJiCaEj7fycdwpNdx5BvNkoswqLqrk_dX6D35DIkc&e=>:193) 20/04/27 09:12:16 INFO LineBufferedStream: at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__ArrayList.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=1efQPGAljBR3scZEp0Z6TUZV-xdiDlc9OR2SFDjAn0M&e=>:1382) 20/04/27 09:12:16 INFO LineBufferedStream: at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__AbstractPipeline.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=5tBDEX1CqoedvIU7uZOuuFCe6l9rL9zpvfLOtQbLJ48&e=>:481) 20/04/27 09:12:16 INFO LineBufferedStream: at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__AbstractPipeline.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=5tBDEX1CqoedvIU7uZOuuFCe6l9rL9zpvfLOtQbLJ48&e=>:471) 20/04/27 09:12:16 INFO LineBufferedStream: at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__ReduceOps.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=kgGI-HEV-KrhnFePW7QjNxMJRhlUcozHxC0kAk6QPHM&e=>:708) 20/04/27 09:12:16 INFO LineBufferedStream: at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__AbstractPipeline.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=5tBDEX1CqoedvIU7uZOuuFCe6l9rL9zpvfLOtQbLJ48&e=>:234) 20/04/27 09:12:16 INFO LineBufferedStream: at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__ReferencePipeline.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=RcLJiCaEj7fycdwpNdx5BvNkoswqLqrk_dX6D35DIkc&e=>:499) 20/04/27 09:12:16 INFO LineBufferedStream: at org.apache.beam.sdk.io.hcatalog.SchemaUtils.toBeamSchema(SchemaUtils.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__SchemaUtils.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=76JA525sdVMYK1G-Vha1lBn2XUF2h_R_WwUNc1Q9-qc&e=>:53) 20/04/27 09:12:16 INFO LineBufferedStream: at org.apache.beam.sdk.io.hcatalog.HCatalogBeamSchema.getTableSchema(HCatalogBeamSchema.java<https://urldefense.proofpoint.com/v2/url?u=http-3A__HCatalogBeamSchema.java&d=DwQFaQ&c=j-EkbjBYwkAB4f8ZbVn1Fw&r=weSBkYYkcVBTPEmCbP7YQPAjEPSbBXyqPYqK6vLTOAM&m=eCupsSkN3aRgRJ6bFLVuP8YSNUCJ-FVHiW68E9ssX9w&s=Lu2fV9icqfqmkJLl0ZSZG5jJg1jauUMwgVjrIVATJzY&e=>:83) Thanx in advance, Noam
