Hello Ravi, When you wget this url
wget http://<api_server>:9091/schema?name=ed&store=parquet& isMutated=true&table=ed&secbypass=testing' Do you get avsc file? Regards, Jagat Singh On Sat, 27 Jun 2020, 7:01 am ravi kanth, <ravi....@gmail.com> wrote: > Just want to follow up on the below email. > > Thanks, > Ravi > > > On Thu, Jun 25, 2020 at 5:37 PM ravi kanth <ravi....@gmail.com> wrote: > >> Hi Community, >> >> Hive Version: 3.1.2 >> >> We are working on building a Hive Avro table on a few Avro files. I am >> able to succesfully create the table and query it when I have the Avro >> schema definition(avsc) file on hdfs with no issues. >> >> However, when trying to load the same schema from a rest API(as mentioned >> in >> https://cwiki.apache.org/confluence/display/Hive/AvroSerDe#AvroSerDe-CreatingAvro-backedHivetables), >> hive throws an exception and fails to create the table. >> >> *Sample table:* >> CREATE EXTERNAL TABLE ed_avro_1 >> STORED AS AVRO >> LOCATION '/tmp/sample/yyyymmdd=20200206' >> TBLPROPERTIES ('avro.schema.literal'='http:// >> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing'); >> >> When launched hive in INFO mode below is the trace of the problem which >> looks like Hive is interpreting the URL as a file name and throwing out a >> FileNotFoundException. >> >> I have tried using avro.schema.literal instead of avro.schema.url, >> however it turns out that hive is interpreting URL as a String and throwing >> a jackson parsing error. >> >> Can anyone help look into this? Is this a bug in Hive-3.1.2? Any details >> will be of great help. >> >> Thanks, >> Ravi >> >> >> StackTrace: >> >>> 2020-06-26T00:06:03,283 INFO [main] org.apache.hadoop.hive.conf.HiveConf >>>> - Using the default value passed in for log id: >>>> 646da35b-84b0-43aa-9b68-5d668ebbfc36 >>> >>> 2020-06-26T00:06:03,283 INFO [main] >>>> org.apache.hadoop.hive.ql.session.SessionState - Updating thread name to >>>> 646da35b-84b0-43aa-9b68-5d668ebbfc36 main >>> >>> 2020-06-26T00:06:03,286 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] ql.Driver: Compiling >>>> command(queryId=hdfs_20200626000603_0992e79f-6e1c-4383-be62-a6466c4c1cf2): >>>> CREATE EXTERNAL TABLE ed_avro_1 >>> >>> STORED AS AVRO >>> >>> LOCATION '/tmp/event_detail/yyyymmdd=20200206' >>> >>> TBLPROPERTIES ('avro.schema.url'='http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing') >>> >>> 2020-06-26T00:06:03,630 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] ql.Driver: Concurrency mode is disabled, not creating a lock manager >>> >>> 2020-06-26T00:06:03,638 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] parse.CalcitePlanner: Starting Semantic Analysis >>> >>> 2020-06-26T00:06:03,669 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController >>>> for session context : HiveAuthzSessionContext >>>> [sessionString=646da35b-84b0-43aa-9b68-5d668ebbfc36, clientType=HIVECLI] >>> >>> 2020-06-26T00:06:03,673 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 main] >>>> org.apache.hadoop.hive.ql.session.SessionState - METASTORE_FILTER_HOOK will >>>> be ignored, since hive.security.authorization.manager is set to instance of >>>> HiveAuthorizerFactory. >>> >>> 2020-06-26T00:06:03,673 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.HiveMetaStoreClient: Mestastore configuration >>>> metastore.filter.hook changed from >>>> org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to >>>> org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook >>> >>> 2020-06-26T00:06:03,675 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.HiveMetaStore: 0: Cleaning up thread local RawStore... >>> >>> 2020-06-26T00:06:03,675 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=Cleaning up >>>> thread local RawStore... >>> >>> 2020-06-26T00:06:03,676 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore >>> >>> 2020-06-26T00:06:03,676 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=Done >>>> cleaning up thread local RawStore >>> >>> 2020-06-26T00:06:03,680 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.HiveMetaStore: 0: Opening raw store with implementation >>>> class:org.apache.hadoop.hive.metastore.ObjectStore >>> >>> 2020-06-26T00:06:03,680 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to >>>> unsupported value null . Setting it to value: ignored >>> >>> 2020-06-26T00:06:03,681 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.ObjectStore: ObjectStore, initialize called >>> >>> 2020-06-26T00:06:03,691 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is >>>> MYSQL >>> >>> 2020-06-26T00:06:03,691 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.ObjectStore: Initialized ObjectStore >>> >>> 2020-06-26T00:06:03,692 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.RetryingMetaStoreClient: RetryingMetaStoreClient >>>> proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient >>>> ugi=hdfs (auth:SIMPLE) retries=1 delay=1 lifetime=0 >>> >>> 2020-06-26T00:06:03,704 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] parse.CalcitePlanner: Creating table scratch.ed_avro_1 position=22 >>> >>> 2020-06-26T00:06:03,719 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to >>>> unsupported value null . Setting it to value: ignored >>> >>> 2020-06-26T00:06:03,719 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.ObjectStore: ObjectStore, initialize called >>> >>> 2020-06-26T00:06:03,730 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is >>>> MYSQL >>> >>> 2020-06-26T00:06:03,730 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.ObjectStore: Initialized ObjectStore >>> >>> 2020-06-26T00:06:03,731 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.RetryingMetaStoreClient: RetryingMetaStoreClient >>>> proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient >>>> ugi=hdfs (auth:SIMPLE) retries=1 delay=1 lifetime=0 >>> >>> 2020-06-26T00:06:03,731 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.HiveMetaStore: 0: get_database: @hive#scratch >>> >>> 2020-06-26T00:06:03,731 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=get_database: >>>> @hive#scratch >>> >>> 2020-06-26T00:06:03,754 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] ql.Driver: Semantic Analysis Completed (retrial = false) >>> >>> 2020-06-26T00:06:03,762 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, >>>> properties:null) >>> >>> 2020-06-26T00:06:03,763 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] ql.Driver: Completed compiling >>>> command(queryId=hdfs_20200626000603_0992e79f-6e1c-4383-be62-a6466c4c1cf2); >>>> Time taken: 0.477 seconds >>> >>> 2020-06-26T00:06:03,763 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] reexec.ReExecDriver: Execution #1 of query >>> >>> 2020-06-26T00:06:03,763 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] ql.Driver: Concurrency mode is disabled, not creating a lock manager >>> >>> 2020-06-26T00:06:03,763 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] ql.Driver: Executing >>>> command(queryId=hdfs_20200626000603_0992e79f-6e1c-4383-be62-a6466c4c1cf2): >>>> CREATE EXTERNAL TABLE ed_avro_1 >>> >>> STORED AS AVRO >>> >>> LOCATION '/tmp/event_detail/yyyymmdd=20200206' >>> >>> TBLPROPERTIES ('avro.schema.url'='http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing') >>> >>> 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] ql.Driver: Starting task [Stage-0:DDL] in serial mode >>> >>> 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.HiveMetaStoreClient: Mestastore configuration >>>> metastore.filter.hook changed from >>>> org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook >>>> to org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl >>> >>> 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.HiveMetaStore: 0: Cleaning up thread local RawStore... >>> >>> 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=Cleaning up >>>> thread local RawStore... >>> >>> 2020-06-26T00:06:03,765 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metastore.HiveMetaStore: 0: Done cleaning up thread local RawStore >>> >>> 2020-06-26T00:06:03,766 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] HiveMetaStore.audit: ugi=hdfs ip=unknown-ip-addr cmd=Done >>>> cleaning up thread local RawStore >>> >>> 2020-06-26T00:06:03,790 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] avro.AvroSerDe: AvroSerde::initialize(): Preset value of >>>> avro.schema.literal == null >>> >>> 2020-06-26T00:06:03,809 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] avro.AvroSerDe: Encountered AvroSerdeException determining schema. >>>> Returning signal schema to indicate problem >>> >>> org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read >>>> schema from given path: http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:146) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:197) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:111) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:84) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:663) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:646) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:898) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> ~[?:1.8.0_252] >>> >>> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] >>> >>> at org.apache.hadoop.util.RunJar.run(RunJar.java:318) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:232) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> Caused by: java.io.FileNotFoundException: http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing >>> >>> at >>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) >>>> ~[?:1.8.0_252] >>> >>> at >>>> org.apache.hadoop.fs.http.AbstractHttpFileSystem.open(AbstractHttpFileSystem.java:61) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at org.apache.hadoop.fs.http.HttpFileSystem.open(HttpFileSystem.java:23) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:169) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> ... 35 more >>> >>> 2020-06-26T00:06:03,871 ERROR [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] hive.log: error in initSerDe: >>>> org.apache.hadoop.hive.serde2.SerDeException Encountered AvroSerdeException >>>> determining schema. Returning signal schema to indicate problem: Unable to >>>> read schema from given path: http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing >>> >>> org.apache.hadoop.hive.serde2.SerDeException: Encountered >>>> AvroSerdeException determining schema. Returning signal schema to indicate >>>> problem: Unable to read schema from given path: http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing >>> >>> at >>>> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:543) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:663) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:646) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:898) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> ~[?:1.8.0_252] >>> >>> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] >>> >>> at org.apache.hadoop.util.RunJar.run(RunJar.java:318) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:232) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> 2020-06-26T00:06:03,874 ERROR [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] metadata.Table: Unable to get field from serde: >>>> org.apache.hadoop.hive.serde2.avro.AvroSerDe >>> >>> java.lang.RuntimeException: >>>> MetaException(message:org.apache.hadoop.hive.serde2.SerDeException >>>> Encountered AvroSerdeException determining schema. Returning signal schema >>>> to indicate problem: Unable to read schema from given path: http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing) >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:291) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getColsInternal(Table.java:663) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:646) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:898) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> ~[?:1.8.0_252] >>> >>> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] >>> >>> at org.apache.hadoop.util.RunJar.run(RunJar.java:318) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:232) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: >>>> org.apache.hadoop.hive.serde2.SerDeException Encountered AvroSerdeException >>>> determining schema. Returning signal schema to indicate problem: Unable to >>>> read schema from given path: http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing >>> >>> at >>>> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:98) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> ... 28 more >>> >>> 2020-06-26T00:06:03,874 INFO [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] avro.AvroSerDe: AvroSerde::initialize(): Preset value of >>>> avro.schema.literal == null >>> >>> 2020-06-26T00:06:03,878 WARN [646da35b-84b0-43aa-9b68-5d668ebbfc36 >>>> main] avro.AvroSerDe: Encountered AvroSerdeException determining schema. >>>> Returning signal schema to indicate problem >>> >>> org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read >>>> schema from given path: http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:146) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:197) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:111) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:84) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:540) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:90) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:271) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:900) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:937) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:4954) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:428) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:218) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:683) >>>> ~[hive-cli-3.1.2.jar:3.1.2] >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>> ~[?:1.8.0_252] >>> >>> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_252] >>> >>> at org.apache.hadoop.util.RunJar.run(RunJar.java:318) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:232) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> Caused by: java.io.FileNotFoundException: http:// >>>> <api_server>:9091/schema?name=ed&store=parquet&isMutated=true&table=ed&secbypass=testing >>> >>> at >>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1896) >>>> ~[?:1.8.0_252] >>> >>> at >>>> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498) >>>> ~[?:1.8.0_252] >>> >>> at >>>> org.apache.hadoop.fs.http.AbstractHttpFileSystem.open(AbstractHttpFileSystem.java:61) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at org.apache.hadoop.fs.http.HttpFileSystem.open(HttpFileSystem.java:23) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:899) >>>> ~[hadoop-common-3.1.2.jar:?] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromFS(AvroSerdeUtils.java:169) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> at >>>> org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:139) >>>> ~[hive-exec-3.1.2.jar:3.1.2] >>> >>> ... 33 more >>> >>> Thanks, >> Ravi >> >
