To answer my own question, the Hive Bridge needs visibility to the full
hadoop class path to successfully interact with HDFS. I¹ll create a JIRA
and provide a patch for further feedback.
On 12/22/15, 1:16 PM, "Aaron.Dossett" <aaron.doss...@target.com> wrote:
>I was researching my ATLAS-381 errors and came across possibly a more
>fundamental problem. My avro tables define their schema by reference to
>a schema file on HDFS. The DDL includes:
>
>TBLPROPERTIES
>('avro.schema.url'='hdfs://redstackambari.vagrant.tgt:8020/user/vagrant/sc
>hema/foo.avsc');
>
>When the bridge tries to read that schema I get the error below. The
>fact that AvroSerdeUtils is calling java.net.URL() with the hdfs://...
>string means that it could find load the schema directly:
>
>https://github.com/apache/hive/blob/branch-1.2/serde/src/java/org/apache/h
>adoop/hive/serde2/avro/AvroSerdeUtils.java#L115
>
>From past experiences, I would guess that the attempt to get the HDFS
>file system here isn't working due to kerberos authentication issues:
>https://github.com/apache/hive/blob/branch-1.2/serde/src/java/org/apache/h
>adoop/hive/serde2/avro/AvroSerdeUtils.java#L131
>
>I am trying to debug this further, but does anyone have other thoughts on
>why I may be getting this error?
>
>Thanks! -Aaron
>
>
>org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read
>schema from given path:
>hdfs://redstackambari.vagrant.tgt:8020/user/vagrant/schema/foo.avsc
> at
>org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowEx
>ception(AvroSerdeUtils.java:119)
> at
>org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorS
>chema(AvroSerDe.java:167)
> at
>org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:103
>)
> at
>org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:80)
> at
>org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:5
>21)
> at
>org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreU
>tils.java:391)
> at
>org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Tabl
>e.java:276)
> at
>org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258)
> at
>org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605)
> at
>org.apache.atlas.hive.bridge.HiveMetaStoreBridge.createTableInstance(HiveM
>etaStoreBridge.java:271)
> at
>org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerTable(HiveMetaSto
>reBridge.java:302)
> at
>org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStor
>eBridge.java:217)
> at
>org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaS
>toreBridge.java:105)
> at
>org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMe
>taStoreBridge.java:97)
> at
>org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.
>java:508)
>Caused by: java.net.MalformedURLException: unknown protocol: hdfs
> at java.net.URL.<init>(URL.java:592)
> at java.net.URL.<init>(URL.java:482)
> at java.net.URL.<init>(URL.java:431)
> at
>org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowEx
>ception(AvroSerdeUtils.java:115)
> ... 14 more
>
>
