But doing so I assume it does not detect the columns on it's own, I have to specify such manually - or am I wrong? The orc file I finally want to work with contains ~28000 columns (513MB size, ~500000 rows, 3 structs with 2 of them containing ~14000 fields each) ...
The hive documentation for the create table statement shows the columns part being optional. In fact it seems required, at least I found no way to avoid it. For testing purposes I started with a smaller one and found two ways of bringing the data to hive. Unfortunately I actually fail on accessing it: a) create external table: Succeeding statement: create external table if not exists CFA1_Fan_Speed_DMC(record struct<normalizedTime:bigint,RAW:bigint,ENG:float>) stored as ORC location '...'; with the location having specified containig my existing orc file named exactly like the table, CFA1_Fan_Speed_DMC. But every selection for data results in: Error: java.io.IOException: java.lang.RuntimeException: Char length 256 out of allowed range [1, 255] (state=,code=0) Tried with: - select * from CFA1_Fan_Speed_DMC; - select record from CFA1_Fan_Speed_DMC; - select record.normalizedTime from CFA1_Fan_Speed_DMC; b) create table and load from file Succeeding statements: create table cfa1(record struct<normalizedTime:bigint,RAW:bigint,ENG:float>) stored as orc; load data inpath '.../CFA1_Fan_Speed_DMC' into table cfa1; Same statements for querying as above (of course using the different table name) still fail, but now with: Error: java.io.IOException: java.io.IOException: ORC does not support type conversion from file type bigint (1) to reader type struct<normalizedTime:bigint,RAW:bigint,ENG:float> (1) (state=,code=0) So what is wrong with the above? I should mention, that I created the orc files having used using the latest orc-core lib (1.1.2). That seems not to be the same implementation for orc files access as being used in hive. Thanks for all hints! Am Mittwoch, 3. August 2016, 08:45:45 CEST schrieb Marcin Tustin: > Yes. Create an external table whose location contains only the orc file(s) > you want to include in the table. > > On Wed, Aug 3, 2016 at 7:53 AM, Johannes Stamminger < > > johannes.stammin...@airbus.com> wrote: > > Hi, > > > > > > is it possible to write data to an orc file(s) using the hive-orc api and > > to > > use such by hive (create a table from it)? > > > > > > Regards > > This email (including any attachments) may contain confidential and/or > > privileged information or information otherwise protected from disclosure. > > If you are not the intended recipient, please notify the sender > > immediately, do not copy this message or any attachments and do not use it > > for any purpose or disclose its content to any person, but delete this > > message and any attachments from your system. Astrium and Airbus Group > > companies disclaim any and all liability if this email transmission was > > virus corrupted, altered or falsified. > > --------------------------------------------------------- > > Airbus DS GmbH > > Vorsitzender des Aufsichtsrates: Bernhard Gerwert > > Geschäftsführung: Evert Dudok (Vorsitzender), Dr. Lars Immisch, Dr. > > Michael Menking, Dr. Johannes von Thadden > > Sitz der Gesellschaft: München - Registergericht: Amtsgericht München, HRB > > Nr. 107 647 > > Ust. Ident. Nr. /VAT reg. no. DE167015356 -- johannes.stammin...@airbus.com [2FE783D0 http://wwwkeys.PGP.net] ------ ----<--{(@ ------------------ AIRBUS Defence & Space Koenigsberger Str. 17, 28857 Barrien Ground SW Eng. & Del. (TSOTC 6) +49 4242 169582 (Tel + FAX) Airbus Allee 1, 28199 Bremen +49 174 7731593 (Mobile) +49 421 539 4152 (Tel) / 4378 (FAX) This email (including any attachments) may contain confidential and/or privileged information or information otherwise protected from disclosure. If you are not the intended recipient, please notify the sender immediately, do not copy this message or any attachments and do not use it for any purpose or disclose its content to any person, but delete this message and any attachments from your system. Astrium and Airbus Group companies disclaim any and all liability if this email transmission was virus corrupted, altered or falsified. --------------------------------------------------------- Airbus DS GmbH Vorsitzender des Aufsichtsrates: Bernhard Gerwert Geschäftsführung: Evert Dudok (Vorsitzender), Dr. Lars Immisch, Dr. Michael Menking, Dr. Johannes von Thadden Sitz der Gesellschaft: München - Registergericht: Amtsgericht München, HRB Nr. 107 647 Ust. Ident. Nr. /VAT reg. no. DE167015356
