I disagree with Wes. He's right that you *could* just use binary and keep
extra metadata somewhere, it is very unlikely that Parquet would ever
support such a scheme. And it is bad for the community when people attempt
to go around the format spec, as we see with the INT96 timestamp mess.

I think the right thing is to transcode your data to UTF-8. That should not
be at all hard in any language, will create standards-compliant Parquet
that can be used in any processing engine, and you'll save space when it is
stored. UTF-16 to UTF-8 is not lossy, so it should be fine. The fact that
you don't control the input data encoding doesn't mean you can't write it
to Parquet correctly.

rb

On Wed, Feb 6, 2019 at 6:39 AM Wes McKinney <[email protected]> wrote:

> hi Manik -- you are free to store the data as opaque binary as
> BYTE_ARRAY type and add some metadata to the schema so your readers
> can recognize that it's UTF-16 stored as binary
>
> On Wed, Feb 6, 2019 at 12:24 AM Manik Singla <[email protected]> wrote:
> >
> > I am not the producer of data so I can not control encoding. I do receive
> > ByteBuffer and encoding.
> > I can decode data with given encoding and covert to UTF-8 for storing
> with
> > Parquet.
> > I was thinking to remove that overhead if possible
> >
> > Regards
> > Manik Singla
> > +91-9996008893
> > +91-9665639677
> >
> > "Life doesn't consist in holding good cards but playing those you hold
> > well."
> >
> >
> > On Wed, Feb 6, 2019 at 11:37 AM Uwe L. Korn <[email protected]> wrote:
> >
> > > Hello Manik,
> > >
> > > this is not possible at the moment. As Parquet is a portable on-disk
> > > format, we focus on having a single representation for each data type.
> Thus
> > > implementing readers/writers is limited to these to make their
> > > implementation simpler. Especially as you are the producer but not the
> > > consumer, even adding a new type would not solve your problem. You
> really
> > > can only use a new logical type when it has been implemented in all the
> > > readers and your consumers have all updated to these reader versions.
> > >
> > > As Unicode and thus UTF-8 support all characters one can think off, you
> > > should always be able to convert strings to it. Given that Parquet
> files
> > > encode and compress the data anyway afterwards, the conversion is a
> bit of
> > > a CPU overhead but should not make a difference in size and form of the
> > > data actually stored in the files. Also I guess that the UTF-16->UTF-8
> > > conversion costs less CPU that the Parquet compression process.
> > >
> > > Did this help you or is there any reason why you really cannot convert
> > > your data to UTF-8?
> > >
> > > Uwe
> > >
> > > On Wed, Feb 6, 2019, at 6:19 AM, Manik Singla wrote:
> > > > Hi
> > > >
> > > > I am new to Parquet. I am trying to save UTF-16 or some other
> encoding
> > > than
> > > > UTF-8.
> > > > I am also trying to use encoding hint when saving ByteBuffer.
> > > >
> > > > I don't find way to use any thing other than UTF-8.
> > > > https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
> > > says
> > > > we can extend primitive types to solve cases.
> > > >
> > > > Other thing I want to mention is I am only the producer of parquet
> file
> > > but
> > > > not consumer.
> > > >
> > > > Could you guide me which examples I can look into or which will be
> right
> > > way
> > > >
> > > >
> > > > Regards
> > > > Manik Singla
> > > > +91-9996008893
> > > > +91-9665639677
> > > >
> > > > "Life doesn't consist in holding good cards but playing those you
> hold
> > > > well."
> > >
>


-- 
Ryan Blue
Software Engineer
Netflix

Reply via email to