[
https://issues.apache.org/jira/browse/PARQUET-2378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787046#comment-17787046
]
Jiashen Zhang edited comment on PARQUET-2378 at 11/17/23 6:49 AM:
------------------------------------------------------------------
This *Cat* supports multiple file format - Parquet, Avro, Text, it internally
needs to converts MessageType to Avro schema. Another solution we could do is
to add a new command to cat parquet file format without converting to Avro
Schema and we directly print content given a parquet file? Below is some code
sample:
{code:java}
String input = <parquet file>;
ParquetReader<SimpleRecord> reader = null;
try {
PrintWriter writer = new PrintWriter(Main.out, true);
reader = ParquetReader.builder(new SimpleReadSupport(), new
Path(input)).build();
ParquetMetadata metadata = ParquetFileReader.readFooter(new
Configuration(), new Path(input));
JsonRecordFormatter.JsonGroupFormatter formatter =
JsonRecordFormatter.fromSchema(metadata.getFileMetaData().getSchema());
for (SimpleRecord value = reader.read(); value != null; value =
reader.read()) {
value.prettyPrint(writer);
writer.println();
}
} finally {
if (reader != null) {
try {
reader.close();
} catch (Exception ex) {
}
}
}
}{code}
Output sample:
{code:java}
.......
id = 15012
category = open_qa
original-instruction = What is the difference between a road bike and a
mountain bike?
original-context =
original-response = Road bikes are built to be ridden on asphalt and cement
surfaces and have thin tires, whereas mountain bikes are built to be ridden on
dirt and have wider tires. Road bikes also have more aerodynamic handle bars
while mountain bike handle bars a built for less responsive steering while
bouncing around off the road.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = What is the difference between a road bike and a mountain bike?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = Road bikes are built to be ridden on asphalt and cement surfaces and
have thin tires, whereas mountain bikes are built to be ridden on dirt and have
wider tires. Road bikes also have more aerodynamic handle bars while mountain
bike handle bars a built for less responsive steering while bouncing around off
the road.
.status:
..list:
...item = submitted
id = 15013
category = general_qa
original-instruction = How does GIS help in the real estate investment industry?
original-context =
original-response = Real estate investors depend on precise, accurate location
intelligence for competitive insights about the markets and locations where
they do business. Real estate investment teams use GIS to bring together
location-specific data, mapping, and visualization technology. This enables
them to provide the latest insights about real estate markets and their
investments, now and in the future. Using thousands of global datasets,
investors can quickly understand how their real estate investments are
performing across town or around the world, quickly access precise local data
about real estate assets, on any device, anywhere, anytime, including
information on occupancy, building maintenance, property valuation, and
more.Real estate companies and investors use GIS to research markets, identify
new opportunities for growth and expansion, and manage their investments at the
market and neighborhood levels. They can also use GIS to create professional
digital and printed materials—such as 3D renderings and virtual
walk-throughs—to help market investments across platforms. Real estate
investors can use mobile data collection tools to gather property information
directly from the field and analyze and share insights across their
organizations in real time. Investors can leverage precise local knowledge
about their assets across geographies. GIS maps and dashboards help investors
see, in real-time, relevant data that can affect properties, and streamline
investment management with access to all relevant data about every asset in any
portfolio.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = How does GIS help in the real estate investment industry?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = Real estate investors depend on precise, accurate location
intelligence for competitive insights about the markets and locations where
they do business. Real estate investment teams use GIS to bring together
location-specific data, mapping, and visualization technology. This enables
them to provide the latest insights about real estate markets and their
investments, now and in the future. Using thousands of global datasets,
investors can quickly understand how their real estate investments are
performing across town or around the world, quickly access precise local data
about real estate assets, on any device, anywhere, anytime, including
information on occupancy, building maintenance, property valuation, and
more.Real estate companies and investors use GIS to research markets, identify
new opportunities for growth and expansion, and manage their investments at the
market and neighborhood levels. They can also use GIS to create professional
digital and printed materials—such as 3D renderings and virtual
walk-throughs—to help market investments across platforms. Real estate
investors can use mobile data collection tools to gather property information
directly from the field and analyze and share insights across their
organizations in real time. Investors can leverage precise local knowledge
about their assets across geographies. GIS maps and dashboards help investors
see, in real-time, relevant data that can affect properties, and streamline
investment management with access to all relevant data about every asset in any
portfolio.
.status:
..list:
...item = submitted
id = 15014
category = general_qa
original-instruction = What is the Masters?
original-context =
original-response = The Masters Tournament is a golf tournament held annually
in the first week of April at Augusta National Golf Club in Augusta, Georgia.
The Masters is one of four Major golf tournaments and the only one to be played
at the same course every year. The course is renowned for its iconic holes,
impeccable groundskeeping, and colorful flowers that are typically in bloom.
The winner earns a coveted Green Jacket and a lifetime invitation back to
compete. Many players and fans consider The Masters to be their favorite
tournament given these traditions and the historical moments that have occurred
in past tournaments.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = What is the Masters?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = The Masters Tournament is a golf tournament held annually in the
first week of April at Augusta National Golf Club in Augusta, Georgia. The
Masters is one of four Major golf tournaments and the only one to be played at
the same course every year. The course is renowned for its iconic holes,
impeccable groundskeeping, and colorful flowers that are typically in bloom.
The winner earns a coveted Green Jacket and a lifetime invitation back to
compete. Many players and fans consider The Masters to be their favorite
tournament given these traditions and the historical moments that have occurred
in past tournaments.
.status:
..list:
...item = submitted{code}
Or Print with Json format, such as:
{code:java}
{"id":"15012","category":"open_qa","original-instruction":"What is the
difference between a road bike and a mountain
bike?","original-context":"","original-response":"Road bikes are built to be
ridden on asphalt and cement surfaces and have thin tires, whereas mountain
bikes are built to be ridden on dirt and have wider tires. Road bikes also have
more aerodynamic handle bars while mountain bike handle bars a built for less
responsive steering while bouncing around off the
road.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"What
is the difference between a road bike and a mountain
bike?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"Road
bikes are built to be ridden on asphalt and cement surfaces and have thin
tires, whereas mountain bikes are built to be ridden on dirt and have wider
tires. Road bikes also have more aerodynamic handle bars while mountain bike
handle bars a built for less responsive steering while bouncing around off the
road."}]},"status":{"list":[{"item":"submitted"}]}}}
{"id":"15013","category":"general_qa","original-instruction":"How does GIS help
in the real estate investment
industry?","original-context":"","original-response":"Real estate investors
depend on precise, accurate location intelligence for competitive insights
about the markets and locations where they do business. Real estate investment
teams use GIS to bring together location-specific data, mapping, and
visualization technology. This enables them to provide the latest insights
about real estate markets and their investments, now and in the future. Using
thousands of global datasets, investors can quickly understand how their real
estate investments are performing across town or around the world, quickly
access precise local data about real estate assets, on any device, anywhere,
anytime, including information on occupancy, building maintenance, property
valuation, and more.\n\nReal estate companies and investors use GIS to research
markets, identify new opportunities for growth and expansion, and manage their
investments at the market and neighborhood levels. They can also use GIS to
create professional digital and printed materials—such as 3D renderings and
virtual walk-throughs—to help market investments across platforms. Real estate
investors can use mobile data collection tools to gather property information
directly from the field and analyze and share insights across their
organizations in real time. Investors can leverage precise local knowledge
about their assets across geographies. GIS maps and dashboards help investors
see, in real-time, relevant data that can affect properties, and streamline
investment management with access to all relevant data about every asset in any
portfolio.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"How
does GIS help in the real estate investment
industry?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"Real
estate investors depend on precise, accurate location intelligence for
competitive insights about the markets and locations where they do business.
Real estate investment teams use GIS to bring together location-specific data,
mapping, and visualization technology. This enables them to provide the latest
insights about real estate markets and their investments, now and in the
future. Using thousands of global datasets, investors can quickly understand
how their real estate investments are performing across town or around the
world, quickly access precise local data about real estate assets, on any
device, anywhere, anytime, including information on occupancy, building
maintenance, property valuation, and more.\n\nReal estate companies and
investors use GIS to research markets, identify new opportunities for growth
and expansion, and manage their investments at the market and neighborhood
levels. They can also use GIS to create professional digital and printed
materials—such as 3D renderings and virtual walk-throughs—to help market
investments across platforms. Real estate investors can use mobile data
collection tools to gather property information directly from the field and
analyze and share insights across their organizations in real time. Investors
can leverage precise local knowledge about their assets across geographies. GIS
maps and dashboards help investors see, in real-time, relevant data that can
affect properties, and streamline investment management with access to all
relevant data about every asset in any
portfolio."}]},"status":{"list":[{"item":"submitted"}]}}}
{"id":"15014","category":"general_qa","original-instruction":"What is the
Masters?","original-context":"","original-response":"The Masters Tournament is
a golf tournament held annually in the first week of April at Augusta National
Golf Club in Augusta, Georgia. The Masters is one of four Major golf
tournaments and the only one to be played at the same course every year. The
course is renowned for its iconic holes, impeccable groundskeeping, and
colorful flowers that are typically in bloom. The winner earns a coveted Green
Jacket and a lifetime invitation back to compete. Many players and fans
consider The Masters to be their favorite tournament given these traditions and
the historical moments that have occurred in past
tournaments.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"What
is the
Masters?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"The
Masters Tournament is a golf tournament held annually in the first week of
April at Augusta National Golf Club in Augusta, Georgia. The Masters is one of
four Major golf tournaments and the only one to be played at the same course
every year. The course is renowned for its iconic holes, impeccable
groundskeeping, and colorful flowers that are typically in bloom. The winner
earns a coveted Green Jacket and a lifetime invitation back to compete. Many
players and fans consider The Masters to be their favorite tournament given
these traditions and the historical moments that have occurred in past
tournaments."}]},"status":{"list":[{"item":"submitted"}]}}} {code}
was (Author: JIRAUSER280855):
What about we directly print content given a parquet file? Below is some code
sample:
{code:java}
String input = <parquet file>;
ParquetReader<SimpleRecord> reader = null;
try {
PrintWriter writer = new PrintWriter(Main.out, true);
reader = ParquetReader.builder(new SimpleReadSupport(), new
Path(input)).build();
ParquetMetadata metadata = ParquetFileReader.readFooter(new
Configuration(), new Path(input));
JsonRecordFormatter.JsonGroupFormatter formatter =
JsonRecordFormatter.fromSchema(metadata.getFileMetaData().getSchema());
for (SimpleRecord value = reader.read(); value != null; value =
reader.read()) {
value.prettyPrint(writer);
writer.println();
}
} finally {
if (reader != null) {
try {
reader.close();
} catch (Exception ex) {
}
}
}
}{code}
Output sample:
{code:java}
.......
id = 15012
category = open_qa
original-instruction = What is the difference between a road bike and a
mountain bike?
original-context =
original-response = Road bikes are built to be ridden on asphalt and cement
surfaces and have thin tires, whereas mountain bikes are built to be ridden on
dirt and have wider tires. Road bikes also have more aerodynamic handle bars
while mountain bike handle bars a built for less responsive steering while
bouncing around off the road.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = What is the difference between a road bike and a mountain bike?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = Road bikes are built to be ridden on asphalt and cement surfaces and
have thin tires, whereas mountain bikes are built to be ridden on dirt and have
wider tires. Road bikes also have more aerodynamic handle bars while mountain
bike handle bars a built for less responsive steering while bouncing around off
the road.
.status:
..list:
...item = submitted
id = 15013
category = general_qa
original-instruction = How does GIS help in the real estate investment industry?
original-context =
original-response = Real estate investors depend on precise, accurate location
intelligence for competitive insights about the markets and locations where
they do business. Real estate investment teams use GIS to bring together
location-specific data, mapping, and visualization technology. This enables
them to provide the latest insights about real estate markets and their
investments, now and in the future. Using thousands of global datasets,
investors can quickly understand how their real estate investments are
performing across town or around the world, quickly access precise local data
about real estate assets, on any device, anywhere, anytime, including
information on occupancy, building maintenance, property valuation, and
more.Real estate companies and investors use GIS to research markets, identify
new opportunities for growth and expansion, and manage their investments at the
market and neighborhood levels. They can also use GIS to create professional
digital and printed materials—such as 3D renderings and virtual
walk-throughs—to help market investments across platforms. Real estate
investors can use mobile data collection tools to gather property information
directly from the field and analyze and share insights across their
organizations in real time. Investors can leverage precise local knowledge
about their assets across geographies. GIS maps and dashboards help investors
see, in real-time, relevant data that can affect properties, and streamline
investment management with access to all relevant data about every asset in any
portfolio.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = How does GIS help in the real estate investment industry?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = Real estate investors depend on precise, accurate location
intelligence for competitive insights about the markets and locations where
they do business. Real estate investment teams use GIS to bring together
location-specific data, mapping, and visualization technology. This enables
them to provide the latest insights about real estate markets and their
investments, now and in the future. Using thousands of global datasets,
investors can quickly understand how their real estate investments are
performing across town or around the world, quickly access precise local data
about real estate assets, on any device, anywhere, anytime, including
information on occupancy, building maintenance, property valuation, and
more.Real estate companies and investors use GIS to research markets, identify
new opportunities for growth and expansion, and manage their investments at the
market and neighborhood levels. They can also use GIS to create professional
digital and printed materials—such as 3D renderings and virtual
walk-throughs—to help market investments across platforms. Real estate
investors can use mobile data collection tools to gather property information
directly from the field and analyze and share insights across their
organizations in real time. Investors can leverage precise local knowledge
about their assets across geographies. GIS maps and dashboards help investors
see, in real-time, relevant data that can affect properties, and streamline
investment management with access to all relevant data about every asset in any
portfolio.
.status:
..list:
...item = submitted
id = 15014
category = general_qa
original-instruction = What is the Masters?
original-context =
original-response = The Masters Tournament is a golf tournament held annually
in the first week of April at Augusta National Golf Club in Augusta, Georgia.
The Masters is one of four Major golf tournaments and the only one to be played
at the same course every year. The course is renowned for its iconic holes,
impeccable groundskeeping, and colorful flowers that are typically in bloom.
The winner earns a coveted Green Jacket and a lifetime invitation back to
compete. Many players and fans consider The Masters to be their favorite
tournament given these traditions and the historical moments that have occurred
in past tournaments.
new-instruction:
.user_id:
..list:
.value:
..list:
...item = What is the Masters?
.status:
..list:
...item = submitted
new-context:
.user_id:
..list:
.value:
..list:
...item =
.status:
..list:
...item = submitted
new-response:
.user_id:
..list:
.value:
..list:
...item = The Masters Tournament is a golf tournament held annually in the
first week of April at Augusta National Golf Club in Augusta, Georgia. The
Masters is one of four Major golf tournaments and the only one to be played at
the same course every year. The course is renowned for its iconic holes,
impeccable groundskeeping, and colorful flowers that are typically in bloom.
The winner earns a coveted Green Jacket and a lifetime invitation back to
compete. Many players and fans consider The Masters to be their favorite
tournament given these traditions and the historical moments that have occurred
in past tournaments.
.status:
..list:
...item = submitted{code}
Or Print with Json format, such as:
{code:java}
{"id":"15012","category":"open_qa","original-instruction":"What is the
difference between a road bike and a mountain
bike?","original-context":"","original-response":"Road bikes are built to be
ridden on asphalt and cement surfaces and have thin tires, whereas mountain
bikes are built to be ridden on dirt and have wider tires. Road bikes also have
more aerodynamic handle bars while mountain bike handle bars a built for less
responsive steering while bouncing around off the
road.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"What
is the difference between a road bike and a mountain
bike?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"Road
bikes are built to be ridden on asphalt and cement surfaces and have thin
tires, whereas mountain bikes are built to be ridden on dirt and have wider
tires. Road bikes also have more aerodynamic handle bars while mountain bike
handle bars a built for less responsive steering while bouncing around off the
road."}]},"status":{"list":[{"item":"submitted"}]}}}
{"id":"15013","category":"general_qa","original-instruction":"How does GIS help
in the real estate investment
industry?","original-context":"","original-response":"Real estate investors
depend on precise, accurate location intelligence for competitive insights
about the markets and locations where they do business. Real estate investment
teams use GIS to bring together location-specific data, mapping, and
visualization technology. This enables them to provide the latest insights
about real estate markets and their investments, now and in the future. Using
thousands of global datasets, investors can quickly understand how their real
estate investments are performing across town or around the world, quickly
access precise local data about real estate assets, on any device, anywhere,
anytime, including information on occupancy, building maintenance, property
valuation, and more.\n\nReal estate companies and investors use GIS to research
markets, identify new opportunities for growth and expansion, and manage their
investments at the market and neighborhood levels. They can also use GIS to
create professional digital and printed materials—such as 3D renderings and
virtual walk-throughs—to help market investments across platforms. Real estate
investors can use mobile data collection tools to gather property information
directly from the field and analyze and share insights across their
organizations in real time. Investors can leverage precise local knowledge
about their assets across geographies. GIS maps and dashboards help investors
see, in real-time, relevant data that can affect properties, and streamline
investment management with access to all relevant data about every asset in any
portfolio.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"How
does GIS help in the real estate investment
industry?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"Real
estate investors depend on precise, accurate location intelligence for
competitive insights about the markets and locations where they do business.
Real estate investment teams use GIS to bring together location-specific data,
mapping, and visualization technology. This enables them to provide the latest
insights about real estate markets and their investments, now and in the
future. Using thousands of global datasets, investors can quickly understand
how their real estate investments are performing across town or around the
world, quickly access precise local data about real estate assets, on any
device, anywhere, anytime, including information on occupancy, building
maintenance, property valuation, and more.\n\nReal estate companies and
investors use GIS to research markets, identify new opportunities for growth
and expansion, and manage their investments at the market and neighborhood
levels. They can also use GIS to create professional digital and printed
materials—such as 3D renderings and virtual walk-throughs—to help market
investments across platforms. Real estate investors can use mobile data
collection tools to gather property information directly from the field and
analyze and share insights across their organizations in real time. Investors
can leverage precise local knowledge about their assets across geographies. GIS
maps and dashboards help investors see, in real-time, relevant data that can
affect properties, and streamline investment management with access to all
relevant data about every asset in any
portfolio."}]},"status":{"list":[{"item":"submitted"}]}}}
{"id":"15014","category":"general_qa","original-instruction":"What is the
Masters?","original-context":"","original-response":"The Masters Tournament is
a golf tournament held annually in the first week of April at Augusta National
Golf Club in Augusta, Georgia. The Masters is one of four Major golf
tournaments and the only one to be played at the same course every year. The
course is renowned for its iconic holes, impeccable groundskeeping, and
colorful flowers that are typically in bloom. The winner earns a coveted Green
Jacket and a lifetime invitation back to compete. Many players and fans
consider The Masters to be their favorite tournament given these traditions and
the historical moments that have occurred in past
tournaments.","new-instruction":{"user_id":{"list":[{}]},"value":{"list":[{"item":"What
is the
Masters?"}]},"status":{"list":[{"item":"submitted"}]}},"new-context":{"user_id":{"list":[{}]},"value":{"list":[{"item":""}]},"status":{"list":[{"item":"submitted"}]}},"new-response":{"user_id":{"list":[{}]},"value":{"list":[{"item":"The
Masters Tournament is a golf tournament held annually in the first week of
April at Augusta National Golf Club in Augusta, Georgia. The Masters is one of
four Major golf tournaments and the only one to be played at the same course
every year. The course is renowned for its iconic holes, impeccable
groundskeeping, and colorful flowers that are typically in bloom. The winner
earns a coveted Green Jacket and a lifetime invitation back to compete. Many
players and fans consider The Masters to be their favorite tournament given
these traditions and the historical moments that have occurred in past
tournaments."}]},"status":{"list":[{"item":"submitted"}]}}} {code}
> Problem with a cat
> ------------------
>
> Key: PARQUET-2378
> URL: https://issues.apache.org/jira/browse/PARQUET-2378
> Project: Parquet
> Issue Type: Bug
> Reporter: Rémy Léone
> Priority: Major
> Attachments: image-2023-11-16-21-40-07-628.png
>
>
> *$* parquet cat train-00000-of-00001-15a05aeec7726f9d.parquet
>
> Unknown error
> shaded.parquet.org.apache.avro.SchemaParseException: Illegal character in:
> original-instruction
> at shaded.parquet.org.apache.avro.Schema.validateName(Schema.java:1607)
> at shaded.parquet.org.apache.avro.Schema.access$400(Schema.java:92)
> at shaded.parquet.org.apache.avro.Schema$Field.<init>(Schema.java:556)
> at shaded.parquet.org.apache.avro.Schema$Field.<init>(Schema.java:595)
> at
> org.apache.parquet.avro.AvroSchemaConverter.convertFields(AvroSchemaConverter.java:295)
> at
> org.apache.parquet.avro.AvroSchemaConverter.convert(AvroSchemaConverter.java:279)
> at org.apache.parquet.cli.util.Schemas.fromParquet(Schemas.java:89)
> at org.apache.parquet.cli.BaseCommand.getAvroSchema(BaseCommand.java:405)
> at org.apache.parquet.cli.commands.CatCommand.run(CatCommand.java:66)
> at org.apache.parquet.cli.Main.run(Main.java:163)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.parquet.cli.Main.main(Main.java:193)
> the data set in question is:
> [https://huggingface.co/datasets/argilla/databricks-dolly-15k-curated-en/tree/main/data]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
