[
https://issues.apache.org/jira/browse/IMPALA-10186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
guojingfeng updated IMPALA-10186:
---------------------------------
Description:
Current parquet writer write -1 of PageLocation.offset and
PageLocation.first_row_index when meet a empty page.
hdfs-parquet-file-writer.cc Line: 808 ~ 819
{code:java}
// Write data pages
for (const DataPage& page : pages_) {
if (page.header.data_page_header.num_values == 0) {
// Skip empty pages
location.offset = -1;
location.compressed_page_size = 0;
location.first_row_index = -1;
AddLocationToOffsetIndex(location);
continue;
}
{code}
But -1 values may cause ComputeCandidatePages function run into unexpected
status.
{code:java}
bool ComputeCandidatePages(
const vector<parquet::PageLocation>& page_locations,
const vector<RowRange>& candidate_ranges,
const int64_t num_rows, vector<int>* candidate_pages) {
if (!ValidatePageLocations(page_locations, num_rows)) return false
{code}
and then cause IMPALA-9952
was:
Current parquet writer write -1 of PageLocation.offset and
PageLocation.first_row_index when meet a empty page.
hdfs-parquet-file-writer.cc Line: 808 ~ 819
{code:java}
// Write data pages
for (const DataPage& page : pages_) {
parquet::PageLocation location; if
(page.header.data_page_header.num_values == 0) {
// Skip empty pages
location.offset = -1;
location.compressed_page_size = 0;
location.first_row_index = -1;
AddLocationToOffsetIndex(location);
continue;
}
{code}
But -1 values may cause ComputeCandidatePages function run into unexpected
status.
{code:java}
bool ComputeCandidatePages(
const vector<parquet::PageLocation>& page_locations,
const vector<RowRange>& candidate_ranges,
const int64_t num_rows, vector<int>* candidate_pages) {
if (!ValidatePageLocations(page_locations, num_rows)) return false
{code}
and then cause IMPALA-9952
> Write invalid parquet PageLocations which table sort by some columns
> --------------------------------------------------------------------
>
> Key: IMPALA-10186
> URL: https://issues.apache.org/jira/browse/IMPALA-10186
> Project: IMPALA
> Issue Type: Bug
> Reporter: guojingfeng
> Priority: Major
>
> Current parquet writer write -1 of PageLocation.offset and
> PageLocation.first_row_index when meet a empty page.
> hdfs-parquet-file-writer.cc Line: 808 ~ 819
> {code:java}
> // Write data pages
> for (const DataPage& page : pages_) {
> if (page.header.data_page_header.num_values == 0) {
> // Skip empty pages
> location.offset = -1;
> location.compressed_page_size = 0;
> location.first_row_index = -1;
> AddLocationToOffsetIndex(location);
> continue;
> }
> {code}
> But -1 values may cause ComputeCandidatePages function run into unexpected
> status.
> {code:java}
> bool ComputeCandidatePages(
> const vector<parquet::PageLocation>& page_locations,
> const vector<RowRange>& candidate_ranges,
> const int64_t num_rows, vector<int>* candidate_pages) {
> if (!ValidatePageLocations(page_locations, num_rows)) return false
> {code}
> and then cause IMPALA-9952
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
