[jira] [Commented] (ARROW-16692) [C++] Segfault in datasets
[
https://issues.apache.org/jira/browse/ARROW-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545121#comment-17545121
]
Antoine Pitrou commented on ARROW-16692:
At worse, you could limit the max recursion level to eg. 2000 and raise an
error when it's reached. A nice error message is always more helpful than a
hard crash.
> [C++] Segfault in datasets
> --
>
> Key: ARROW-16692
> URL: https://issues.apache.org/jira/browse/ARROW-16692
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Jonathan Keane
>Assignee: Weston Pace
>Priority: Major
> Attachments: backtrace.txt
>
>
> I'm still working to make a minimal reproducer for this, though I can
> reliably reproduce it below (though that means needing to download a bunch of
> data first...). I've cleaned out much of the unnecessary code (so this query
> below is a bit silly, and not what I'm actually trying to do), but haven't
> been able to make a constructed dataset that reproduces this.
> Working on some example with the new | more cleaned taxi dataset at
> {{s3://ursa-labs-taxi-data-v2}}, I've run into a segfault:
> {code}
> library(arrow)
> library(dplyr)
> ds <- open_dataset("path/to/new_taxi/")
> ds %>%
> filter(!is.na(pickup_location_id)) %>%
> summarise(n = n()) %>% collect()
> {code}
> Most of the time ends in a segfault (though I have gotten it to work on
> occasion). I've tried with smaller files | constructed datasets and haven't
> been able to replicate it yet. One thing that might be important is:
> {{pickup_location_id}} is all NAs | nulls in the first 8 years of the data or
> so.
> I've attached a backtrace in case that's enough to see what's going on here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (ARROW-16692) [C++] Segfault in datasets
[
https://issues.apache.org/jira/browse/ARROW-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545120#comment-17545120
]
Weston Pace commented on ARROW-16692:
-
ARROW-16072 will likely not be merged until (maybe) the end of June.
I'm not sure I could do a short term fix to the merged generator before the end
of next week but I can try and take a look on Friday.
> [C++] Segfault in datasets
> --
>
> Key: ARROW-16692
> URL: https://issues.apache.org/jira/browse/ARROW-16692
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Jonathan Keane
>Assignee: Weston Pace
>Priority: Major
> Attachments: backtrace.txt
>
>
> I'm still working to make a minimal reproducer for this, though I can
> reliably reproduce it below (though that means needing to download a bunch of
> data first...). I've cleaned out much of the unnecessary code (so this query
> below is a bit silly, and not what I'm actually trying to do), but haven't
> been able to make a constructed dataset that reproduces this.
> Working on some example with the new | more cleaned taxi dataset at
> {{s3://ursa-labs-taxi-data-v2}}, I've run into a segfault:
> {code}
> library(arrow)
> library(dplyr)
> ds <- open_dataset("path/to/new_taxi/")
> ds %>%
> filter(!is.na(pickup_location_id)) %>%
> summarise(n = n()) %>% collect()
> {code}
> Most of the time ends in a segfault (though I have gotten it to work on
> occasion). I've tried with smaller files | constructed datasets and haven't
> been able to replicate it yet. One thing that might be important is:
> {{pickup_location_id}} is all NAs | nulls in the first 8 years of the data or
> so.
> I've attached a backtrace in case that's enough to see what's going on here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (ARROW-16692) [C++] Segfault in datasets
[
https://issues.apache.org/jira/browse/ARROW-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544909#comment-17544909
]
Jonathan Keane commented on ARROW-16692:
Thanks! Is there a rough timeline for when that work might be done?
I came across this prepping some demos for a talk next week — the queries do
_sometimes_ complete (and tend to complete more reliable with the bigger
queries). But I might need to change what queries I show if we don't think this
will be done in the near term.
> [C++] Segfault in datasets
> --
>
> Key: ARROW-16692
> URL: https://issues.apache.org/jira/browse/ARROW-16692
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Jonathan Keane
>Assignee: Weston Pace
>Priority: Major
> Attachments: backtrace.txt
>
>
> I'm still working to make a minimal reproducer for this, though I can
> reliably reproduce it below (though that means needing to download a bunch of
> data first...). I've cleaned out much of the unnecessary code (so this query
> below is a bit silly, and not what I'm actually trying to do), but haven't
> been able to make a constructed dataset that reproduces this.
> Working on some example with the new | more cleaned taxi dataset at
> {{s3://ursa-labs-taxi-data-v2}}, I've run into a segfault:
> {code}
> library(arrow)
> library(dplyr)
> ds <- open_dataset("path/to/new_taxi/")
> ds %>%
> filter(!is.na(pickup_location_id)) %>%
> summarise(n = n()) %>% collect()
> {code}
> Most of the time ends in a segfault (though I have gotten it to work on
> occasion). I've tried with smaller files | constructed datasets and haven't
> been able to replicate it yet. One thing that might be important is:
> {{pickup_location_id}} is all NAs | nulls in the first 8 years of the data or
> so.
> I've attached a backtrace in case that's enough to see what's going on here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (ARROW-16692) [C++] Segfault in datasets
[
https://issues.apache.org/jira/browse/ARROW-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544616#comment-17544616
]
Weston Pace commented on ARROW-16692:
-
{quote}By the backtrace, it looks like a recursion-induced stack blowup.{quote}
Yes, I agree. The culprit seems to be the merged generator, which probably
makes sense. We could fix merged generator but it is going away as part of
ARROW-16072. I've assigned it to myself to test after I finish ARROW-16072.
> [C++] Segfault in datasets
> --
>
> Key: ARROW-16692
> URL: https://issues.apache.org/jira/browse/ARROW-16692
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Jonathan Keane
>Assignee: Weston Pace
>Priority: Major
> Attachments: backtrace.txt
>
>
> I'm still working to make a minimal reproducer for this, though I can
> reliably reproduce it below (though that means needing to download a bunch of
> data first...). I've cleaned out much of the unnecessary code (so this query
> below is a bit silly, and not what I'm actually trying to do), but haven't
> been able to make a constructed dataset that reproduces this.
> Working on some example with the new | more cleaned taxi dataset at
> {{s3://ursa-labs-taxi-data-v2}}, I've run into a segfault:
> {code}
> library(arrow)
> library(dplyr)
> ds <- open_dataset("path/to/new_taxi/")
> ds %>%
> filter(!is.na(pickup_location_id)) %>%
> summarise(n = n()) %>% collect()
> {code}
> Most of the time ends in a segfault (though I have gotten it to work on
> occasion). I've tried with smaller files | constructed datasets and haven't
> been able to replicate it yet. One thing that might be important is:
> {{pickup_location_id}} is all NAs | nulls in the first 8 years of the data or
> so.
> I've attached a backtrace in case that's enough to see what's going on here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (ARROW-16692) [C++] Segfault in datasets
[
https://issues.apache.org/jira/browse/ARROW-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544577#comment-17544577
]
Jonathan Keane commented on ARROW-16692:
bq. One thing that might be important is: pickup_location_id is all NAs | nulls
in the first 8 years of the data or so.
This is almost certainly a redherring now that I come back to it; the following
*also* segfaults without referencing that specific column.
{code}
ds %>%
filter(pickup_datetime > as.Date("2017-01-01")) %>%
summarise(n = n()) %>% collect()
{code}
> [C++] Segfault in datasets
> --
>
> Key: ARROW-16692
> URL: https://issues.apache.org/jira/browse/ARROW-16692
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Jonathan Keane
>Priority: Major
> Attachments: backtrace.txt
>
>
> I'm still working to make a minimal reproducer for this, though I can
> reliably reproduce it below (though that means needing to download a bunch of
> data first...). I've cleaned out much of the unnecessary code (so this query
> below is a bit silly, and not what I'm actually trying to do), but haven't
> been able to make a constructed dataset that reproduces this.
> Working on some example with the new | more cleaned taxi dataset at
> {{s3://ursa-labs-taxi-data-v2}}, I've run into a segfault:
> {code}
> library(arrow)
> library(dplyr)
> ds <- open_dataset("path/to/new_taxi/")
> ds %>%
> filter(!is.na(pickup_location_id)) %>%
> summarise(n = n()) %>% collect()
> {code}
> Most of the time ends in a segfault (though I have gotten it to work on
> occasion). I've tried with smaller files | constructed datasets and haven't
> been able to replicate it yet. One thing that might be important is:
> {{pickup_location_id}} is all NAs | nulls in the first 8 years of the data or
> so.
> I've attached a backtrace in case that's enough to see what's going on here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (ARROW-16692) [C++] Segfault in datasets
[
https://issues.apache.org/jira/browse/ARROW-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544415#comment-17544415
]
Antoine Pitrou commented on ARROW-16692:
By the backtrace, it looks like a recursion-induced stack blowup.
> [C++] Segfault in datasets
> --
>
> Key: ARROW-16692
> URL: https://issues.apache.org/jira/browse/ARROW-16692
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Jonathan Keane
>Priority: Major
> Attachments: backtrace.txt
>
>
> I'm still working to make a minimal reproducer for this, though I can
> reliably reproduce it below (though that means needing to download a bunch of
> data first...). I've cleaned out much of the unnecessary code (so this query
> below is a bit silly, and not what I'm actually trying to do), but haven't
> been able to make a constructed dataset that reproduces this.
> Working on some example with the new | more cleaned taxi dataset at
> {{s3://ursa-labs-taxi-data-v2}}, I've run into a segfault:
> {code}
> library(arrow)
> library(dplyr)
> ds <- open_dataset("path/to/new_taxi/")
> ds %>%
> filter(!is.na(pickup_location_id)) %>%
> summarise(n = n()) %>% collect()
> {code}
> Most of the time ends in a segfault (though I have gotten it to work on
> occasion). I've tried with smaller files | constructed datasets and haven't
> been able to replicate it yet. One thing that might be important is:
> {{pickup_location_id}} is all NAs | nulls in the first 8 years of the data or
> so.
> I've attached a backtrace in case that's enough to see what's going on here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (ARROW-16692) [C++] Segfault in datasets
[
https://issues.apache.org/jira/browse/ARROW-16692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544407#comment-17544407
]
Jonathan Keane commented on ARROW-16692:
cc [~westonpace]
> [C++] Segfault in datasets
> --
>
> Key: ARROW-16692
> URL: https://issues.apache.org/jira/browse/ARROW-16692
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++
>Reporter: Jonathan Keane
>Priority: Major
> Attachments: backtrace.txt
>
>
> I'm still working to make a minimal reproducer for this, though I can
> reliably reproduce it below (though that means needing to download a bunch of
> data first...). I've cleaned out much of the unnecessary code (so this query
> below is a bit silly, and not what I'm actually trying to do), but haven't
> been able to make a constructed dataset that reproduces this.
> Working on some example with the new | more cleaned taxi dataset at
> {{s3://ursa-labs-taxi-data-v2}}, I've run into a segfault:
> {code}
> library(arrow)
> library(dplyr)
> ds <- open_dataset("path/to/new_taxi/")
> ds %>%
> filter(!is.na(pickup_location_id)) %>%
> summarise(n = n()) %>% collect()
> {code}
> Most of the time ends in a segfault (though I have gotten it to work on
> occasion). I've tried with smaller files | constructed datasets and haven't
> been able to replicate it yet. One thing that might be important is:
> {{pickup_location_id}} is all NAs | nulls in the first 8 years of the data or
> so.
> I've attached a backtrace in case that's enough to see what's going on here.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
