[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2022-11-09 Thread wondertx (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17631119#comment-17631119 ] wondertx commented on ARROW-11007: -- Also encountered memory leak when using

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2022-11-08 Thread Julius Uotila (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17630821#comment-17630821 ] Julius Uotila commented on ARROW-11007: --- Hi, I am having the exact same issue as Jan Skorepa, but

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2022-08-17 Thread Ninh Chu (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580959#comment-17580959 ] Ninh Chu commented on ARROW-11007: -- Hi, I also encounter memory problem in v9.0.0. But in my case, the

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2022-08-17 Thread Jan Skorepa (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580727#comment-17580727 ] Jan Skorepa commented on ARROW-11007: - [~apitrou] Thanks for your fast reply. Unfortunately it has

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2022-08-17 Thread Antoine Pitrou (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580698#comment-17580698 ] Antoine Pitrou commented on ARROW-11007: [~skorepaj] Can you try calling

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2022-08-17 Thread Jan Skorepa (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580695#comment-17580695 ] Jan Skorepa commented on ARROW-11007: -   I have been struggling with memory leak in `to_table`

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2022-04-13 Thread Cory Nezin (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17521806#comment-17521806 ] Cory Nezin commented on ARROW-11007: I am also seeing similar behavior with pd.read_parquet and the

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2022-02-03 Thread Peter Gaultney (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486601#comment-17486601 ] Peter Gaultney commented on ARROW-11007: Hi, I think this bug still exists in 6.0.0 of pyarrow.

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-03-03 Thread shadowdsp (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294938#comment-17294938 ] shadowdsp commented on ARROW-11007: --- [~westonpace] thank you very much! > [Python] Memory leak in

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-03-03 Thread Weston Pace (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294934#comment-17294934 ] Weston Pace commented on ARROW-11007: - [~shadowdsp] Thanks for the great reproducible test case.  I

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-03-03 Thread shadowdsp (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17294410#comment-17294410 ] shadowdsp commented on ARROW-11007: --- I have the similar issue in `nested data` on Ubuntu16.04 pyarrow

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-02-06 Thread Dmitry Kashtanov (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17280160#comment-17280160 ] Dmitry Kashtanov commented on ARROW-11007: -- It looks like it's a zero-copy operation since

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-02-05 Thread Antoine Pitrou (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279763#comment-17279763 ] Antoine Pitrou commented on ARROW-11007: Ah, I didn't know that gRPC was involved. Since Arrow

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-02-05 Thread Dmitry Kashtanov (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279733#comment-17279733 ] Dmitry Kashtanov commented on ARROW-11007: --   > That doesn't really answer the question: what

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-02-05 Thread Dmitry Kashtanov (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279688#comment-17279688 ] Dmitry Kashtanov commented on ARROW-11007: -- {code:java}

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-02-05 Thread Antoine Pitrou (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279687#comment-17279687 ] Antoine Pitrou commented on ARROW-11007: > "Mem usage" is by `memory_profiler`. That doesn't

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-02-05 Thread Dmitry Kashtanov (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279681#comment-17279681 ] Dmitry Kashtanov commented on ARROW-11007: -- "Mem usage" is by `memory_profiler`. And as we may

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-02-05 Thread Antoine Pitrou (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279664#comment-17279664 ] Antoine Pitrou commented on ARROW-11007: As you can see, the memory was returned to the

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-02-05 Thread Dmitry Kashtanov (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279642#comment-17279642 ] Dmitry Kashtanov commented on ARROW-11007: -- I have a somewhat similar issue observed both on

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2021-01-04 Thread Joris Van den Bossche (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17258165#comment-17258165 ] Joris Van den Bossche commented on ARROW-11007: --- Given that this comes up from time to

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2020-12-28 Thread Antoine Pitrou (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17255644#comment-17255644 ] Antoine Pitrou commented on ARROW-11007: Also note that "does not return memory immediately" is

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2020-12-22 Thread Wes McKinney (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253714#comment-17253714 ] Wes McKinney commented on ARROW-11007: -- Depending on how you obtained pyarrow, it may be using

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2020-12-22 Thread Michael Peleshenko (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253686#comment-17253686 ] Michael Peleshenko commented on ARROW-11007: [~westonpace] Thanks for the detailed comment.

[jira] [Commented] (ARROW-11007) [Python] Memory leak in pq.read_table and table.to_pandas

2020-12-22 Thread Weston Pace (Jira)
[ https://issues.apache.org/jira/browse/ARROW-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17253676#comment-17253676 ] Weston Pace commented on ARROW-11007: - Hello, thank you for writing up this analysis.  Pyarrow uses