lidavidm commented on pull request #12252:
URL: https://github.com/apache/arrow/pull/12252#issuecomment-1021487828
I'm not sure why use_threads doesn't affect this (but GLog below is showing
various thread IDs):
```
pyarrow/tests/test_dataset.py::test_write_dataset_max_open_files WARNING:
Logging before InitGoogleLogging() is written to STDERR
W20220125 13:26:12.820377 27754 file_base.cc:297] Next batch: c1: [
1,
2,
3,
4,
0,
10,
20,
30,
40
]
c2: [
"a",
"b",
"c",
"d",
"e",
"a",
"a",
"b",
"c"
]
W20220125 13:26:12.820533 27755 file_base.cc:297] Next batch: c1: [
5,
6,
7,
8,
0,
1,
10,
20,
30
]
c2: [
"a",
"b",
"c",
"d",
"e",
"c",
"a",
"b",
"c"
]
W20220125 13:26:12.820778 27757 file_base.cc:297] Next batch: c1: [
9,
10,
11,
12,
0,
1,
30,
40,
50
]
c2: [
"a",
"b",
"c",
"d",
"e",
"d",
"a",
"b",
"c"
]
W20220125 13:26:12.820991 27752 file_base.cc:297] Next batch: c1: [
13,
14,
15,
16,
0,
1,
10,
20,
30,
40,
50
]
c2: [
"a",
"b",
"c",
"d",
"e",
"b",
"b",
"a",
"d",
"b",
"a"
]
W20220125 13:26:12.821352 27754 file_base.cc:319] Next partition c1: [
1,
10,
20
]
W20220125 13:26:12.821383 27754 file_base.cc:320] Next destination c2=a
W20220125 13:26:12.821465 27754 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.821472 27754 dataset_writer.cc:309] Opening file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=a/part-0.parquet
W20220125 13:26:12.821590 27755 file_base.cc:319] Next partition c1: [
5,
10
]
W20220125 13:26:12.821632 27755 file_base.cc:320] Next destination c2=a
W20220125 13:26:12.821770 27757 file_base.cc:319] Next partition c1: [
9,
30
]
W20220125 13:26:12.821805 27757 file_base.cc:320] Next destination c2=a
W20220125 13:26:12.821910 27752 file_base.cc:319] Next partition c1: [
13,
20,
50
]
W20220125 13:26:12.821943 27752 file_base.cc:320] Next destination c2=a
W20220125 13:26:12.822041 27755 file_base.cc:319] Next partition c1: [
6,
20
]
W20220125 13:26:12.822079 27755 file_base.cc:320] Next destination c2=b
W20220125 13:26:12.822149 27755 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.822168 27755 dataset_writer.cc:502] Need to close file
W20220125 13:26:12.824494 27761 dataset_writer.cc:321] Closing file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=a/part-0.parquet
W20220125 13:26:12.824517 27761 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.824522 27761 dataset_writer.cc:309] Opening file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=b/part-0.parquet
W20220125 13:26:12.824643 27761 file_base.cc:319] Next partition c1: [
2,
30
]
W20220125 13:26:12.824677 27761 file_base.cc:320] Next destination c2=b
W20220125 13:26:12.824752 27761 file_base.cc:319] Next partition c1: [
10,
40
]
W20220125 13:26:12.824780 27761 file_base.cc:320] Next destination c2=b
W20220125 13:26:12.824851 27761 file_base.cc:319] Next partition c1: [
14,
1,
10,
40
]
W20220125 13:26:12.824880 27761 file_base.cc:320] Next destination c2=b
W20220125 13:26:12.824952 27761 file_base.cc:319] Next partition c1: [
7,
1,
30
]
W20220125 13:26:12.824980 27761 file_base.cc:320] Next destination c2=c
W20220125 13:26:12.825034 27761 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.825044 27761 dataset_writer.cc:502] Need to close file
W20220125 13:26:12.826997 27762 dataset_writer.cc:321] Closing file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=b/part-0.parquet
W20220125 13:26:12.827020 27762 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.827025 27762 dataset_writer.cc:309] Opening file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=c/part-0.parquet
W20220125 13:26:12.827148 27762 file_base.cc:319] Next partition c1: [
3,
40
]
W20220125 13:26:12.827184 27762 file_base.cc:320] Next destination c2=c
W20220125 13:26:12.827260 27762 file_base.cc:319] Next partition c1: [
11,
50
]
W20220125 13:26:12.827288 27762 file_base.cc:320] Next destination c2=c
W20220125 13:26:12.827361 27762 file_base.cc:319] Next partition c1: [
15
]
W20220125 13:26:12.827389 27762 file_base.cc:320] Next destination c2=c
W20220125 13:26:12.827461 27762 file_base.cc:319] Next partition c1: [
4
]
W20220125 13:26:12.827487 27762 file_base.cc:320] Next destination c2=d
W20220125 13:26:12.827543 27762 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.827553 27762 dataset_writer.cc:502] Need to close file
W20220125 13:26:12.829382 27761 dataset_writer.cc:321] Closing file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=c/part-0.parquet
W20220125 13:26:12.829403 27761 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.829409 27761 dataset_writer.cc:309] Opening file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=d/part-0.parquet
W20220125 13:26:12.829531 27761 file_base.cc:319] Next partition c1: [
8
]
W20220125 13:26:12.829560 27761 file_base.cc:320] Next destination c2=d
W20220125 13:26:12.829636 27761 file_base.cc:319] Next partition c1: [
12,
1
]
W20220125 13:26:12.829663 27761 file_base.cc:320] Next destination c2=d
W20220125 13:26:12.829735 27761 file_base.cc:319] Next partition c1: [
16,
30
]
W20220125 13:26:12.829762 27761 file_base.cc:320] Next destination c2=d
W20220125 13:26:12.829834 27761 file_base.cc:319] Next partition c1: [
0
]
W20220125 13:26:12.829860 27761 file_base.cc:320] Next destination c2=e
W20220125 13:26:12.829912 27761 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.829921 27761 dataset_writer.cc:502] Need to close file
W20220125 13:26:12.831791 27762 dataset_writer.cc:321] Closing file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=d/part-0.parquet
W20220125 13:26:12.831813 27762 dataset_writer.cc:499] Need to open file
W20220125 13:26:12.831820 27762 dataset_writer.cc:309] Opening file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=e/part-0.parquet
W20220125 13:26:12.831940 27762 file_base.cc:319] Next partition c1: [
0
]
W20220125 13:26:12.831971 27762 file_base.cc:320] Next destination c2=e
W20220125 13:26:12.832048 27762 file_base.cc:319] Next partition c1: [
0
]
W20220125 13:26:12.832075 27762 file_base.cc:320] Next destination c2=e
W20220125 13:26:12.832150 27762 file_base.cc:319] Next partition c1: [
0
]
W20220125 13:26:12.832176 27762 file_base.cc:320] Next destination c2=e
W20220125 13:26:12.834204 27762 dataset_writer.cc:321] Closing file
/tmp/pytest-of-lidavidm/pytest-232/test_write_dataset_max_open_fi0/ds/max_1/c2=e/part-0.parquet
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
