jnturton commented on PR #2610: URL: https://github.com/apache/drill/pull/2610#issuecomment-1193860317
@luocooong @paul-rogers a bit of data related to the heap memory usage limiting from some experiments run by a helpful user. I did test the recommendation given by you and it worked well. Below are steps of testing > drill.exec.http.memory.heap.failure.threshold = 2 > > Before making changes in the drill-override.conf > 1. Opened 4 sessions of drill web UI query > 2. Fired same query in all 4 sessions. This query will scan through 8000 parquet compress files > > Results - all 4 sessions gave heap error. > Observations > 1. The heap memory displayed on "metrics" tab varies between 30-80% while query is running > 2. The heap memory is constant at 87.89% after queries are failed > > After making the changes in the drill-overide.conf as recommended by you > 1. Opened 4 sessions of drill web UI query > 2. Fired same query in all 4 sessions. This query will scan through 8000 parquet compress files > Results - All 4 sessions returned the results. It took average 6+ minutes to return the results > Observations > 1. Could not observe the heap memory displayed on "metrics" as display was just in running state and then gave error "page cannot be displayed" during query run > 2. The heap memory is varying between 69.71% after queries are successfully completed > > I ran the same queries again and again and below is the heap memory after every run > 2nd run - Between 61-62% > 3rd run - . Between 82-83% > 4th Run - Between 69-70% > 5th Run - Between 65-66% > 6th run - Between 69-70% > 7th run - Between 70-71% > 8th run - increased one session, so now total are 5 web sessions - 1 session gave "Page cannot be displayed error. 4 returned the results - Memory 79-80% > 9th run (5 sessions) - - 1 session gave "Page cannot be displayed error. 4 returned the results - Memory 66-67% > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@drill.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
