Andrey Zimovnov created SPARK-9539:
--------------------------------------

             Summary: Repeated sc.close() in PySpark causes JVM memory leak
                 Key: SPARK-9539
                 URL: https://issues.apache.org/jira/browse/SPARK-9539
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 1.4.1
            Reporter: Andrey Zimovnov
            Priority: Minor


Example code in Python:
for i in range(20):
        print i
        conf = SparkConf().setAppName("test")
        sc = SparkContext(conf=conf)
        hivec = HiveContext(sc)
        hivec.sql("select id from details_info limit 1").show()
        sc.stop()
        del hivec
        del sc

Jstat output:
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU 
   YGC     YGCT    FGC    FGCT     GCT
196608,0 196608,0 97566,2  0,0   1179648,0 542150,0 3145728,0    120,0    
154112,0 153613,2      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 679041,7 3145728,0    120,0    
164352,0 164183,3      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 907928,4 3145728,0    120,0    
164352,0 164200,3      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 912132,7 3145728,0    120,0    
164352,0 164200,5      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 913741,5 3145728,0    120,0    
164352,0 164200,8      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 929458,6 3145728,0    120,0    
164352,0 164206,0      4    0,434   0      0,000    0,434
196608,0 196608,0 97566,2  0,0   1179648,0 1003138,1 3145728,0    120,0    
168960,0 168646,0      4    0,434   0      0,000    0,434
131584,0 196608,0  0,0   109725,6 1179648,0   0,0    3145728,0    128,0    
175104,0 174802,1      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 152654,9 3145728,0    128,0    
175104,0 174803,3      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 158586,1 3145728,0    128,0    
175104,0 174803,3      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 160659,8 3145728,0    128,0    
175104,0 174805,7      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 181935,2 3145728,0    128,0    
175104,0 174819,7      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 283389,1 3145728,0    128,0    
185856,0 185371,0      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 342596,4 3145728,0    128,0    
185856,0 185379,3      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 547634,7 3145728,0    128,0    
185856,0 185385,8      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 555930,9 3145728,0    128,0    
185856,0 185385,8      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 557888,6 3145728,0    128,0    
185856,0 185386,0      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 573907,5 3145728,0    128,0    
185856,0 185397,5      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 637955,0 3145728,0    128,0    
189952,0 189533,1      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 895866,1 3145728,0    128,0    
196096,0 195968,5      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 948046,5 3145728,0    128,0    
196096,0 195969,4      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 952427,2 3145728,0    128,0    
196096,0 195969,4      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 957977,5 3145728,0    128,0    
196096,0 195973,4      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 977811,1 3145728,0    128,0    
196096,0 195977,7      5    0,591   0      0,000    0,591
131584,0 196608,0  0,0   109725,6 1179648,0 1118722,0 3145728,0    128,0    
206848,0 206539,0      5    0,591   0      0,000    0,591
131584,0 144384,0 118692,5  0,0   1284096,0 183470,8 3145728,0    136,0    
206848,0 206543,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 189718,5 3145728,0    136,0    
206848,0 206543,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 192165,0 3145728,0    136,0    
206848,0 206543,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 199848,4 3145728,0    136,0    
206848,0 206546,9      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 219687,6 3145728,0    136,0    
206848,0 206552,2      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 358272,4 3145728,0    136,0    
217600,0 217100,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 573543,6 3145728,0    136,0    
217600,0 217109,4      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 593569,9 3145728,0    136,0    
217600,0 217109,6      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 593576,3 3145728,0    136,0    
217600,0 217109,6      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 608832,1 3145728,0    136,0    
217600,0 217114,6      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 649072,6 3145728,0    136,0    
218624,0 218234,7      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 774293,0 3145728,0    136,0    
227840,0 227734,6      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 971194,9 3145728,0    136,0    
227840,0 227739,2      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 992291,6 3145728,0    136,0    
227840,0 227739,2      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 992297,7 3145728,0    136,0    
227840,0 227739,2      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 1002356,2 3145728,0    136,0    
227840,0 227745,1      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 1020944,8 3145728,0    136,0    
227840,0 227754,0      6    0,773   0      0,000    0,773
131584,0 144384,0 118692,5  0,0   1284096,0 1169159,0 3145728,0    136,0    
238592,0 238332,9      6    0,773   0      0,000    0,773
147456,0 109056,0  0,0   108867,6 1284096,0 110659,1 3145728,0   19167,5   
238592,0 238479,1      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 116954,2 3145728,0   19167,5   
238592,0 238491,0      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 116960,6 3145728,0   19167,5   
238592,0 238491,0      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 132244,5 3145728,0   19167,5   
238592,0 238491,9      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 218816,2 3145728,0   19167,5   
245760,0 244549,2      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 467076,0 3145728,0   19167,5   
249344,0 249189,3      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 508023,8 3145728,0   19167,5   
249344,0 249189,8      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 511485,1 3145728,0   19167,5   
249344,0 249191,3      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 513094,4 3145728,0   19167,5   
249344,0 249191,9      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 527507,0 3145728,0   19167,5   
249344,0 249197,0      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 636565,3 3145728,0   19167,5   
260096,0 259705,2      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 819041,0 3145728,0   19167,5   
260096,0 259749,7      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 917166,3 3145728,0   19167,5   
260096,0 259749,7      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 919595,7 3145728,0   19167,5   
260096,0 259749,7      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 925158,0 3145728,0   19167,5   
260096,0 259750,1      7    0,942   0      0,000    0,942
147456,0 109056,0  0,0   108867,6 1284096,0 946351,0 3145728,0   19167,5   
260096,0 259751,1      7    0,942   0      0,000    0,942
151552,0 1024,0  0,0   924,7  1284608,0   0,0    3145728,0   119860,5  262144,0 
262143,9      9    1,038   2      2,021    3,059
148480,0 153088,0  0,0    64,0  1262080,0   0,0    3145728,0   84562,9   
262144,0 262143,9     17    1,167  10      6,988    8,155
99840,0 93696,0 160,0   0,0   1379328,0   0,0    3145728,0   84035,5   262144,0 
262143,6     26    1,296  19     12,022   13,318
50688,0 54272,0  0,0   224,0  1460224,0   0,0    3145728,0   83958,1   262144,0 
262143,8     35    1,474  28     16,642   18,116
23552,0 25088,0  0,0    96,0  1520640,0   0,0    3145728,0   84007,6   262144,0 
262144,0     45    1,638  38     21,492   23,130

It looks like it hits -XX:MaxPermSize=256m for the driver and dies with OOM: 
Exception: ("You must build Spark with Hive. Export 'SPARK_HIVE=true' and run 
build/sbt assembly", Py4JJavaError(u'An error occurred while calling 
None.org.apache.spark.sql.hive.HiveContext.\n', JavaObject id=o753))



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to