Hi, community We are trying to load data in kudu using Impala, but we are encountering the following error:
Status: Memory limit exceeded: Error occurred on backend slave3:22000 by fragment eb410bbc56495c0e:50f46d0800000000 Memory left in process limit: -438.06 MB Query(eb410bbc56495c0e:50f46d0800000000): Reservation=0 ReservationLimit=30.18 GB OtherMemory=35.95 GB Total=35.95 GB Peak=35.95 GB Fragment eb410bbc56495c0e:50f46d0800000000: Reservation=0 OtherMemory=35.95 GB Total=35.95 GB Peak=35.95 GB SORT_NODE (id=1): Total=34.62 GB Peak=34.62 GB HDFS_SCAN_NODE (id=0): Total=1.33 GB Peak=1.37 GB KrpcDataStreamSender (dst_id=2): Total=736.00 B Peak=736.00 B CodeGen: Total=11.72 KB Peak=1.41 MBProcess: memory limit exceeded. Limit=37.72 GB Total=38.15 GB Peak=38.20 GB Buffer Pool: Free Buffers: Total=0 Buffer Pool: Clean Pages: Total=0 Buffer Pool: Unused Reservation: Total=0 Data Stream Service Queue: Limit=1.89 GB Total=0 Peak=0 Data Stream Manager Early RPCs: Total=0 Peak=0 TCMalloc Overhead: Total=75.40 MB RequestPool=default-pool: Total=35.95 GB Peak=35.95 GB Query(eb410bbc56495c0e:50f46d0800000000): Reservation=0 ReservationLimit=30.18 GB OtherMemory=35.95 GB Total=35.95 GB Peak=35.95 GB Untracked Memory: Total=2.13 GB The quey that we executed is as follows: * insert** into lineitem select * from PARQUETIMPALA500.LINEITEM order by L_ORDERKEY limit 750000000 offset 536870913* We are loading data from a parquet table and we are performing the queries on* 500 GB* of* TPCH* data. We also tried the following query: * insert into lineitem select * from PARQUETIMPALA500.LINEITEM* but in this case, the impalad process got aborted after 23hrs of executing the query. We are executing the queries on 8 node cluster with the following configuration Cluster : 8 Node Cluster (48 GB RAM , 8 CPU Core and 2 TB hard-disk each, Intel(R) Core(TM) i7 CPU 950 @ 3.07GHz -- Regards, Geetika Gupta