Cloudera Apache impala查询：超出内存限制：无法分配行批次EXCHANGE_NODE？

编程问答 2022-06-11

问题描述

我运行此查询，它有效：

insert into   default.dw_partitioned_table
            partition (partition_islemtarih_string)
select *,cast(replace(strleft(recorddate,10),'-','')as int) as partition_islemtarih_string
from default.dw_hive_table
where recorddate <to_timestamp('2019-05-01 00:00:00','yyyy-MM-dd HH:mm:ss')
 and  recorddate>=to_timestamp('2019-01-01 00:00:00','yyyy-MM-dd HH:mm:ss');

但是当同时运行2个不同的查询时，会出现如下错误：

Memory limit exceeded: Failed to allocate row batch EXCHANGE_NODE (id=1) Could not allocate 8.00 KB without exceeding limit. 
Error occurred on backend xxx.xxyy.intra:22000 
Memory left in process limit: 6.54 GB Query(sss:vvv): Reservation=26.20 GB
ReservationLimit=36.72 GB OtherMemory=37.43 MB Total=26.24 GB Peak=26.24 GB 
Fragment sss:vvv: Reservation=25.94 GB OtherMemory=6.60 MB 
Total=25.95 GB Peak=25.95 GB SORT_NODE (id=2): Reservation=25.94 GB 
OtherMemory=40.00 KB Total=25.94 GB Peak=25.94 GB 
EXCHANGE_NODE (id=1): Reservation=6.55 MB
------------------------------------------------------------------------

我该如何解决？

解决方法

在许多默认配置中，保护栏的数量有限，可以防止一个查询占用群集中的所有内存。

为群集中的资源池启用基于内存的准入控制将防止这种情况-https://docs.cloudera.com/documentation/enterprise/latest/topics/impala_rm_example.html

clouderaimpala