问题描述
Table « public.weather_data »
Colonne | Type | Collationnement | NULL-able | Par défaut
-------------+--------------------------+-----------------+-----------+------------
timestamp | timestamp with time zone | | not null |
location_id | integer | | not null |
type_id | integer | | not null |
value | double precision | | |
Index :
"weather_data_pkey" PRIMARY KEY,btree (location_id,"timestamp",type_id)
Contraintes de clés étrangères :
"weather_data_location_id_fkey" FOREIGN KEY (location_id) REFERENCES locations(id)
"weather_data_type_id_fkey" FOREIGN KEY (type_id) REFERENCES weather_data_types(id)
Triggers :
ts_insert_blocker BEFORE INSERT ON weather_data FOR EACH ROW EXECUTE PROCEDURE _timescaledb_internal.insert_blocker()
在我的开发环境中,数据量很少
SELECT * FROM hypertable_detailed_size('weather_data');
table_bytes | index_bytes | toast_bytes | total_bytes | node_name
-------------+-------------+-------------+-------------+-----------
12902400 | 9887744 | 0 | 22790144 |
查询使用索引
EXPLAIN SELECT weather_data.timestamp AS anon_1,weather_data.value AS weather_data_value FROM weather_data WHERE 31 = weather_data.location_id AND weather_data.timestamp >= '2000-01-01' AND weather_data.timestamp < '2020-02-02' AND weather_data.type_id = 1 ORDER BY weather_data.timestamp;
Custom Scan (ChunkAppend) on weather_data (cost=0.28..100.10 rows=53 width=16)
Order: weather_data."timestamp"
-> Index Scan using "7790_23369_weather_data_pkey" on _hyper_49_7790_chunk (cost=0.28..1.88 rows=1 width=16)
Index Cond: ((31 = location_id) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone) AND (type_id = 1))
-> Index Scan using "7791_23372_weather_data_pkey" on _hyper_49_7791_chunk (cost=0.28..1.89 rows=1 width=16)
Index Cond: ((31 = location_id) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone) AND (type_id = 1))
[...]
在我的生产环境中,存储了更多数据
SELECT * FROM hypertable_detailed_size('weather_data');
table_bytes | index_bytes | toast_bytes | total_bytes | node_name
-------------+-------------+-------------+-------------+-----------
24146182144 | 34957369344 | 0 | 59103551488 |
索引似乎没有被使用。
Sort (cost=835599.15..836053.40 rows=181699 width=16)
Sort Key: _hyper_1_33_chunk."timestamp"
-> Append (cost=157.08..816618.66 rows=181699 width=16)
-> Bitmap Heap Scan on _hyper_1_33_chunk (cost=157.08..560.33 rows=124 width=16)
Recheck Cond: ((31 = location_id) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone) AND (type_id = 1))
-> Bitmap Index Scan on "33_98_weather_data_pkey" (cost=0.00..157.05 rows=124 width=0)
Index Cond: ((31 = location_id) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone) AND (type_id = 1))
-> Bitmap Heap Scan on _hyper_1_34_chunk (cost=263.76..878.92 rows=198 width=16)
Recheck Cond: ((31 = location_id) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone) AND (type_id = 1))
[...]
我能想到的唯一区别是数据量。
还有别的吗?
我知道,如果查询可能以另一种方式更快,则可能需要权衡导致引擎不使用索引,但这里似乎并非如此。
Table « public.weather_data »
Colonne | Type | Collationnement | NULL-able | Par défaut
-------------+--------------------------+-----------------+-----------+------------
timestamp | timestamp with time zone | | not null |
location_id | integer | | not null |
type_id | integer | | not null |
value | double precision | | |
Index :
"weather_data_pkey" PRIMARY KEY,type_id)
"weather_data_location_id_type_id_timestamp_idx" UNIQUE,type_id,"timestamp")
显然,索引大小变大了:
SELECT * FROM hypertable_detailed_size('weather_data');
table_bytes | index_bytes | toast_bytes | total_bytes | node_name
-------------+-------------+-------------+-------------+-----------
24146182144 | 49657004032 | 0 | 73803186176 |
而且这个索引似乎没有多大帮助。
Sort (cost=604117.07..604571.26 rows=181676 width=16)
Sort Key: _hyper_1_33_chunk."timestamp"
-> Append (cost=6.31..585138.76 rows=181676 width=16)
-> Bitmap Heap Scan on _hyper_1_33_chunk (cost=6.31..409.57 rows=124 width=16)
Recheck Cond: ((31 = location_id) AND (type_id = 1) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone))
-> Bitmap Index Scan on _hyper_1_33_chunk_weather_data_location_id_type_id_timestamp_id (cost=0.00..6.28 rows=124 width=0)
Index Cond: ((31 = location_id) AND (type_id = 1) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone))
-> Bitmap Heap Scan on _hyper_1_34_chunk (cost=7.44..622.60 rows=198 width=16)
Recheck Cond: ((31 = location_id) AND (type_id = 1) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone))
-> Bitmap Index Scan on _hyper_1_34_chunk_weather_data_location_id_type_id_timestamp_id (cost=0.00..7.39 rows=198 width=0)
Index Cond: ((31 = location_id) AND (type_id = 1) AND ("timestamp" >= '2000-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2020-02-02 00:00:00+01'::timestamp with time zone))
知道为什么索引不允许在我的生产环境中进行快速查询以及我可以做些什么吗?
编辑:EXPLAIN (ANALYZE,BUFFERS)
。还修复查询以确保两个数据库返回相同的数据。在我上面的查询中,prod DB 会返回更多数据(实际上,dev DB 不会返回任何数据)。现在查询一整年的每小时数据(即 8760 行)。
开发:
Custom Scan (ChunkAppend) on weather_data (cost=0.42..36800.07 rows=9371 width=16) (actual time=0.033..11.025 rows=8760 loops=1)
Order: weather_data."timestamp"
Buffers: shared hit=1672
-> Index Scan using _hyper_1_137_chunk_weather_data_location_id_type_id_timestamp_i on _hyper_1_137_chunk (cost=0.42..225.73 rows=55 width=16) (actual time=0.031..0.078 rows=49 loops=1)
Index Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Buffers: shared hit=12
-> Index Scan using _hyper_1_138_chunk_weather_data_location_id_type_id_timestamp_i on _hyper_1_138_chunk (cost=0.42..801.22 rows=205 width=16) (actual time=0.017..0.181 rows=168 loops=1)
Index Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Buffers: shared hit=32
关闭 bitmapscan
的生产,与开发相同:
Custom Scan (ChunkAppend) on weather_data (cost=0.42..36800.07 rows=9371 width=16) (actual time=0.033..11.086 rows=8760 loops=1)
Order: weather_data."timestamp"
Buffers: shared hit=1672
-> Index Scan using _hyper_1_137_chunk_weather_data_location_id_type_id_timestamp_i on _hyper_1_137_chunk (cost=0.42..225.73 rows=55 width=16) (actual time=0.031..0.078 rows=49 loops=1)
Index Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Buffers: shared hit=12
-> Index Scan using _hyper_1_138_chunk_weather_data_location_id_type_id_timestamp_i on _hyper_1_138_chunk (cost=0.42..801.22 rows=205 width=16) (actual time=0.017..0.181 rows=168 loops=1)
Index Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Buffers: shared hit=32
使用 bitmapscan
进行生产:
Sort (cost=30667.13..30690.56 rows=9371 width=16) (actual time=14.427..15.353 rows=8760 loops=1)
Sort Key: _hyper_1_137_chunk."timestamp"
Sort Method: quicksort Memory: 795kB
Buffers: shared hit=1672
-> Append (cost=5.26..30048.93 rows=9371 width=16) (actual time=0.044..11.541 rows=8760 loops=1)
Buffers: shared hit=1672
-> Bitmap Heap Scan on _hyper_1_137_chunk (cost=5.26..202.75 rows=55 width=16) (actual time=0.043..0.089 rows=49 loops=1)
Recheck Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Heap Blocks: exact=9
Buffers: shared hit=12
-> Bitmap Index Scan on _hyper_1_137_chunk_weather_data_location_id_type_id_timestamp_i (cost=0.00..5.25 rows=55 width=0) (actual time=0.029..0.029 rows=49 loops=1)
Index Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Buffers: shared hit=3
-> Bitmap Heap Scan on _hyper_1_138_chunk (cost=7.55..642.37 rows=205 width=16) (actual time=0.047..0.189 rows=168 loops=1)
Recheck Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Heap Blocks: exact=28
Buffers: shared hit=32
-> Bitmap Index Scan on _hyper_1_138_chunk_weather_data_location_id_type_id_timestamp_i (cost=0.00..7.50 rows=205 width=0) (actual time=0.037..0.037 rows=168 loops=1)
Index Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Buffers: shared hit=4
编辑 2 使用覆盖索引。
create index on weather_data (location_id,timestamp,value);
Table « public.weather_data »
Colonne | Type | Collationnement | NULL-able | Par défaut
-------------+--------------------------+-----------------+-----------+------------
timestamp | timestamp with time zone | | not null |
location_id | integer | | not null |
type_id | integer | | not null |
value | double precision | | |
Index :
"weather_data_pkey" PRIMARY KEY,type_id)
"weather_data_location_id_type_id_timestamp_value_idx" btree (location_id,value)
索引现在是表格的两倍多。我想我可以使用新索引代替 PK。
SELECT * FROM hypertable_detailed_size('weather_data');
table_bytes | index_bytes | toast_bytes | total_bytes | node_name
-------------+-------------+-------------+-------------+-----------
24146182144 | 53868445696 | 0 | 78014627840 |
查询似乎并没有更快:
Sort (cost=30671.13..30694.56 rows=9371 width=16) (actual time=31.547..32.394 rows=8760 loops=1)
Sort Key: _hyper_1_137_chunk."timestamp"
Sort Method: quicksort Memory: 795kB
Buffers: shared read=1673
-> Append (cost=5.26..30052.93 rows=9371 width=16) (actual time=0.125..28.312 rows=8760 loops=1)
Buffers: shared read=1673
-> Bitmap Heap Scan on _hyper_1_137_chunk (cost=5.26..202.75 rows=55 width=16) (actual time=0.123..0.265 rows=49 loops=1)
Recheck Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Heap Blocks: exact=9
Buffers: shared read=13
-> Bitmap Index Scan on _hyper_1_137_chunk_weather_data_location_id_type_id_timestamp_v (cost=0.00..5.25 rows=55 width=0) (actual time=0.092..0.092 rows=49 loops=1)
Index Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Buffers: shared read=4
-> Bitmap Heap Scan on _hyper_1_138_chunk (cost=11.55..646.37 rows=205 width=16) (actual time=0.100..0.573 rows=168 loops=1)
Recheck Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Heap Blocks: exact=28
Buffers: shared read=32
-> Bitmap Index Scan on _hyper_1_138_chunk_weather_data_location_id_type_id_timestamp_v (cost=0.00..11.50 rows=205 width=0) (actual time=0.079..0.079 rows=168 loops=1)
Index Cond: ((32 = location_id) AND (type_id = 1) AND ("timestamp" >= '2002-01-01 00:00:00+01'::timestamp with time zone) AND ("timestamp" < '2003-01-01 00:00:00+01'::timestamp with time zone))
Buffers: shared read=4
解决方法
您在 (location_id,type_id,"timestamp")
上的 BTREE 索引走在正确的轨道上。
如果您在其定义中使用 INCLUDE(value)
,postgreSQL 可以将其视为 covering index,因此可以使用它来完全满足您的查询,而无需返回主表的堆。这应该很有帮助。
此外,不需要它是唯一的。您的主键索引会为您处理唯一性规则。
CREATE INDEX weather_data_location_id_type_id_timestamp_idx
ON weather_data
USING BTREE
(location_id,"timestamp")
INCLUDE (value);
更好的是,将值包含在索引中。
CREATE INDEX weather_data_location_id_type_id_timestamp_idx
ON weather_data
USING BTREE
(location_id,"timestamp",value);
那么您的索引将能够满足诸如
之类的汇总查询 SELECT DATE_TRUNC('month',timestamp) AS month,location_id,MAX(value) as high,MIN(value) as low
FROM weather_data
WHERE type_id = 1
AND location_id = 31
AND timestamp >= '2000-01-01'
AND timestamp < '2020-02-02'
GROUP BY DATE_TRUNC('month',timestamp),type_id
当然,没有什么魔法可以让处理二十年的天气数据特别快。 postgreSQL 的查询计划器可能会认为通过扫描整个表来满足如此广泛的查询更有效。尝试缩小日期范围。
在玩具大小的表中,简单地将整个表放入 RAM 并扫描它以满足查询可能更有效。我个人从来没有尝试过在玩具大小的桌子上理解执行计划的运气。