上下文
我有一个应用程序从表中选择加权随机条目,其中前缀总和(权重)是关键部分.简化的表定义如下所示：
CREATE TABLE entries (
id INT NOT NULL PRIMARY KEY AUTO_INCREMENT,weight DECIMAL(9,3),fenwick DECIMAL(9,3)
) ENGINE=MEMORY;
其中`fenwick`将值存储在`weights`的Fenwick树表示中.
让每个条目的“范围”跨越其前缀和与其前缀相加的权重.应用程序必须在0和SUM(权重)之间生成一个随机数@r,并找到其范围包含@r的条目,如下所示：
Fenwick树结合MEMORY引擎和二进制搜索,应该允许我在O(lg ^ 2(n))时间内找到适当的条目,而不是使用朴素查询的O(n)时间：
SELECT a.id-1 FROM (SELECT *,(@x:=@x+weight) AS counter FROM entries
CROSS JOIN (SELECT @x:=0) a
HAVING counter>@r LIMIT 1) a;
研究
由于多个查询的开销,我一直在尝试将前缀sum操作压缩成一个查询(而不是脚本语言中的几个数组访问).在这个过程中,我意识到传统的求和方法,即涉及按降序键顺序访问元素,只会求和第一个元素.我怀疑当WHERE子句中存在变量时,MysqL会线性地运行表.这是查询：
SELECT
SUM(1) INTO @garbage
FROM entries
CROSS JOIN (
SELECT @sum:=0,@n:=@entryid
) a
WHERE id=@n AND @n>0 AND (@n:=@n-(@n&(-@n))) AND (@sum:=@sum+entries.fenwick);
/*SELECT @sum*/
其中@entryid是我们正在计算的前缀和的条目的ID.我确实创建了一个有效的查询(以及返回整数最左边位的函数lft)：
SET @n:=lft(@entryid);
SET @sum:=0;
SELECT
SUM(1) INTO @garbage
FROM entries
WHERE id=@n
AND @n<=@entryid
AND (@n:=@n+lft(@entryid^@n))
AND (@sum:=@sum+entries.fenwick);
/*SELECT @sum*/
但它只证实了我对线性搜索的怀疑. EXPLAIN查询也是如此：
+------+-------------+---------+------+---------------+------+---------+------+--------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+------+-------------+---------+------+---------------+------+---------+------+--------+-------------+
| 1 | SIMPLE | entries | ALL | NULL | NULL | NULL | NULL | 752544 | Using where |
+------+-------------+---------+------+---------------+------+---------+------+--------+-------------+
1 row in set (0.00 sec)
指数：
SHOW INDEXES FROM entries;
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| entries | 0 | PRIMARY | 1 | id | NULL | 752544 | NULL | NULL | | HASH | | |
+---------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
1 row in set (0.00 sec)
现在,我已经看到很多问题,询问如何消除WHERE子句中的变量,以便优化器可以处理查询.但是,如果没有id = @n,我无法想到这个查询的方式.我已经考虑将我想要求的条目的关键值放入一个表并使用连接,但我相信我会得到不良影响：要么过多的表,要么通过评估@entryid来进行线性搜索.
题
最佳答案
放弃
芬威克树对我来说是新的,我发现这篇文章时才发现它们.
这里给出的结果是基于我的理解和一些研究,
但我绝不是一个芬威克树专家,我可能错过了一些东西.
参考资料
说明fenwick树是如何工作的
https://stackoverflow.com/a/15444954/1157540转载自
https://cs.stackexchange.com/a/10541/38148
https://cs.stackexchange.com/a/42816/38148
使用芬威克树
https://en.wikipedia.org/wiki/Fenwick_tree
https://en.wikipedia.org/wiki/Prefix_sum
问题1,找到给定条目的权重
鉴于下表
CREATE TABLE `entries` (
`id` int(11) NOT NULL AUTO_INCREMENT,`weight` decimal(9,3) DEFAULT NULL,`fenwick` decimal(9,3) NOT NULL DEFAULT '0.000',PRIMARY KEY (`id`)
) ENGINE=INNODB;
并且已经填充了数据(参见concat提供的http://sqlfiddle.com/#!9/be1f2/1),
如何计算给定条目@entryid的权重？
这里要理解的关键概念是,fenwick索引的结构基于id值本身的数学和按位运算.
查询通常应仅使用主键查找(WHERE ID = value).
使用排序(ORDER BY)或范围(WHERE(value1< ID)AND(ID< value2))的任何查询都会错过该点,并且不会按预期的顺序遍历树. 例如,使用密钥60：
SET @entryid := 60;
让我们用二进制分解值60
MysqL> SELECT (@entryid & 0x0080) as b8,-> (@entryid & 0x0040) as b7,-> (@entryid & 0x0020) as b6,-> (@entryid & 0x0010) as b5,-> (@entryid & 0x0008) as b4,-> (@entryid & 0x0004) as b3,-> (@entryid & 0x0002) as b2,-> (@entryid & 0x0001) as b1;
+------+------+------+------+------+------+------+------+
| b8 | b7 | b6 | b5 | b4 | b3 | b2 | b1 |
+------+------+------+------+------+------+------+------+
| 0 | 0 | 32 | 16 | 8 | 4 | 0 | 0 |
+------+------+------+------+------+------+------+------+
1 row in set (0.00 sec)
换句话说,只保留位设置,我们有
32 + 16 + 8 + 4 = 60
现在,逐个删除设置的最低位以导航树：
32 + 16 + 8 + 4 = 60
32 + 16 + 8 = 56
32 + 16 = 48
32
这给出了访问元件60的路径(32,48,56,60).
注意,将60转换为(32,60)仅需要对ID值本身进行位计算：不需要访问表或数据库,并且可以在发出查询的客户端中完成此计算.
然后是元素60的分数权重
MysqL> select sum(fenwick) from entries where id in (32,60);
+--------------+
| sum(fenwick) |
+--------------+
| 32.434 |
+--------------+
1 row in set (0.00 sec)
验证
MysqL> select sum(weight) from entries where id <= @entryid;
+-------------+
| sum(weight) |
+-------------+
| 32.434 |
+-------------+
1 row in set (0.00 sec)
现在,让我们比较这些查询的效率.
MysqL> explain select sum(fenwick) from entries where id in (32,60);
+----+-------------+---------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | entries | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 4 | 100.00 | Using where |
+----+-------------+---------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
或者,有所不同
explain format=json select sum(fenwick) from entries where id in (32,60);
{
"query_block": {
"select_id": 1,"cost_info": {
"query_cost": "5.61"
},"table": {
"table_name": "entries","access_type": "range","possible_keys": [
"PRIMARY"
],"key": "PRIMARY","used_key_parts": [
"id"
],"key_length": "4","rows_examined_per_scan": 4,"rows_produced_per_join": 4,"filtered": "100.00","cost_info": {
"read_cost": "4.81","eval_cost": "0.80","prefix_cost": "5.61","data_read_per_join": "64"
},"used_columns": [
"id","fenwick"
],"attached_condition": "(`test`.`entries`.`id` in (32,60))"
}
}
因此,优化器通过主键获取4行(IN子句中有4个值).
当不使用fenwick索引时,我们有
MysqL> explain select sum(weight) from entries where id <= @entryid;
+----+-------------+---------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+---------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
| 1 | SIMPLE | entries | NULL | range | PRIMARY | PRIMARY | 4 | NULL | 60 | 100.00 | Using where |
+----+-------------+---------+------------+-------+---------------+---------+---------+------+------+----------+-------------+
或者,表达方式不同
explain format=json select sum(weight) from entries where id <= @entryid;
{
"query_block": {
"select_id": 1,"cost_info": {
"query_cost": "25.07"
},"rows_examined_per_scan": 60,"rows_produced_per_join": 60,"cost_info": {
"read_cost": "13.07","eval_cost": "12.00","prefix_cost": "25.07","data_read_per_join": "960"
},"weight"
],"attached_condition": "(`test`.`entries`.`id` <= (@`entryid`))"
}
}
优化器在此执行索引扫描,读取60行.
ID = 60时,fenwick的好处是4次,而60次.
现在,考虑一下如何扩展,例如值高达64K.
对于fenwick,16位值将设置最多16位,因此要查找的元素数量最多为16.
如果没有fenwick,扫描最多可以读取64K条目(平均读数为32K).
问题2,找到给定重量的条目
OP问题是找到给定重量的条目.
例如
SET @search_weight := 35.123;
为了说明算法,这篇文章详细说明了如何完成查找(对不起,如果这太冗长了)
SET @found_id := 0;
首先,找出有多少条目.
SET @max_id := (select id from entries order by id desc limit 1);
在测试数据中,max_id为156.
因为128< = max_id< 256,开始搜索的最高位是128.
MysqL> set @search_id := @found_id + 128;
MysqL> select id,fenwick,@search_weight,-> if (fenwick <= @search_weight,"keep","discard") as action
-> from entries where id = @search_id;
+-----+---------+----------------+---------+
| id | fenwick | @search_weight | action |
+-----+---------+----------------+---------+
| 128 | 66.540 | 35.123 | discard |
+-----+---------+----------------+---------+
重量66.540大于我们的搜索,因此丢弃128,继续下一位.
MysqL> set @search_id := @found_id + 64;
MysqL> select id,"discard") as action
-> from entries where id = @search_id;
+----+---------+----------------+--------+
| id | fenwick | @search_weight | action |
+----+---------+----------------+--------+
| 64 | 33.950 | 35.123 | keep |
+----+---------+----------------+--------+
在这里我们需要保持这个位(64),并计算找到的重量：
set @found_id := @search_id,@search_weight := @search_weight - 33.950;
然后继续下一个位：
MysqL> set @search_id := @found_id + 32;
MysqL> select id,"discard") as action
-> from entries where id = @search_id;
+----+---------+----------------+---------+
| id | fenwick | @search_weight | action |
+----+---------+----------------+---------+
| 96 | 16.260 | 1.173 | discard |
+----+---------+----------------+---------+
MysqL> set @search_id := @found_id + 16;
MysqL> select id,"discard") as action
-> from entries where id = @search_id;
+----+---------+----------------+---------+
| id | fenwick | @search_weight | action |
+----+---------+----------------+---------+
| 80 | 7.394 | 1.173 | discard |
+----+---------+----------------+---------+
MysqL> set @search_id := @found_id + 8;
MysqL> select id,"discard") as action
-> from entries where id = @search_id;
+----+---------+----------------+---------+
| id | fenwick | @search_weight | action |
+----+---------+----------------+---------+
| 72 | 3.995 | 1.173 | discard |
+----+---------+----------------+---------+
MysqL> set @search_id := @found_id + 4;
MysqL> select id,"discard") as action
-> from entries where id = @search_id;
+----+---------+----------------+---------+
| id | fenwick | @search_weight | action |
+----+---------+----------------+---------+
| 68 | 1.915 | 1.173 | discard |
+----+---------+----------------+---------+
MysqL> set @search_id := @found_id + 2;
MysqL> select id,"discard") as action
-> from entries where id = @search_id;
+----+---------+----------------+--------+
| id | fenwick | @search_weight | action |
+----+---------+----------------+--------+
| 66 | 1.146 | 1.173 | keep |
+----+---------+----------------+--------+
set @found_id := @search_id,@search_weight := @search_weight - 1.146;
MysqL> set @search_id := @found_id + 1;
MysqL> select id,"discard") as action
-> from entries where id = @search_id;
+----+---------+----------------+--------+
| id | fenwick | @search_weight | action |
+----+---------+----------------+--------+
| 67 | 0.010 | 0.027 | keep |
+----+---------+----------------+--------+
还有一个
set @found_id := @search_id,@search_weight := @search_weight - 0.010;
最终的搜索结果是：
MysqL> select @found_id,@search_weight;
+-----------+----------------+
| @found_id | @search_weight |
+-----------+----------------+
| 67 | 0.017 |
+-----------+----------------+
验证
MysqL> select sum(weight) from entries where id <= 67;
+-------------+
| sum(weight) |
+-------------+
| 35.106 |
+-------------+
MysqL> select sum(weight) from entries where id <= 68;
+-------------+
| sum(weight) |
+-------------+
| 35.865 |
+-------------+
事实上,
35.106 (fenwick[67]) <= 35.123 (search) <= 35.865 (fenwick[68])
搜索查找值一次解析1位,每个查找结果决定要搜索的下一个ID的值.
此处给出的查询仅供参考.在实际应用程序中,代码应该只是一个包含以下内容的循环：
SELECT fenwick from entries where id = ?;
使用应用程序代码(或存储过程)实现与@ found_id,@ search_id和@search_weight相关的逻辑.
普通的留言
> Fenwick树用于前缀计算.如果要解决的问题首先涉及前缀,那么使用这些树才有意义.维基百科有一些应用程序的指针.不幸的是,OP没有描述fenwick树的用途.
> Fenwick树是查找复杂性和更新复杂性之间的权衡,因此它们只对非静态数据有意义.对于静态数据,计算一次完整前缀将更有效.
>执行的测试使用了一个INNODB表,主键索引被排序,因此计算max_id是一个简单的O(1)操作.如果max_id已在其他地方可用,我认为即使使用带有HASH索引ID的MEMORY表也可以,因为只使用主键查找.
附：
sqlfiddle今天已经关闭了,所以发布使用的原始数据(最初由concat提供),以便有兴趣的人可以重新运行测试.
INSERT INTO `entries` VALUES (1,0.480,0.480),(2,0.542,1.022),(3,0.269,0.269),(4,0.721,2.012),(5,0.798,0.798),(6,0.825,1.623),(7,0.731,0.731),(8,0.181,4.547),(9,0.711,0.711),(10,0.013,0.724),(11,0.930,0.930),(12,0.613,2.267),(13,0.276,0.276),(14,0.539,0.815),(15,0.867,0.867),(16,0.718,9.214),(17,0.991,0.991),(18,0.801,1.792),(19,0.033,0.033),(20,0.759,2.584),(21,0.698,0.698),(22,0.212,0.910),(23,0.965,0.965),(24,0.189,4.648),(25,0.049,0.049),(26,0.678,0.727),(27,0.245,0.245),(28,0.190,1.162),(29,0.214,0.214),(30,0.502,0.716),(31,0.868,0.868),(32,0.834,17.442),(33,0.566,0.566),(34,0.327,0.893),(35,0.939,0.939),(36,0.713,2.545),(37,0.747,0.747),(38,0.595,1.342),(39,0.733,0.733),(40,0.884,5.504),(41,0.218,0.218),(42,0.437,0.655),(43,0.532,0.532),(44,0.350,1.537),(45,0.154,0.154),(46,0.875),(47,0.140,0.140),(48,0.538,8.594),(49,0.271,0.271),(50,0.739,1.010),(51,0.884),(52,0.203,2.097),(53,0.361,0.361),(54,0.197,0.558),(55,0.903,0.903),(56,0.923,4.481),(57,0.906,0.906),(58,0.761,1.667),(59,0.089,0.089),(60,0.161,1.917),(61,0.537,0.537),(62,0.201,0.738),(63,0.397,0.397),(64,0.381,33.950),(65,0.715,0.715),(66,0.431,1.146),(67,0.010,0.010),(68,1.915),(69,0.763,0.763),(70,1.300),(71,0.399,0.399),(72,3.995),(73,0.709,0.709),(74,0.401,1.110),(75,0.880,0.880),(76,0.198,2.188),(77,0.348,0.348),(78,0.148,0.496),(79,0.693,0.693),(80,0.022,7.394),(81,0.031,0.031),(82,0.120),(83,0.353,0.353),(84,0.498,0.971),(85,0.428,0.428),(86,0.650,1.078),(87,0.963,0.963),(88,0.866,3.878),(89,0.442,0.442),(90,0.610,1.052),(91,0.725,0.725),(92,0.797,2.574),(93,0.808,0.808),(94,0.648,1.456),(95,0.817,0.817),(96,0.141,16.260),(97,0.256,0.256),(98,0.855,1.111),(99,0.508,0.508),(100,0.976,2.595),(101,(102,0.840,1.193),(103,0.139,0.139),(104,0.178,4.105),(105,0.469,0.469),(106,0.814,1.283),(107,0.664,0.664),(108,0.876,2.823),(109,0.390,0.390),(110,0.323,0.713),(111,(112,0.241,8.324),(113,0.881,0.881),(114,0.681,1.562),(115,0.760,0.760),(116,3.082),(117,0.518,0.518),(118,0.313,0.831),(119,0.008,0.008),(120,0.103,4.024),(121,0.488,0.488),(122,0.135,0.623),(123,0.207,0.207),(124,0.633,1.463),(125,0.542),(126,0.812,1.354),(127,0.433,0.433),(128,0.732,66.540),(129,0.358,0.358),(130,0.594,0.952),(131,0.897,0.897),(132,0.701,2.550),(133,0.815,(134,0.973,1.788),(135,0.419,0.419),(136,0.175,4.932),(137,0.620,0.620),(138,0.573,(139,0.004,0.004),(140,0.304,1.501),(141,(142,0.629,1.137),(143,0.618,0.618),(144,0.206,8.394),(145,0.175),(146,0.255,0.430),(147,0.750,0.750),(148,0.987,2.167),(149,0.683,0.683),(150,0.453,1.136),(151,0.219,0.219),(152,0.734,4.256),(153,0.016,0.016),(154,0.874,0.891),(155,0.325,0.325),(156,0.002,1.217);
附： 2
