问题描述
注意：我尝试搜索解决此问题的其他问题和答案，但找不到真正符合我的问题并提供完整解决方案的问题和答案。
我正在尝试使用 sql 生成随机的合成数据来测试我的数据库架构。虽然使用 random() 在 Postgresql 中生成一堆随机值很容易，但生成随机数据集来保留我期望看到的数据的约束和特征则不然。具体来说，我有以下表格：
CREATE TABLE suites(
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,name TEXT
);
INSERT INTO suites(name)
SELECT 'suite' || g FROM generate_series(1,50) g;
CREATE TABLE tests(
id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,name TEXT
);
INSERT INTO tests(name)
SELECT 'test' || g FROM generate_series(1,100) g;
CREATE TABLE tests_in_suites(
suite_id BIGINT,test_id BIGINT,PRIMARY KEY (suite_id,test_id)
);
我想用随机值填充 tests_in_suites，这样每个套件都包含从 tests 统一选择的随机数（3 到 7 个）测试。我希望选择是随机和统一的，并避免循环和其他重复模式。我尝试了以下方法：
SELECT s.id,t.id FROM
(select id from suites) s,(SELECT id FROM tests ORDER BY random() LIMIT 2 + ceil(random() * 5)) t
ORDER BY s.id,t.id;
但它总是为每个套件选择相同数量的测试并且选择的测试是相同的，因为优化器将 s 的子查询替换为常量。我尝试引入对当前正在考虑的套件的依赖，但它抱怨无法访问我尝试使用的值：
SELECT s.id,(SELECT id FROM tests ORDER BY random() LIMIT 2 + ceil(random() * 5 + s.id*0)) t
ORDER BY s.id,t.id;
ERROR: invalid reference to FROM-clause entry for table "s"
LINE 3: ...s ORDER BY random() LIMIT 2 + ceil(random() * 5 + s.id*0)) t
^
HINT: There is an entry for table "s",but it cannot be referenced from this part of the query.
如何生成随机数据而不会成为查询中优化器或无效数据依赖项的牺牲品？
解决方法
我想用随机值填充 tests_in_suites，这样每个套件都包含一个随机数（3 到 7 个）的测试，从测试中统一选择
这听起来像是横向连接的一个很好的用例...
INSERT INTO tests_in_suites(suite_id,test_id)
SELECT suites.id,t.id
FROM suites
CROSS JOIN LATERAL (SELECT id,suites.id AS lol FROM tests ORDER BY random() LIMIT (3+random()*4)) t;
横向联接为联接左侧的表的每一行重新计算联接表，这正是我们在这里想要的。但是如果连接表子查询看起来不是依赖子查询，postgres 会优化它。因此，我在连接表中添加了suites.id 以使其看起来连接表确实依赖于表套件中的行。
可能也有一种方法可以使用 array_agg() 和 unnest() 来做到这一点。
,我找到的解决方案的灵感来自我在网上看到的几个食谱（尤其是在使用 row_number() 来随机选择行时），但它包含了我自己的见解，我还没有看到这种方法在任何地方都可以使用。
关键组件是将生成随机行的艰巨任务分解为一系列更简单的任务，其中每一步我只生成随机整数。然后，为了生成行，我使用递归 CTE，最后在窗口函数 (JOIN) 上使用 row_number() 将行合并到我的结果表中。
下面的解决方案已经在 PostgreSQL 10 和 12 上进行了测试，但它应该适用于任何支持递归 CTE 和窗口函数的版本。它也应该很容易适应支持这些的任何其他 RDBMS。
-- For each suite,add a random number (between 3 and 7) of tests
-- mapped. Because it's difficult to join a random number of rows
-- between two tables in SQL without violating data dependency rules
-- and/or having the optimiser lift it out into a constant,repeating
-- pattern,instead we do it in several steps:
--
-- * For each suite ID,generate a random number between 3 and 7
-- representing the number of tests we want to include
-- * Then,using a recursive CTE,for each suite ID generate rows,-- each with a random integer no larger than the number of
-- tests. Limit the number of rows to the small integer generated in
-- the previous step
-- * Join the table generated in the above CTE with tests on row
-- number,using the random int generated as the row number to
-- pick. This gives us a table containing three values: suite_id,-- test_id,random row number. By extracting only the IDs,we have
-- now generate the values to insert into tests_in_suies
INSERT INTO tests_in_suites
-- "+ id*0" serves to ensure the optimiser sees a dependency on the
-- current row and doesn't lift the random() out as a constant
WITH s(id,n_tests) AS (SELECT id,2 + ceil(random() * 5) + id*0 FROM suites),cnt AS (SELECT COUNT(*) FROM tests),t AS (SELECT id,row_number() over (ORDER BY random()) AS rn FROM tests),sr AS (SELECT * FROM
(WITH RECURSIVE subtests(sid,n,rn) AS (
SELECT s.id,n_tests + 1,NULL::bigint FROM s
UNION
SELECT sid,n - 1,ceil(random() * (SELECT * FROM cnt))::bigint
FROM subtests
WHERE n > 1)
SELECT * FROM subtests) x
WHERE rn IS NOT NULL
ORDER BY sid)
SELECT sid,t.id FROM sr JOIN t USING(rn)
ORDER BY sid,t.id
-- The above will process generate a couple duplicates. They're not a
-- big deal,so just skip them
ON CONFLICT DO NOTHING;
SELECT seen,total,seen / total::double precision as "fraction used" FROM
(SELECT count(*) AS seen FROM (SELECT DISTINCT test_id FROM tests_in_suites) t) x,(SELECT count(*) AS total FROM tests) y;
SELECT suite_id,count(suite_id) FROM tests_in_suites GROUP BY suite_id;
SELECT * FROM tests_in_suites;