如何构建所有数据源共享相同结构的管道架构？

问题描述

我正在尝试构建一个数据流管道，其中多个 N 数据源将我的数据流式传输到节点管理器，它将保存或临时保存到 postgresql 类似于用户发送消息然后另一方接收该数据的聊天保存它，但是我想首先面临一些问题，如果我将来自每个数据源的数据作为 dict 列表或单个 dict 流式传输，并且节点管理器它将对所有数据进行分组以执行批量插入

示例：https://avleonov.com/wp-content/uploads/2017/05/home-grown_vulnerability_database.png

我的方法

用于任务队列的芹菜（2 个任务，一个每 24 小时运行一次，另一个每小时运行一次）
节点管理器，用作聊天以传输数据
抓取数据到节点管理器
节点管理器保存数据

详细信息对于速度 cvssv2,cvssv3，它已被删除

{
   "cveid":"CVE-2009-20001","summary":"An issue was discovered in MantisBT before 2.24.5. It associates a unique cookie string with each user. This string is not reset upon logout (i.e.,the user session is still considered valid and active),allowing an attacker who somehow gained access to a user's cookie to login as them.","vulnerability_type":"613","affected_products":"Mantisbt Mantisbt *","advisory_url":"https://mantisbt.org/bugs/view.PHP?id=27976","detection_date":"2021-03-07"
}

NVD

{
   "cveid":"CVE-1999-0002","summary":"Buffer overflow in NFS mountd gives root access to remote attackers,mostly in Linux systems.","vulnerability_type":"CWE-119","affected_products":"N/A","cvssv2":{
      "version":"2.0","vectorString":"AV:N/AC:L/Au:N/C:C/I:C/A:C","accessvector":"NETWORK","accessComplexity":"LOW","authentication":"NONE","confidentialityImpact":"COMPLETE","integrityImpact":"COMPLETE","availabilityImpact":"COMPLETE","basescore":"Decimal(""10.0"")"
   },"cvssv3":{
      
   },"advisory_url":[
      "http://www.securityfocus.com/bid/121","ftp://patches.sgi.com/support/free/security/advisories/19981006-01-I","http://www.ciac.org/ciac/bulletins/j-006.shtml"
   ],"detection_date":datetime.datetime(1998,10,12,4,"tzinfo=tzutc())"
}

https://avleonov.com/wp-content/uploads/2017/05/home-grown_vulnerability_database.png

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

data-stream design-patterns pipeline pipeline