一、mycat的安装
环境准备:准备一台虚拟机192.168.152.128
1. 下载mycat
2. 解压mycat
3. 剪切mycat到/usr/local目录下
4. 启动mycat
说明:这里有个小插曲,启动一会mycat又会自己停止
通过查看日志
vim /usr/local/mycat/logs/wrapper.log
报:Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=64M; support was removed in 8.0
这是因为使用的是jdk8的版本,jdk8以后就没有永久代了,而mycat的wrapper.conf里面配置了jdk8以前的永久代的内存大小,我们只需要注释这个配置就可以了
重新启动mycat:
就不会出现启动一会又自己停止的问题了
连接mycat:
可以看到mycat连接成功
5. linux下mycat常用命令
二、mycat配置详解
1. 首先查看mycat的文件目录
mycat的主要配置文件都在conf目录下,核心的配置文件是schema.xml、server.xml、rule.xml这3个配置文件,下面我们来对这3个配置文件进行详细介绍
2. schema.xml
Mycat主要管理
<span style="color: #008000"><!--</span><span style="color: #008000">
schema标签:指定Mycat的逻辑库(mycat的Schema,可以指定多个schema标签)
checkSQLschema属性:
False 过滤schema定义。
select * from testdb.company;=> select * from company;
True 不过滤schema定义,当我们把testdb输错了或者没有设置大小写敏感就会报错
sqlMaxLimit属性:
Limit 自动加入limit,查询语句如果没有加入分页参数,当数据量很大的时候,mycat认为会有问题时就启用这个参数只返回100条。例如:select * from company。
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">schema </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="TESTDB"</span><span style="color: #ff0000"> checkSQLschema</span><span style="color: #0000ff">="false"</span><span style="color: #ff0000"> sqlMaxLimit</span><span style="color: #0000ff">="100"</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
table标签:指定Mycat中的逻辑表。最后要做数据分片的表。mycat中的表,可以和实体有具体映射关系,也可以没有具体映射关系(如果没有映射关系会报错)。
dataNode属性:数据节点,把相应的表存到对应的DB中。
rule属性:分片规则。对应rule.xml中的规则。
ruleRequired属性:指定该属性的表是否需要分片规则。True 必须制定,如果没有制定,就会报错。 primaryKey属性:值为具体的物理表的主键id,如果使用非主键进行分片,那么Mycat会缓存主键和具体dataNode的信息,如果下次再使用非主键进行查询的时候,就不用广播所有dn。
type属性:逻辑表的类型。普通表和全局表,全局表不需要分片,是为了解决跨库join的,全局表一般是数据量比较小、基本不会增长的表
autoIncrement属性:Mycat根据last_insert_id()返回结果。这个需要和mysql自增长主键配合。
needAddLimit属性:是否自动在每一条SQL语句后面加上limit限制。
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">table </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="travelrecord"</span><span style="color: #ff0000"> dataNode</span><span style="color: #0000ff">="dn1,dn2,dn3"</span><span style="color: #ff0000"> rule</span><span style="color: #0000ff">="auto-sharding-long"</span> <span style="color: #0000ff">/></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> global table is auto cloned to all defined data nodes,so can join
with any table whose sharding node is in the same data node </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">table </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="company"</span><span style="color: #ff0000"> primaryKey</span><span style="color: #0000ff">="ID"</span><span style="color: #ff0000"> type</span><span style="color: #0000ff">="global"</span><span style="color: #ff0000"> dataNode</span><span style="color: #0000ff">="dn1,dn3"</span> <span style="color: #0000ff">/></span>
<span style="color: #0000ff"><</span><span style="color: #800000">table </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="goods"</span><span style="color: #ff0000"> primaryKey</span><span style="color: #0000ff">="ID"</span><span style="color: #ff0000"> type</span><span style="color: #0000ff">="global"</span><span style="color: #ff0000"> dataNode</span><span style="color: #0000ff">="dn1,dn2"</span> <span style="color: #0000ff">/></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> random sharding using mod sharind rule </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">table </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="hotnews"</span><span style="color: #ff0000"> primaryKey</span><span style="color: #0000ff">="ID"</span><span style="color: #ff0000"> autoIncrement</span><span style="color: #0000ff">="true"</span><span style="color: #ff0000"> dataNode</span><span style="color: #0000ff">="dn1,dn3"</span><span style="color: #ff0000">
rule</span><span style="color: #0000ff">="mod-long"</span> <span style="color: #0000ff">/></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <table name="dual" primaryKey="ID" dataNode="dnx,dnoracle2" type="global"
needAddLimit="false"/> <table name="worker" primaryKey="ID" dataNode="jdbc_dn1,jdbc_dn2,jdbc_dn3"
rule="mod-long" /> </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">table </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="employee"</span><span style="color: #ff0000"> primaryKey</span><span style="color: #0000ff">="ID"</span><span style="color: #ff0000"> dataNode</span><span style="color: #0000ff">="dn1,dn2"</span><span style="color: #ff0000">
rule</span><span style="color: #0000ff">="sharding-by-intfile"</span> <span style="color: #0000ff">/></span>
<span style="color: #0000ff"><</span><span style="color: #800000">table </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="customer"</span><span style="color: #ff0000"> primaryKey</span><span style="color: #0000ff">="ID"</span><span style="color: #ff0000"> dataNode</span><span style="color: #0000ff">="dn1,dn2"</span><span style="color: #ff0000">
rule</span><span style="color: #0000ff">="sharding-by-intfile"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">childTable </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="orders"</span><span style="color: #ff0000"> primaryKey</span><span style="color: #0000ff">="ID"</span><span style="color: #ff0000"> joinKey</span><span style="color: #0000ff">="customer_id"</span><span style="color: #ff0000">
parentKey</span><span style="color: #0000ff">="id"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">childTable </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="order_items"</span><span style="color: #ff0000"> joinKey</span><span style="color: #0000ff">="order_id"</span><span style="color: #ff0000">
parentKey</span><span style="color: #0000ff">="id"</span> <span style="color: #0000ff">/></span>
<span style="color: #0000ff"></</span><span style="color: #800000">childTable</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">childTable </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="customer_addr"</span><span style="color: #ff0000"> primaryKey</span><span style="color: #0000ff">="ID"</span><span style="color: #ff0000"> joinKey</span><span style="color: #0000ff">="customer_id"</span><span style="color: #ff0000">
parentKey</span><span style="color: #0000ff">="id"</span> <span style="color: #0000ff">/></span>
<span style="color: #0000ff"></</span><span style="color: #800000">table</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <table name="oc_call" primaryKey="ID" dataNode="dn1$0-743" rule="latest-month-calldate"
/> </span><span style="color: #008000">--></span>
<span style="color: #0000ff"></</span><span style="color: #800000">schema</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
dataNode标签:定义数据节点
dataHost属性: 主机的名称
database属性: 数据库
</span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <dataNode name="dn1$0-743" dataHost="localhost1" database="db$0-743"
/> </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">dataNode </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="dn1"</span><span style="color: #ff0000"> dataHost</span><span style="color: #0000ff">="localhost1"</span><span style="color: #ff0000"> database</span><span style="color: #0000ff">="db1"</span> <span style="color: #0000ff">/></span>
<span style="color: #0000ff"><</span><span style="color: #800000">dataNode </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="dn2"</span><span style="color: #ff0000"> dataHost</span><span style="color: #0000ff">="localhost1"</span><span style="color: #ff0000"> database</span><span style="color: #0000ff">="db2"</span> <span style="color: #0000ff">/></span>
<span style="color: #0000ff"><</span><span style="color: #800000">dataNode </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="dn3"</span><span style="color: #ff0000"> dataHost</span><span style="color: #0000ff">="localhost1"</span><span style="color: #ff0000"> database</span><span style="color: #0000ff">="db3"</span> <span style="color: #0000ff">/></span>
<span style="color: #008000"><!--</span><span style="color: #008000"><dataNode name="dn4" dataHost="sequoiadb1" database="SAMPLE" />
<dataNode name="jdbc_dn1" dataHost="jdbchost" database="db1" />
<dataNode name="jdbc_dn2" dataHost="jdbchost" database="db2" />
<dataNode name="jdbc_dn3" dataHost="jdbchost" database="db3" /> </span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
dataHost标签:主要定义和Mysql集群有关的信息,数据实例、读写分离配置和心跳检测语句。
balance属性:读操作负载均衡配置
0 代表不开启读写分离,所有的读操作都发送到writeHost上。
1 writeHost和readHost都要参与select语句的负载均衡。例如:双主双从模式 M1->S1,M2->S2,M1和M2互为主备。M2/S1/S2都要参与select语句的负载均衡。
2 所有读操作都随机分配给writeHost/readHost
3 所有的读操作随机分发到writeHost下面的readHost上执行。
writeType属性:写操作负载均衡配置
0 所有的写操作都分发到第一个writeHost。如果第一个挂了,分发到第二个。
1 所有的写操作都要随机分发到所有配置的writeHost上。1.5以后不推荐。
dbType属性:支持的数据库类型,mycat支持多种db类型:mysql、oracle等
switchType属性:切换类型,主备切换,主主切换。
-1 代表不自动切换
1 默认值,自动切换。对应的心跳检测语句select user();
2 基于Mysql主从同步的状态决定是否切换。对应的心跳检测语句show slave status;
3 基于MySQL Galera Cluster切换机制。对应的心跳检测语句show status like ‘wsrep%’;
maxCon属性:单个writeHost/readHost节点的最大连接数
minCon属性:单个writeHost/readHost节点的最小连接数
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">dataHost </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="localhost1"</span><span style="color: #ff0000"> maxCon</span><span style="color: #0000ff">="1000"</span><span style="color: #ff0000"> minCon</span><span style="color: #0000ff">="10"</span><span style="color: #ff0000"> balance</span><span style="color: #0000ff">="0"</span><span style="color: #ff0000">
writeType</span><span style="color: #0000ff">="0"</span><span style="color: #ff0000"> dbType</span><span style="color: #0000ff">="mysql"</span><span style="color: #ff0000"> dbDriver</span><span style="color: #0000ff">="native"</span><span style="color: #ff0000"> switchType</span><span style="color: #0000ff">="1"</span><span style="color: #ff0000"> slaveThreshold</span><span style="color: #0000ff">="100"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">heartbeat</span><span style="color: #0000ff">></span>select user()<span style="color: #0000ff"></</span><span style="color: #800000">heartbeat</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> can have multi write hosts </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">writeHost </span><span style="color: #ff0000">host</span><span style="color: #0000ff">="hostM1"</span><span style="color: #ff0000"> url</span><span style="color: #0000ff">="localhost:3306"</span><span style="color: #ff0000"> user</span><span style="color: #0000ff">="root"</span><span style="color: #ff0000">
password</span><span style="color: #0000ff">="123456"</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> can have multi read hosts </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">readHost </span><span style="color: #ff0000">host</span><span style="color: #0000ff">="hostS2"</span><span style="color: #ff0000"> url</span><span style="color: #0000ff">="192.168.1.200:3306"</span><span style="color: #ff0000"> user</span><span style="color: #0000ff">="root"</span><span style="color: #ff0000"> password</span><span style="color: #0000ff">="xxx"</span> <span style="color: #0000ff">/></span>
<span style="color: #0000ff"></</span><span style="color: #800000">writeHost</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">writeHost </span><span style="color: #ff0000">host</span><span style="color: #0000ff">="hostS1"</span><span style="color: #ff0000"> url</span><span style="color: #0000ff">="localhost:3316"</span><span style="color: #ff0000"> user</span><span style="color: #0000ff">="root"</span><span style="color: #ff0000">
password</span><span style="color: #0000ff">="123456"</span> <span style="color: #0000ff">/></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <writeHost host="hostM2" url="localhost:3316" user="root" password="123456"/> </span><span style="color: #008000">--></span>
<span style="color: #0000ff"></</span><span style="color: #800000">dataHost</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
<dataHost name="sequoiadb1" maxCon="1000" minCon="1" balance="0" dbType="sequoiadb" dbDriver="jdbc">
<heartbeat> </heartbeat>
<writeHost host="hostM1" url="sequoiadb://1426587161.dbaas.sequoialab.net:11920/SAMPLE" user="jifeng" password="jifeng"></writeHost>
</dataHost>
<dataHost name="oracle1" maxCon="1000" minCon="1" balance="0" writeType="0" dbType="oracle" dbDriver="jdbc"> <heartbeat>select 1 from dual</heartbeat>
<connectionInitSql>alter session set nls_date_format='yyyy-mm-dd hh24:mi:ss'</connectionInitSql>
<writeHost host="hostM1" url="jdbc:oracle:thin:@127.0.0.1:1521:nange" user="base" password="123456" > </writeHost> </dataHost>
<dataHost name="jdbchost" maxCon="1000" minCon="1" balance="0" writeType="0" dbType="mongodb" dbDriver="jdbc">
<heartbeat>select user()</heartbeat>
<writeHost host="hostM" url="mongodb://192.168.0.99/test" user="admin" password="123456" ></writeHost> </dataHost>
<dataHost name="sparksql" maxCon="1000" minCon="1" balance="0" dbType="spark" dbDriver="jdbc">
<heartbeat> </heartbeat>
<writeHost host="hostM1" url="jdbc:hive2://feng01:10000" user="jifeng" password="jifeng"></writeHost> </dataHost> </span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <dataHost name="jdbchost" maxCon="1000" minCon="10" balance="0" dbType="mysql"
dbDriver="jdbc"> <heartbeat>select user()</heartbeat> <writeHost host="hostM1"
url="jdbc:mysql://localhost:3306" user="root" password="123456"> </writeHost>
</dataHost> </span><span style="color: #008000">--></span>
<span style="color: #0000ff"></
<span style="color: #800000">mycat:schema<span style="color: #0000ff">>3. server.xml
Mycat这个文件主要和
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="sequnceHandlerType"</span><span style="color: #0000ff">></span>2<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <property name="useCompression">1</property></span><span style="color: #008000">--></span> <span style="color: #008000"><!--</span><span style="color: #008000">1为开启mysql压缩协议</span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <property name="fakeMySQLVersion">5.6.20</property></span><span style="color: #008000">--></span> <span style="color: #008000"><!--</span><span style="color: #008000">设置模拟的MySQL版本号</span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <property name="processorBufferChunk">40960</property> </span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
<property name="processors">1</property>
<property name="processorExecutor">32</property>
</span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000">默认为type 0: DirectByteBufferPool | type 1 ByteBufferArena</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="processorBufferPoolType"</span><span style="color: #0000ff">></span>0<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">默认是65535 64K 用于sql解析时最大文本长度 </span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"><property name="maxStringLiteralLength">65535</property></span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"><property name="sequnceHandlerType">0</property></span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"><property name="backSocketNoDelay">1</property></span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"><property name="frontSocketNoDelay">1</property></span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"><property name="processorExecutor">16</property></span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
<property name="serverPort">8066</property> <property name="managerPort">9066</property>
<property name="idleTimeout">300000</property> <property name="bindIp">0.0.0.0</property>
<property name="frontWriteQueueSize">4096</property> <property name="processors">32</property> </span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000">分布式事务开关,0为不过滤分布式事务,1为过滤分布式事务(如果分布式事务内只涉及全局表,则不过滤),2为不过滤分布式事务,但是记录分布式事务日志</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="handleDistributedTransactions"</span><span style="color: #0000ff">></span>0<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
off heap for merge/order/group/limit 1开启 0关闭
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="useOffHeapForMerge"</span><span style="color: #0000ff">></span>1<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
单位为m
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="memoryPageSize"</span><span style="color: #0000ff">></span>1m<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
单位为k
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="spillsFileBufferSize"</span><span style="color: #0000ff">></span>1k<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="useStreamOutput"</span><span style="color: #0000ff">></span>0<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
单位为m
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="systemReserveMemorySize"</span><span style="color: #0000ff">></span>384m<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">是否采用zookeeper协调切换 </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="useZKSwitch"</span><span style="color: #0000ff">></span>true<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">system</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
Firewall标签:全局SQL防火墙设置
定义访问控制策略:如白名单/黑名单
</span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
<firewall>
<whitehost>
<host host="127.0.0.1" user="mycat"/>
<host host="127.0.0.2" user="mycat"/>
</whitehost>
<blacklist check="false">
</blacklist>
</firewall>
</span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
User标签:定义可访问mycat的用户名称/密码/是否只读
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">user </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="root"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="password"</span><span style="color: #0000ff">></span>123456<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="schemas"</span><span style="color: #0000ff">></span>TESTDB<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
Privileges标签:表级 DML 权限设置
控制DML:insert/update/select/delete
单独给select权限:0010
单独给insert权限:1000
</span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
<privileges check="false">
<schema name="TESTDB" dml="0110" >
<table name="tb01" dml="0000"></table>
<table name="tb02" dml="1111"></table>
</schema>
</privileges>
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"></</span><span style="color: #800000">user</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">user </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="user"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="password"</span><span style="color: #0000ff">></span>user<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="schemas"</span><span style="color: #0000ff">></span>TESTDB<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="readOnly"</span><span style="color: #0000ff">></span>true<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">user</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></
<span style="color: #800000">mycat:server<span style="color: #0000ff">>
定义分片规则策略文件。
<span style="color: #008000"><!--</span><span style="color: #008000">
tableRule标签:定义table分片策略
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="rule1"</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
rule标签:策略定义标签
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
columns标签:对应的分片字段
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
algorithm标签:tableRule分片策略对应的function名称
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>func1<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="rule2"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>user_id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>func1<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="sharding-by-intfile"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>sharding_id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>hash-int<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="auto-sharding-long"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>rang-long<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="mod-long"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>mod-long<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="sharding-by-murmur"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>murmur<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="crc32slot"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>crc32slot<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="sharding-by-month"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>create_time<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>partbymonth<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="latest-month-calldate"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>calldate<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>latestMonth<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="auto-sharding-rang-mod"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>rang-mod<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">tableRule </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="jch"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>id<span style="color: #0000ff"></</span><span style="color: #800000">columns</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>jump-consistent-hash<span style="color: #0000ff"></</span><span style="color: #800000">algorithm</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">rule</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">tableRule</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000">
function标签:定义分片函数
</span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="murmur"</span><span style="color: #ff0000">
class</span><span style="color: #0000ff">="io.mycat.route.function.PartitionByMurmurHash"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="seed"</span><span style="color: #0000ff">></span>0<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span><span style="color: #008000"><!--</span><span style="color: #008000"> 默认是0 </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="count"</span><span style="color: #0000ff">></span>2<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span><span style="color: #008000"><!--</span><span style="color: #008000"> 要分片的数据库节点数量,必须指定,否则没法分片 </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="virtualBucketTimes"</span><span style="color: #0000ff">></span>160<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span><span style="color: #008000"><!--</span><span style="color: #008000"> 一个实际的数据库节点被映射为这么多虚拟节点,默认是160倍,也就是虚拟节点数是物理节点数的160倍 </span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <property name="weightMapFile">weightMapFile</property> 节点的权重,没有指定权重的节点默认是1。以properties文件的格式填写,以从0开始到count-1的整数值也就是节点索引为key,以节点权重值为值。所有权重值必须是正整数,否则以1代替 </span><span style="color: #008000">--></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> <property name="bucketMapPath">/etc/mycat/bucketMapPath</property>
用于测试时观察各物理节点与虚拟节点的分布情况,如果指定了这个属性,会把虚拟节点的murmur hash值与物理节点的映射按行输出到这个文件,没有默认值,如果不指定,就不会输出任何东西 </span><span style="color: #008000">--></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="crc32slot"</span><span style="color: #ff0000">
class</span><span style="color: #0000ff">="io.mycat.route.function.PartitionByCRC32PreSlot"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="count"</span><span style="color: #0000ff">></span>2<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span><span style="color: #008000"><!--</span><span style="color: #008000"> 要分片的数据库节点数量,必须指定,否则没法分片 </span><span style="color: #008000">--></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="hash-int"</span><span style="color: #ff0000">
class</span><span style="color: #0000ff">="io.mycat.route.function.PartitionByFileMap"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="mapFile"</span><span style="color: #0000ff">></span>partition-hash-int.txt<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="rang-long"</span><span style="color: #ff0000">
class</span><span style="color: #0000ff">="io.mycat.route.function.AutoPartitionByLong"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="mapFile"</span><span style="color: #0000ff">></span>autopartition-long.txt<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="mod-long"</span><span style="color: #ff0000"> class</span><span style="color: #0000ff">="io.mycat.route.function.PartitionByMod"</span><span style="color: #0000ff">></span>
<span style="color: #008000"><!--</span><span style="color: #008000"> how many data nodes </span><span style="color: #008000">--></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="count"</span><span style="color: #0000ff">></span>3<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="func1"</span><span style="color: #ff0000"> class</span><span style="color: #0000ff">="io.mycat.route.function.PartitionByLong"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="partitionCount"</span><span style="color: #0000ff">></span>8<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="partitionLength"</span><span style="color: #0000ff">></span>128<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="latestMonth"</span><span style="color: #ff0000">
class</span><span style="color: #0000ff">="io.mycat.route.function.LatestMonthPartion"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="splitOneDay"</span><span style="color: #0000ff">></span>24<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="partbymonth"</span><span style="color: #ff0000">
class</span><span style="color: #0000ff">="io.mycat.route.function.PartitionByMonth"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="dateFormat"</span><span style="color: #0000ff">></span>yyyy-MM-dd<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="sBeginDate"</span><span style="color: #0000ff">></span>2015-01-01<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="rang-mod"</span><span style="color: #ff0000"> class</span><span style="color: #0000ff">="io.mycat.route.function.PartitionByRangeMod"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="mapFile"</span><span style="color: #0000ff">></span>partition-range-mod.txt<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">function </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="jump-consistent-hash"</span><span style="color: #ff0000"> class</span><span style="color: #0000ff">="io.mycat.route.function.PartitionByJumpConsistentHash"</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"><</span><span style="color: #800000">property </span><span style="color: #ff0000">name</span><span style="color: #0000ff">="totalBuckets"</span><span style="color: #0000ff">></span>3<span style="color: #0000ff"></</span><span style="color: #800000">property</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></</span><span style="color: #800000">function</span><span style="color: #0000ff">></span>
<span style="color: #0000ff"></
<span style="color: #800000">mycat:rule<span style="color: #0000ff">>rule是定义分片规则的
三、mycat工作原理
四、mycat基础概念
1. mycat支持的数据库
2. 逻辑库(Schema)
在实际应用中,业务开发人员并不需要知道中间件的存在,只需要知道数据库的概念,所以数据库中间件可以被看作是一个或者多个数据库集群构成的逻辑库。
3. 逻辑表
在分布式数据库中,读写数据的表就是逻辑表。逻辑表,可以是数据切分后分布在一个或多个分片库中,也可以不做数据切分不分片,只有一个表构成
4. DataNode、DataHost、Rule
DataNode:数据切分后,一个大表被分到不同的分片数据库上面,每个分片所在的数据库就是分片节点
DataHost:数据切分后,每个分片节点(DataNode)不一定都会独占一台机器,同一机器上面可以有多个分片数据库。一个或多个分片节点(DataNode)所在的机器就是节点主机(DataHost)
Rule:分片规则,选择合适的分片规则很重要,将极大的避免后续处理数据的难度
5. myca分片原则
5.1 数据拆分之前的思考
分片规则虽然能解决大表对数据库系统的压力,但它并不是万能的,也有一些不利之处。因此首要问题是:
1)分不分库
2)分哪些库
3)什么规则分
4)分多少片
5.2 mycat数据切分原则
1)达到一定数量级才分,一般是单表达到800万数据就要考虑数据拆分了。
2)不到800万,但跟大表(超800万的表)有关联查询的表也要拆分,在此称为大表关联表
3)大表关联表如何拆分:小于100万的使用全局表;大于100万小于800万跟大表使用同样的拆分策略;无法跟大表使用相同规则的,可以考虑从java代码上分步骤查询,不用关联查询,或者破例使用全局表。
4)全局表:如品牌表250万,跟大表order关联,又无法跟大表使用同样的拆分策略,也做成了全局表。全局表必须满足的条件:没有太激烈的并发update(如多线程同时update同一条记录);虽有多线程update,但不是操作同一条记录;批量insert没问题。
5)拆分字段是不可修改的
6)拆分字段只能是一个字段,如果想按照两个字段拆分,必须新建一个冗余字段,冗余字段的值使用两个字段的值拼接而成(如大区+年月拼接成:“区域_年月日” 字段)
7)拆分算法的选择和合理性判断:按照选定的算法拆分后每个库中单表不得超过800万记录
8)能不拆分的尽量不拆分。如果某个表不跟其他表关联查询,数据量又少、直接不拆分,使用单库即可
5.3 Mycat分库分表原则
1)能不分就不分,1000万以内的表,不建议分片,通过合适的索引,读写分离等方式,可以很好的解决性能问题
2)分片数量尽量少,分片尽量均匀分布在多个DataHost上,因为一个查询SQL跨分片越多,则总体性能越差,虽然要好于所有数据在一个分片的结果,只在必要的时候进行扩容,增加分片数量。
3)分片规则需要慎重选择,分片规则的选择,需要考虑数据的增长模式,数据的访问模式,分片关联性问题,以及分片扩容问题,最近的分片策略为范围分片,枚举分片,一致性Hash分片,这几种分片都有利于扩容
4)尽量不要在一个事物中的SQL跨越多个分片,分布式事物一直是一个不好处理的问题
5)查询条件尽量优化,尽量避免select * 的方式,大量数据结果集下,会消耗大量的CPU资源和带宽,查询应尽量避免返回大的结果集,并且应尽量为频繁使用的查询语句建立索引
5.4 mycat常用分片规则
1)时间类:按天分片、自然月分片、单月小时分片
2)哈希类:Hash固定分片、日期范围Hash分片、截取数字Hash求模范围分片、截取数字Hash分片、一致性Hash分片
3)取模类:取模分片、取模范围分片、范围求模分片
4)其他类:枚举分片、范围约定分片、应用指定分片、冷热数据分片