canal实时同步mysql数据到mysql/hbase

一、 下载canal

分别下载 canal.deployer-1.1.4 和 canal.adapter-1.1.4

二、安装canal

1. 分别解压canal.deployer-1.1.4和canal.adapter-1.1.4

mkdir /software
tar -xzvf canal.deployer-1.1.4.tar.gz -C /software/canal
tar -xzvf canal.deployer-1.1.4.tar.gz -C /software/canal-adapter-1.1.4

2. mysql相关配置

① mysql开启binlog，修改 my.cnf 

[mysqld]
log-bin=mysql-bin # 开启 binlog
binlog-format=ROW # 选择 ROW 模式
server_id=1 # 配置 MySQL replaction 需要定义，不要和 canal 的 slaveId 重复

查看是否成功

show variables like '%log_bin%'

② 创建canal用户，并赋权

# 创建用户
CREATE USER canal IDENTIFIED BY 'Canal@123456';
# mysql8.0需要在创建用户后，先更改密码加密方式，再赋权
update user set plugin='mysql_native_password' where user='canal'; 
grant all privileges ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;

③ 查看binlog偏移量并记录

show master status;

④ 导出目标表历史数据，并在目标库创建表并导入 (请自行操作，注意时间格式)

3. 配置canal.deployer-1.1.4

配置instance.properties
注意：/software/canal/conf/example路径下，首次启动或者数据库变更后，请将此路径下除instance.properties以外的文件删除，否则可能报错

vim /software/canal/conf/example/instance.properties

## mysql serverId , v1.0.26+ will autoGen
## v1.0.26版本后会自动生成slaveId，所以可以不用配置
# canal.instance.mysql.slaveId=0

# 源数据库地址，此处请修改
canal.instance.master.address=127.0.0.1:3306
# binlog日志名称，此处请修改
canal.instance.master.journal.name=mysql-bin.000005
# mysql主库链接时起始的binlog偏移量，此处请修改
canal.instance.master.position=820633026
# mysql主库链接时起始的binlog的时间戳
canal.instance.master.timestamp=
canal.instance.master.gtid=

# username/password
# 在MySQL服务器授权的账号密码，此处请修改
canal.instance.dbUsername=canal
canal.instance.dbPassword=Canal@123456
# 字符集
canal.instance.connectionCharset = UTF-8
# enable druid Decrypt database password
canal.instance.enableDruid=false

# table regex .*\\..*表示监听所有表 也可以写具体的表名，用，隔开
canal.instance.filter.regex=.*\\..*
# mysql 数据解析表的黑名单，多个表用，隔开
canal.instance.filter.black.regex=

4. 配置canal.adapter-1.1.4

① 修改application.yml

vim /software/canal-adapter-1.1.4/conf/application.yml

server:
  port: 8081
spring:
  jackson:
    date-format: yyyy-MM-dd HH:mm:ss
    time-zone: GMT+8
    default-property-inclusion: non_null

canal.conf:
  mode: tcp # kafka rocketMQ
  canalServerHost: 10.10.10.10:11111 #自己的canal-deployer启动的ip和端口,11111为默认端口
#  zookeeperHosts: slave1:2181
#  mqServers: 127.0.0.1:9092 #or rocketmq
#  flatMessage: true
  batchSize: 500
  syncBatchSize: 1000
  retries: 0
  timeout:
  accessKey:
  secretKey:
  srcDataSources:
    defaultDS:
        # 换成自己mysql源库的地址，此处请修改
      url: jdbc:mysql://127.0.0.1:3306/person?useUnicode=true
      username: canal
      password: Canal@123456
  canalAdapters:
  - instance: example # canal instance Name or mq topic name
    groups:
    - groupId: g1
      outerAdapters:
 #     - name: logger
      - name: rdb
        key: mysql1
        properties:
          # 驱动类，如果是mysql8.0，需要在lib中添加mysql-connector-java-8.0.27.jar，请自行搜索下载
          jdbc.driverClassName: com.mysql.jdbc.Driver
          # 目标库的地址，此处请修改
          jdbc.url: jdbc:mysql://8.8.8.1:3306/person?useUnicode=true
          jdbc.username: root
          jdbc.password: [e-mail]
    - groupId: g2
      outerAdapters:
      - name: hbase
        properties:
          # zookeeper地址，此处请修改
          hbase.zookeeper.quorum: 8.8.8.5,8.8.8.6,8.8.8.7
          hbase.zookeeper.property.clientPort: 2181
          zookeeper.znode.parent: /hbase

② 添加mysql同步配置文件，每张表新建一个文件，名称随意

vim /software/canal-adapter-1.1.4/conf/rdb/teacher.yml

dataSourceKey: defaultDS
destination: example
groupId: g1
outerAdapterKey: mysql1
concurrent: true
dbMapping:
  database: person        #源数据库
  table: teacher   #待同步表
  targetTable: teacher     #目标表
  targetPk:     #主键映射，左侧为源表主键，右侧为目标表主键
    id: id
  mapAll: true  #全同步
  commitBatch: 3000 # 批量提交的大小

vim /software/canal-adapter-1.1.4/conf/rdb/student.yml

dataSourceKey: defaultDS
destination: example
groupId: g1
outerAdapterKey: mysql1
concurrent: true
dbMapping:
  database: person        #源数据库
  table: student   #待同步表
  targetTable: student     #目标表
  targetPk:     #主键映射，左侧为源表主键，右侧为目标表主键
    id: id
  mapAll: true  #全同步
  commitBatch: 3000 # 批量提交的大小

③ 修改hbase同步配置

dataSourceKey: defaultDS
destination: example
hbaseMapping:
  mode: STRING  #NATIVE   #PHOENIX
  database: person  # 数据库名
  table: student     # 数据库表名
  hbaseTable: test:student   # HBase表名
  family: info  # 默认统一Family名称
  uppercaseQualifier: false  # 字段名转大写, 默认为true
  commitBatch: 3000 # 批量提交的大小
  rowKey: vin  # 复合字段rowKey不能和columns中的rowKey重复
  columns:
   id: info:id  
   name: info:name
  excludeColumns: # 忽略字段
   - score

三、启动

1. 启动canal.deployer-1.1.4

# 启动
sh /software/canal/bin/startup.sh
# 重启
sh /software/canal/bin/restart.sh
# 停止
sh /software/canal/bin/stop.sh

2. 启动canal.adapter-1.1.4

# 启动
sh /software/canal-adapter-1.1.4/bin/startup.sh
# 重启
sh /software/canal-adapter-1.1.4/bin/restart.sh
# 停止
sh /software/canal-adapter-1.1.4/bin/stop.sh

3. 查看log

# 查看canal.deployer-1.1.4运行log
tail -100f /software/canal/logs/example/example.log
# 查看canal.adapter-1.1.4运行log
tail -100f /software/canal-adapter-1.1.4/logs/adapter/adapter.log

四、问题

1. 用户赋权报错

SHOW GRANTS FOR 'canal'@'%' > 1141 - There is no such grant defined for user 'canal' on host '%'

解决方法：删除用户，重新创建，赋权

drop user 'canal'@'%';
CREATE USER canal IDENTIFIED BY 'Canal@123456';
# mysql8.0需要在创建用户后，先更改密码加密方式，再赋权
update user set plugin='mysql_native_password' where user='canal'; 
grant all privileges ON *.* TO 'canal'@'%';
FLUSH PRIVILEGES;