问题描述
我想使用 sqoop 将 MysqL 数据传输到 HDFS。 ipv4 MysqL运行良好，但是无法连接ipv6 MysqL。
我确定 MysqL 和 sqoop 没问题。 sqoop 使用这样的图像：
并以这样的命令开始：
sudo docker run --network host -v /tmp/jdbc:/jdbc -it dvoros/sqoop:latest
司机是mysql-connector-java-8.0.19.jar
连接到 MysqL，因为 IPV6 是可以的。
Sqoop 连接 MysqL 作为 IPV4 没问题。
sqoop list-tables --connect jdbc:MysqL://localhost:3306/test --username root --password root
但是 Sqoop 连接 MysqL 作为 IPV6 是错误的。
sqoop list-tables --connect jdbc:MysqL://[::1]:3306/test --username root --password root
更多错误信息：
[root@test01 /]# sqoop list-tables --connect jdbc:MysqL://[::1]:3306/test --username root --password root
Warning: /usr/local/sqoop/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /usr/local/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/local/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /usr/local/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
2021-02-19 08:05:47,048 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
2021-02-19 08:05:47,077 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
2021-02-19 08:05:47,175 INFO manager.MysqLManager: Preparing to use a MysqL streaming resultset.
Loading class `com.MysqL.jdbc.Driver'. This is deprecated. The new driver class is `com.MysqL.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2021-02-19 08:05:47,423 ERROR manager.CatalogQueryManager: Failed to list tables
com.MysqL.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at com.MysqL.cj.jdbc.exceptions.sqlError.createCommunicationsException(sqlError.java:174)
at com.MysqL.cj.jdbc.exceptions.sqlExceptionsMapping.translateException(sqlExceptionsMapping.java:64)
at com.MysqL.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
at com.MysqL.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
at com.MysqL.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
at com.MysqL.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.manager.sqlManager.makeConnection(sqlManager.java:904)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
at org.apache.sqoop.manager.CatalogQueryManager.listTables(CatalogQueryManager.java:102)
at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: com.MysqL.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.MysqL.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.MysqL.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.MysqL.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.MysqL.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.MysqL.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:91)
at com.MysqL.cj.NativeSession.connect(NativeSession.java:144)
at com.MysqL.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:956)
at com.MysqL.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:826)
... 15 more
Caused by: java.net.socketException: Protocol family unavailable
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.socksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.socket.connect(Socket.java:589)
at com.MysqL.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:155)
at com.MysqL.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:65)
... 18 more
2021-02-19 08:05:47,427 ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.RuntimeException: com.MysqL.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
java.lang.RuntimeException: com.MysqL.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at org.apache.sqoop.manager.CatalogQueryManager.listTables(CatalogQueryManager.java:118)
at org.apache.sqoop.tool.ListTablesTool.run(ListTablesTool.java:49)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: com.MysqL.cj.jdbc.exceptions.CommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at com.MysqL.cj.jdbc.exceptions.sqlError.createCommunicationsException(sqlError.java:174)
at com.MysqL.cj.jdbc.exceptions.sqlExceptionsMapping.translateException(sqlExceptionsMapping.java:64)
at com.MysqL.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
at com.MysqL.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
at com.MysqL.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
at com.MysqL.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.manager.sqlManager.makeConnection(sqlManager.java:904)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
at org.apache.sqoop.manager.CatalogQueryManager.listTables(CatalogQueryManager.java:102)
... 7 more
Caused by: com.MysqL.cj.exceptions.CJCommunicationsException: Communications link failure
The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.MysqL.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.MysqL.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
at com.MysqL.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
at com.MysqL.cj.exceptions.ExceptionFactory.createCommunicationsException(ExceptionFactory.java:167)
at com.MysqL.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:91)
at com.MysqL.cj.NativeSession.connect(NativeSession.java:144)
at com.MysqL.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:956)
at com.MysqL.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:826)
... 15 more
Caused by: java.net.socketException: Protocol family unavailable
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.socksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.socket.connect(Socket.java:589)
at com.MysqL.cj.protocol.StandardSocketFactory.connect(StandardSocketFactory.java:155)
at com.MysqL.cj.protocol.a.NativeSocketConnection.connect(NativeSocketConnection.java:65)
... 18 more
解决方法
添加System.setProperty("java.net.preferIPv4Stack","false");
在 SqlManager.makeConnection 之前修复它。
如果你想将 MySQL 表导入到 hdfs。
应该设置
conf.set("mapreduce.map.java.opts","-Djava.net.preferIPv4Stack=false");