错误：bulkCopyToSqlDB不是org.apache.spark.sql.DataFrameWriter的成员

问题描述

使用以下链接从Databricks将数据加载到sql DB时出现以下错误：

command-3227900948916301:23: error: value bulkcopyTosqlDB is not a member of org.apache.spark.sql.DataFrameWriter[org.apache.spark.sql.Row] df.write.mode(SaveMode.Overwrite).bulkcopyTosqlDB(bulkcopyConfig)

https://github.com/Azure/azure-sqldb-spark/blob/fa1cf19ed797648a20d9b7f474d7c2cd88829ada/samples/scripts/BulkCopySample.scala

我的代码如下：

val bulkcopyConfig  = Config(Map(
  "url"          -> url,"databaseName" -> databaseName,"dbTable"      -> "dbo.xxxx","user"         -> user,"password"     -> password,"connectTimeout" -> "120","bulkcopyBatchSize" -> "100000","bulkcopyTableLock" -> "true","bulkcopyTimeout"   -> "0","truncate" -> "true"
//  "queryTimeout"   -> "5"
))

//~

df. write.mode(SaveMode.Overwrite).bulkcopyTosqlDB(bulkcopyConfig)

Any thoughts on why I'm getting the error?

解决方法

您需要正确的导入才能使用其他功能扩展DataFrame：

import com.microsoft.azure.sqldb.spark.connect._

这是我得到的（失败，因为我没有活动的SQL DB，但它找到了功能）：

scala> import com.microsoft.azure.sqldb.spark.config.Config
import com.microsoft.azure.sqldb.spark.config.Config

scala> import com.microsoft.azure.sqldb.spark.connect._
import com.microsoft.azure.sqldb.spark.connect._

scala> val bulkCopyConfig = Config(Map(
     |   "url"               -> "mysqlserver.database.windows.net",|   "databaseName"      -> "MyDatabase",|   "user"              -> "username",|   "password"          -> "*********",|   "dbTable"           -> "dbo.Clients",|   "bulkCopyBatchSize" -> "2500",|   "bulkCopyTableLock" -> "true",|   "bulkCopyTimeout"   -> "600"
     | ))
bulkCopyConfig: com.microsoft.azure.sqldb.spark.config.Config = com.microsoft.azure.sqldb.spark.config.ConfigBuilder$$anon$1@754443e1

scala> val df = spark.range(1,10)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> df.bulkCopyToSqlDB(bulkCopyConfig)
20/10/06 16:24:38 ERROR DataFrameFunctions: Connection cannot be established to the database
com.microsoft.sqlserver.jdbc.SQLServerException: Login failed for user 'username'. ClientConnectionId:b35aa6bb-24ec-4727-98b6-074cae4e6a32
...