将表从 Azure Databricks 迁移到数据仓库 (Azure Synapse)

问题描述

我正在将一个表从 Azure Databricks 迁移到数据仓库 (Azure Synapse) 中的一个表。

集群有这样的配置： 火花：3.0.1 标度：2.12

迁移代码为Scala，如下：

...
val exitValue = exitValueGenerator(List("global_temp.wet_yield_temp"))
val delta = Entity(RenamedTable("global_temp","wet_yield_temp","sensor_damaged","wet_yield_sensor_damaged_combine_history"),None,Some(FullReload),None)
var res = Source.migrate(deltalake,dwh,List(delta))

我有一条错误消息：

Processing global_temp.wet_yield_temp...
SELECT * FROM global_temp.wet_yield_temp
Moving 54 records...
delta: Entity = Entity(RenamedTable(global_temp,wet_yield_temp,sensor_damaged,wet_yield_sensor_damaged_combine_history),None)
res: Iterable[scala.util.Try[String]] =
List(Failure(com.databricks.spark.sqldw.sqlDWSideException: Azure Synapse Analytics Failed to execute the JDBC query produced by the connector.
Underlying sqlException(s):
  - com.microsoft.sqlserver.jdbc.sqlServerException: HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: IllegalArgumentException: Must be 12 bytes [ErrorCode = 106000] [sqlState = S0001]
         ))

为什么我有这个错误？

Databricsk 中表 "wet_yield_temp" 的架构是：

unitsn:string
brand:string
type:string
model:string
date:date
time_start:timestamp
time_end:timestamp
alert:integer
duration_mins:double
severity:integer
three_days_severity:integer
triggerName:string

Azure Synapse 中目标表“sensor_damaged.wet_yield_sensor_damaged_combine_history”的架构是：

CREATE TABLE [sensor_damaged].[wet_yield_sensor_damaged_combine_history]
(
    [unitsn] [nvarchar](250) NULL,[brand] [nvarchar](250) NULL,[type] [nvarchar](250) NULL,[model] [nvarchar](250) NULL,[date] [date] NULL,[time_start] [datetime] NULL,[time_end] [datetime] NULL,[alert] [int] NULL,[duration_mins] [float] NULL,[severity] [int] NULL,[three_days_severity] [int] NULL,[triggerName] [nvarchar](150) NULL
)

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）