Delta Lake：org.apache.spark.sql.AnalysisException：找不到表或视图：default.HISTORY；；

编程问答 2022-05-23

问题描述

技术栈-

火花：3.1.0

增量：0.7.0

斯卡拉：2.12.8

我正在创建我的 Delta 表 -（在 AWS EMR 上，即 emr-6.2.0 ）

val spark = SparkSession.builder()
              .appName("Spark3_Delta_Dema")
              .master("local[*]")
              .config("spark.sql.extensions","io.delta.sql.DeltaSparkSessionExtension")
              .config("spark.sql.catalog.spark_catalog","org.apache.spark.sql.delta.catalog.DeltaCatalog")
              .getorCreate()

spark.sql("CREATE OR REPLACE TABLE delta_table_name ( column1 STRING,column2 Binary,column3 BIGINT) USING DELTA PARTITIONED BY (column3) LOCATION 's3a://...........'")

表格创建正确。

但是当我尝试运行 DESCRIBE HISTORY 查询时出现问题。

我试过了 -

DESCRIBE HISTORY delta.`s3 location`,

描述历史记录delta.delta_table_name，

DESCRIBE HISTORY delta_table_name,

描述历史`s3位置`

但以上所有问题都导致了同样的问题。

错误日志 -

An error was encountered:
org.apache.spark.sql.AnalysisException: Table or view not found: default.history;;
'UnresolvedRelation [default,history]

  at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1(CheckAnalysis.scala:106)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.$anonfun$checkAnalysis$1$adapted(CheckAnalysis.scala:92)
  at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:188)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis(CheckAnalysis.scala:92)
  at org.apache.spark.sql.catalyst.analysis.CheckAnalysis.checkAnalysis$(CheckAnalysis.scala:89)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.checkAnalysis(Analyzer.scala:130)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.$anonfun$executeAndCheck$1(Analyzer.scala:156)
  at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper$.markInAnalyzer(AnalysisHelper.scala:201)
  at org.apache.spark.sql.catalyst.analysis.Analyzer.executeAndCheck(Analyzer.scala:153)
  at org.apache.spark.sql.execution.QueryExecution.$anonfun$analyzed$1(QueryExecution.scala:68)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker.measurePhase(QueryPlanningTracker.scala:149)
  at org.apache.spark.sql.execution.QueryExecution.$anonfun$executePhase$1(QueryExecution.scala:153)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.execution.QueryExecution.executePhase(QueryExecution.scala:153)
  at org.apache.spark.sql.execution.QueryExecution.analyzed$lzycompute(QueryExecution.scala:68)
  at org.apache.spark.sql.execution.QueryExecution.analyzed(QueryExecution.scala:66)
  at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:58)
  at org.apache.spark.sql.Dataset$.$anonfun$ofRows$1(Dataset.scala:92)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:90)
  at org.apache.spark.sql.SparkSession.table(SparkSession.scala:589)
  at org.apache.spark.sql.execution.command.DescribeColumnCommand.run(tables.scala:773)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
  at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
  at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:230)
  at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3667)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:104)
  at org.apache.spark.sql.execution.sqlExecution$.withTracker(sqlExecution.scala:227)
  at org.apache.spark.sql.execution.sqlExecution$.executeQuery$1(sqlExecution.scala:107)
  at org.apache.spark.sql.execution.sqlExecution$.$anonfun$withNewExecutionId$6(sqlExecution.scala:132)
  at org.apache.spark.sql.catalyst.QueryPlanningTracker$.withTracker(QueryPlanningTracker.scala:104)
  at org.apache.spark.sql.execution.sqlExecution$.withTracker(sqlExecution.scala:227)
  at org.apache.spark.sql.execution.sqlExecution$.$anonfun$withNewExecutionId$5(sqlExecution.scala:132)
  at org.apache.spark.sql.execution.sqlExecution$.withsqlConfPropagated(sqlExecution.scala:248)
  at org.apache.spark.sql.execution.sqlExecution$.$anonfun$withNewExecutionId$1(sqlExecution.scala:131)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.execution.sqlExecution$.withNewExecutionId(sqlExecution.scala:68)
  at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3665)
  at org.apache.spark.sql.Dataset.<init>(Dataset.scala:230)
  at org.apache.spark.sql.Dataset$.$anonfun$ofRows$2(Dataset.scala:101)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:98)
  at org.apache.spark.sql.SparkSession.$anonfun$sql$1(SparkSession.scala:607)
  at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:764)
  at org.apache.spark.sql.SparkSession.sql(SparkSession.scala:602)
  ... 53 elided

更新：-

当我做这样的事情时 ->

val deltaTable = DeltaTable.forPath(spark1,"s3a:..................")
val fullHistory = deltaTable.history()
fullHistory.show()

然后我就可以看到结果了。

提前致谢。

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）

amazon-emrapache-sparkdelta-lakescalascala