将历史事件发布到 ATSv2 时出错

问题描述

我们正在尝试将历史事件发布到 ATSv2。我们启用了 yarn.timeline-service.version 2.0f。

我们已经成功地将 jira https://issues.apache.org/jira/browse/TEZ-3820 中提到的补丁应用于 tez-0.9.2 源代码。

if (conf.getBoolean(YarnConfiguration.TIMELINE_SERVICE_ENABLED,YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ENABLED)) {

  timelineV2Client =
  TimelineV2Client.createTimelineClient(appContext.getApplicationID());
  timelineV2Client.init(conf);

  // assumption is at this point AMRMClient is created! So callback handler is set to null
  TezAMRMClientAsyncProvider.createAMRMClientAsync(1000,null)
      .registerTimelineV2Client(timelineV2Client);

  LOG.info("timelineV2Client inited.");
} else {
  this.timelineV2Client = null;
  if (conf.get(TezConfiguration.TEZ_HISTORY_LOGGING_SERVICE_CLASS,"")
      .equals(ATSV2HistoryLoggingService.class.getName())) {
    LOG.warn(ATSV2HistoryLoggingService.class.getName()
        + " is disabled due to Timeline Service being disabled,"
        + YarnConfiguration.TIMELINE_SERVICE_ENABLED + " set to false");
 }

当我们使用以下命令运行 tez 作业时

    hadoop jar /usr/lib/tez/tez-tests-0.9.2.jar mrrsleep -m 10 -r 20

我们不断收到空指针异常以及 dag await，如下所示

ag.app.web.WebUIService] |http.HttpServer2|: Jetty bound to port 34923
 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService]|mortbay.log|: jetty-6.1.26
[INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService]|mortbay.log|: Extract jar:file:/usr/lib/hadoop/hadoop-common-2.10.1-tests.jar!/webapps/ to /hadoop/yarn/nm-local-dir/usercache/yadavsanjeev/appcache/application_1620837070665_0001/container_e01_1620837070665_0001_01_000001/tmp/Jetty_0_0_0_0_34923_webapps____fsve58/webapp
 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |mortbay.log|: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:34923
 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |webapp.WebApps|: Web app  started at 34923
 [INFO] [ServiceThread:org.apache.tez.dag.app.web.WebUIService] |web.WebUIService|: Instantiated WebUIService at http://mysandbox.internal:34923/ui/
[INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.TaskSchedulerManager|: Creating YARN TaskScheduler: org.apache.tez.dag.app.rm.YarnTaskSchedulerService
 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |conf.Configuration|: resource-types.xml not found
 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |resource.ResourceUtils|: Unable to find 'resource-types.xml'.
 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |resource.ResourceUtils|: Adding resource type - name = memory-mb,units = Mi,type = COUNTABLE
[INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |resource.ResourceUtils|: Adding resource type - name = vcores,units =,type = COUNTABLE
 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |Configuration.deprecation|: io.bytes.per.checksum is deprecated. Instead,use dfs.bytes-per-checksum
 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |rm.YarnTaskSchedulerService|: YarnTaskScheduler initialized with configuration: maxRMHeartbeatInterval: 1000,containerReuseEnabled: true,reuseRackLocal: true,reuseNonLocal: false,localitySchedulingDelay: 250,preemptionPercentage: 10,preemptionMaxWaitTime: 60000,numHeartbeatsBetweenPreemptions: 3,idleContainerMinTimeout: 5000,idleContainerMaxTimeout: 10000,sessionMinHeldContainers: 0
 [INFO] [ServiceThread:org.apache.tez.dag.app.rm.TaskSchedulerManager] |client.RMProxy|: Connecting to ResourceManager at mysandbox/10.0.0.79:8030
 [INFO] [main] |HistoryEventHandler.criticalEvents|: [HISTORY][DAG:N/A][Event:AM_STARTED]: appAttemptId=appattempt_1620837070665_0001_000001,startTime=1620837589967
 [INFO] [main] |app.DAGAppMaster|: In Non-Session mode.
 [INFO] [ATSV2HistoryEventHandlingThread] |ats.ATSV2HistoryLoggingService|: Publishing entity TimelineEntity[type='TEZ_APPLICATION_ATTEMPT',id='tez_appattempt_1620837070665_0001_000001']
 [INFO] [main] |app.DAGAppMaster|: Merging AM credentials into DAG credentials
 [ERROR] [AMRM Callback Handler Thread] |yarn.YarnUncaughtExceptionHandler|: Thread Thread[AMRM Callback Handler Thread,5,main] threw an Exception.
java.lang.NullPointerException
at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:371)
 [INFO] [main] |app.DAGAppMaster|: Generating DAG graphviz file,dagId=dag_1620837070665_0001_1,filePath=/var/log/hadoop-yarn/userlogs/application_1620837070665_0001/container_e01_1620837070665_0001_01_000001/dag_1620837070665_0001_1.dot
 [INFO] [main] |common.TezUtilsInternal|: Redirecting log file based on addend: dag_1620837070665_0001_1

如jira中提到的，假设此时AMRMClient已创建，因此回调处理程序设置为null，但不清楚这里是如何使用的。

因此时间线服务地址为空，无法发布事件。

解决方法

暂无找到可以解决该程序问题的有效方法，小编努力寻找整理中！

如果你已经找到好的解决方法，欢迎将解决方案带上本链接一起发送给小编。

小编邮箱:dio#foxmail.com (将#修改为@）