问题描述
更新:
VNC 进入构建机器(不做任何更改)以某种方式修复了这个......奇怪
背景
我们使用 Mac EC2 实例 CI 集群来运行 UI 测试。集群设置为每晚回收。这意味着新集群是在早上配置的。我们注意到测试在一天中变得越来越不稳定。
- 构建 312 是早上的第一个构建。
- 红色是失败的测试,橙色是静音测试,黄色是片状测试,灰色是未知状态
- 提交只是触发构建的空白自动提交。测试内容相同。
- 注意:Bluepill 配置为每台机器运行一个模拟器(无并发)
日志
片状测试或失败通常是由于“等待某些资源空闲”并达到超时,或某些网络请求完成。失败或不稳定的测试也有一个共同点,即它们通常与视频有关
这里有几个示例日志
等待应用空闲
Timed out waiting for app to idle.
The following idling resources are busy.
1. GREYAppStateTracker:
Waiting for network requests to finish. By default,EarlGrey tracks all network requests. To change this behavior,refer to GREYConfiguration.
<__NSCFLocalDataTask:0x7fcfcfd87450,URL:"https://v.myorgimg.com/videos/mc/hls/e7/01/ca/e701caebac12684056eb699f316fc411_360w.m3u8"> => Waiting for network requests to finish. By default,refer to GREYConfiguration.
(
0 MyorgDevelopmentEG2 0x000000010ebf3205 -[GREYAppStateTrackerObject setState:] + 69
1 MyorgDevelopmentEG2 0x000000010ebf22b3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
2 MyorgDevelopmentEG2 0x000000010ebf18f0 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
3 MyorgDevelopmentEG2 0x000000010ebf1e34 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:] + 1076
4 MyorgDevelopmentEG2 0x000000010ebf0ac6 -[GREYAppStateTracker trackState:forObject:] + 102
5 MyorgDevelopmentEG2 0x000000010ebca867 -[__NSCFLocalDataTask_GREYApp grey_track] + 135
6 MyorgDevelopmentEG2 0x000000010ebcac55 -[__NSCFLocalDataTask_GREYApp greyswizzled_resume] + 181
7 MediaToolBox 0x00007fff2b617285 figHttpRequestSetupNSURLSessionTask + 1380
8 MediaToolBox 0x00007fff2b616bb6 _figHTTPRequestCreateWithNSURLSession + 2923
9 MediaToolBox 0x00007fff2b64aaaf figHTTPRequestSessionNSCreateHTTPRequest + 199
10 MediaToolBox 0x00007fff2b71f5b1 segPumpCreateHTTPRequest + 944
11 MediaToolBox 0x00007fff2b721de6 segPumpSendindexFileRequest + 1703
12 MediaToolBox 0x00007fff2b7431d5 segPumpRequestIndexForStream + 753
13 MediaToolBox 0x00007fff2b7421ee segPumpSetAlternateForStream + 3145
14 MediaToolBox 0x00007fff2b73e9e5 segPumpSetCurrentAlternate + 9898
15 MediaToolBox 0x00007fff2b791d6a ProduceStreamingAssetProperty + 661
16 MediaToolBox 0x00007fff2b78e07c URLAssetPropertyWorkFunction + 872
17 libdispatch.dylib 0x00007fff201078df _dispatch_client_callout + 8
18 libdispatch.dylib 0x00007fff2010de15 _dispatch_lane_serial_drain + 715
19 libdispatch.dylib 0x00007fff2010e9c3 _dispatch_lane_invoke + 455
20 libdispatch.dylib 0x00007fff20117a8e _dispatch_root_queue_drain + 350
21 libdispatch.dylib 0x00007fff2011786c _dispatch_worker_thread + 222
22 libsystem_pthread.dylib 0x00007fff60342950 _pthread_start + 224
23 libsystem_pthread.dylib 0x00007fff6033e47b thread_start + 15
)
等待网络请求完成
9. GREYAppStateTracker:
Waiting for network requests to finish. By default,refer to GREYConfiguration.
Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
<ASCollectionView: 0x7feca21ad000> => Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
(
0 MyorgDevelopmentEG2 0x000000010bb2e335 -[GREYAppStateTrackerObject setState:] + 69
1 MyorgDevelopmentEG2 0x000000010bb2d3e3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
2 MyorgDevelopmentEG2 0x000000010bb2ca20 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
3 MyorgDevelopmentEG2 0x000000010bb2cf64 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:] + 1076
4 MyorgDevelopmentEG2 0x000000010bb2bbf6 -[GREYAppStateTracker trackState:forObject:] + 102
5 MyorgDevelopmentEG2 0x000000010bb02b0d -[UIView(GREYApp) greyswizzled_setNeedsLayout] + 93
6 UIKitCore 0x00007fff24ba5675 -[UIScrollView setNeedsLayout] + 79
7 UIKitCore 0x00007fff23dc0390 -[UICollectionView _updateAnimationDidStop:finished:context:] + 2236
8 UIKitCore 0x00007fff23dbf9e4 __102-[UICollectionView _updateWithItems:tentativelyForReordering:propertyAnimator:collectionViewAnimator:]_block_invoke.2110 + 77
9 MyorgDevelopmentEG2 0x000000010bb2fdef __60-[GREYdispatchQueueTracker grey_dispatchAsyncCallWithBlock:]_block_invoke + 47
10 libdispatch.dylib 0x00007fff2010670d _dispatch_call_block_and_release + 12
11 libdispatch.dylib 0x00007fff201078df _dispatch_client_callout + 8
12 libdispatch.dylib 0x00007fff20114a27 _dispatch_main_queue_callback_4CF + 1045
13 CoreFoundation 0x00007fff203908f8 __CFRUNLOOP_IS_SERVICING_THE_MAIN_disPATCH_QUEUE__ + 9
14 CoreFoundation 0x00007fff2038b169 __CFRunLoopRun + 2781
15 CoreFoundation 0x00007fff2038a1a7 CFRunLoopRunSpecific + 567
16 GraphicsServices 0x00007fff2b874d85 GSEventRunModal + 139
17 UIKitCore 0x00007fff246c14df -[UIApplication _run] + 912
18 UIKitCore 0x00007fff246c639c UIApplicationMain + 101
19 MyorgDevelopmentEG2 0x0000000108f175de main + 190
20 libdyld.dylib 0x00007fff2025abbd start + 1
21 ??? 0x0000000000000005 0x0 + 5
)
<_ASdisplayLayer: 0x600007996580> => Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
假设
这让我认为有一些孤立的子进程或守护进程仍然占有资源。或者没有正确清理工件。
我尝试过的东西
xcrun simctl shutdown all
xcrun simctl erase all
xcrun simctl delete all
osascript -e 'tell application "iOS Simulator" to quit'
osascript -e 'tell application "Simulator" to quit'
- 在构建之间终止 CoreSimulator 相关服务(无帮助)
launchctl remove com.apple.CoreSimulator.CoreSimulatorService || true
launchctl remove com.apple.CoreSimulator.SimLaunchHost-x86 || true
launchctl remove com.apple.CoreSimulator.SimulatorTrampoline || true
launchctl remove com.apple.CoreSimulator.SimLaunchHost-arm64 || true
- 在构建之间删除模拟器设备、日志和缓存。这实际上有点帮助。片状最初消失了,但在几次构建后很快又回来了,而机器仍在构建之间执行清洁......不确定为什么。
echo "Delete all simulator devices,cache,and temp"
rm -rf ~/Library/Developer/CoreSimulator
echo "Delete Simulator Logs"
rm -rf ~/Library/Logs/CoreSimulator
- 在 EarlGrey 上增加 Animation Timeout,Interaction Timeout
- Blacklist the URL 请求挂起。这确实通过忽略症状而不修复实际问题来提高测试套件的稳定性,所以我希望我可以避免这种情况。
还有其他人遇到同样的问题吗?感谢有人可以对此有所了解或提供尝试建议。
注意:我们使用 Bluepill 和 EarlGrey2 在 buildkite CI 管道上运行 UI 测试(iOS 14.5、Xcode 12.5)。
解决方法
你有没有试过跑步:
sudo systemsetup -setcomputersleep never