SwitchAccess 兼容的虚拟视图节点的深层树状结构

问题描述

我正在尝试在 Android 中创建一个可访问的自定义 View。 View 包含一个虚拟节点的树层次结构，在我的情况下可以是多级深度。 树不一定是二元的：每一层的每个节点都可以有用户想要的任意数量的子节点。 具体来说，我试图使其与 TalkBack 的 SwitchAccess 辅助功能服务兼容。 最终目标是使其兼容所有 TalkBack 的无障碍服务，但至少必须与 SwitchAccess 兼容。 在这篇文章中，我正在尝试单开关访问。 树的结构事先未知，其扫描顺序也未知。它们都是在运行时定义的。 我正在使用 ExploreByTouchHelper 类，它是 AccessibilityDelegate 的便利包装器。

以下类似于事件序列和虚拟树的图像：

一个示例扫描序列由节点内的数字表示。 从用户的角度来看，他们将叶子视为某种随机颜色的矩形。 这只是自定义 View 的内容。 描绘的扫描顺序如下：

1. 首先，用户看到所有的节点都被一起扫描（有根被扫描）。他们按下开关并选择根，将扫描过程移到树的第二级。
2. 然后扫描第一个内部节点，用户忽略它，所以扫描移动到第二个根的子节点，用户点击开关，将扫描过程移动到树的第三级，即点击节点的子节点。
3. 依此类推，直到用户点击/选择叶节点（在名为 7. Click! 的步骤），在这种情况下会发生自定义操作。

命名法：

1. 什么是 SwitchAccess？这是一项服务，允许用户通过将单个（或几个）硬件开关连接到他们的手机来键入键/字母。 你可以把它想象成一个残障人士的物理键盘，它有一个（或几个）开关，而不是每个字母一个开关。 用户在一个称为扫描的过程的帮助下输入一个字母，其中字母表中的每个字母（或任何类型的键）都被一个一个扫描，当用户点击他们的单个开关然后输入相应的字母。 这就像点击一个开关，点击频率被这项服务转换成字母。
2. 什么是 TalkBack？一组可供 Android 用户使用的辅助功能服务（包括 SwitchAccess）。 这是一个普通的 Android 应用程序（Android Accessibility Suite），如果您的系统中尚未包含该应用程序，则可以从 here 下载。

我尝试过（但失败了）：

1. 试验每个 AccessibilityNodeInfo 的焦点状态、选中状态和无障碍焦点状态。
2. 每次点击只报告我感兴趣的子树部分。这意味着每次点击都会报告不同的树（通过 getVisibleVirtualViews）。
3. 受 AccessibilityNodeInfo.CollectionInfo 的实现启发，对 AccessibilityNodeInfo.CollectionItemInfo 和 GridView 进行实验。
4. 仅在用户点击叶子时发送 AccessibilityEvents（而不是内部节点），以便让扫描在点击内部节点时继续。
5. 仅报告叶组而不是内部节点。
6. 在每次用户点击时重新安装/更改整个 AccessibilityNodeProvider 和/或 AccessibilityDelegate。

按照我迄今为止的最大努力（这是我上面的一些努力的组合），我们也可以就此进行讨论：

import android.content.Context;
import android.graphics.Canvas;
import android.graphics.Color;
import android.graphics.Paint;
import android.graphics.Point;
import android.graphics.Rect;
import android.graphics.RectF;
import android.os.Bundle;
import android.view.KeyEvent;
import android.view.MotionEvent;
import android.view.View;
import android.view.ViewGroup;
import android.view.accessibility.AccessibilityEvent;

import androidx.annotation.NonNull;
import androidx.annotation.Nullable;
import androidx.appcompat.app.AppCompatActivity;
import androidx.core.view.ViewCompat;
import androidx.core.view.accessibility.AccessibilityNodeInfoCompat;
import androidx.customview.widget.ExploreByTouchHelper;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Locale;
import java.util.Random;

public class MainActivity extends AppCompatActivity {

    public static int randomColor(final int alpha,final Random rand) {
        final byte[] components = new byte[3];
        rand.nextBytes(components);
        return Color.argb(alpha,components[0] & 0xFF,components[1] & 0xFF,components[2] & 0xFF);
    }

    public static class Node {
        public Node parent = null;
        public final ArrayList<Node> children = new ArrayList<>();
        public final Point index = new Point(),//Location of this Node in its parent. Root Node does not use this.
                            size = new Point(); //Number of children this Node has per dimension (x == columns,y == rows).
        public String text = null; //The text associated with each Node.
        public int id = -1,//It is initialized at Tree construction.
                color = 0; //It is only used for leaves,but for simplicity included in every Node.

        /**
         * Used as a default way to create the children of this Node.
         * @param title Some value used to construct the text of each children.
         * @param sizeX Number of columns each children will be initialized to have.
         * @param sizeY Number of rows each children will be initialized to have.
         * @param rand Used for producing each child's color.
         */
        public void addChildren(final String title,final int sizeX,final int sizeY,final Random rand) {
            for (int row = 0; row < size.y; ++row) {
                for (int col = 0; col < size.x; ++col) {
                    final Node child = new Node();
                    child.parent = this;
                    children.add(child);
                    child.index.set(col,row);
                    child.size.set(sizeX,sizeY);
                    child.text = String.format(Locale.ENGLISH /*Just use ENGLISH for only the demonstration purposes.*/,"%s|%s:%d,%d",text,title,row,col);
                    child.color = randomColor(255,rand);
                }
            }
        }

        /** @param bounds Serves as input (initialized with the root Node's bounds) and as output (giving the bounds relative to root for the calling Node). */
        public void updateBounds(final RectF bounds) {
            if (parent != null) {
                parent.updateBounds(bounds);
                //Adjust parent bounds to locate the current node:
                final float cellWidth = bounds.width() / parent.size.x,cellHeight = bounds.height() / parent.size.y;
                bounds.left += (cellWidth * index.x);
                bounds.top += (cellHeight * index.y);
                bounds.right -= (cellWidth * (parent.size.x - index.x - 1));
                bounds.bottom -= (cellHeight * (parent.size.y - index.y - 1));
            }
        }
    }

    /**
     * Gets a subtree (starting from the given Node) of nodes into the given lists.
     * @param node the root of the subtree we are interested in.
     * @param allNodes all nodes of the subtree will go in here.
     * @param leavesOnly only the leaves of the subtree will go in here.
     */
    public static void getNodes(final Node node,final ArrayList<Node> allNodes,final ArrayList<Node> leavesOnly) {
        allNodes.add(node);
        if (node.children.isEmpty())
            leavesOnly.add(node);
        else
            for (final Node child: node.children)
                getNodes(child,allNodes,leavesOnly);
    }

    /** Sacrificing memory for speed: this is essentially a huge cache. */
    public static class Tree {
        public final Node root;
        public final List<Node> nodes,//All nodes of the tree.
                                leaves; //Only leaves of the tree (which will exist in both 'nodes' property and in 'leaves' property).

        public Tree(final Node root) {
            this.root = root;
            final ArrayList<Node> nodesList = new ArrayList<>();
            final ArrayList<Node> leavesList = new ArrayList<>();
            getNodes(root,nodesList,leavesList);
            nodes = Collections.unmodifiableList(nodesList);
            leaves = Collections.unmodifiableList(leavesList);
            final int sz = nodes.size();
            for (int i = 0; i < sz; ++i)
                nodes.get(i).id = i; //As you can see the id corresponds exactly to the index of the Node in the list (so as to have easier+faster retrieval of Node by its id).
        }
    }

    /** @return a Tree for testing. */
    public static Tree buildTestTree() {
        final Random rand = new Random();
        final Node root = new Node();
        root.size.set(2,1); //2 columns,1 row.
        root.text = "Root";
        root.addChildren("Inner",2,1,rand); //2 columns,1 row.
        for (final Node rootChild: root.children) {
            rootChild.addChildren("Inner",1 row.
            for (final Node rootInnerChild: rootChild.children)
                rootInnerChild.addChildren("Leaf",rand); //1 column,1 row. Basically a leaf.
        }
        return new Tree(root);
    }

    /** @return a value conforming to the measureSpec,while being as close as possible to the preferredSizeInPixels. */
    public static int getViewSize(final int preferredSizeInPixels,final int measureSpec) {
        int result = preferredSizeInPixels;
        final int specMode = View.MeasureSpec.getMode(measureSpec);
        final int specsize = View.MeasureSpec.getSize(measureSpec);
        switch (specMode) {
            case View.MeasureSpec.UNSPECIFIED: result = preferredSizeInPixels; break;
            case View.MeasureSpec.AT_MOST: result = Math.min(preferredSizeInPixels,specsize); break;
            case View.MeasureSpec.EXACTLY: result = specsize; break;
        }
        return result;
    }

    /** The custom View which maintains the tree hierarchy of virtual views. */
    public static class HierarchyView extends View {
        private final MyAccessibilityDelegate delegate; //The ExploreByTouchHelper implementation.
        public final Tree tree; //The tree of virtual views.
        public Node selected; //The last 'clicked' node from all the nodes in the tree.
        private final int preferredWidth,preferredHeight; //The preferred size of this View.
        private final Paint tmpPaint; //Used for drawing.
        private final RectF tmpBounds; //Used for drawing.

        public HierarchyView(final Context context) {
            super(context);
            tmpPaint = new Paint();
            tmpBounds = new RectF();
            selected = null;

            //Hardcoded magic numbers for the dimensions of this View,only in order to keep things simple in this demonstration:
            preferredWidth = 600;
            preferredHeight = 300;

            tree = buildTestTree();
            super.setContentDescription("Hierarchy");
            ViewCompat.setImportantForAccessibility(this,ViewCompat.IMPORTANT_FOR_ACCESSIBILITY_YES);
            ViewCompat.setAccessibilityDelegate(this,delegate = new MyAccessibilityDelegate(this));
        }

        @Override
        protected void onMeasure(final int widthMeasureSpec,final int heightMeasureSpec) {
            setMeasuredDimension(getViewSize(preferredWidth,widthMeasureSpec),getViewSize(preferredHeight,heightMeasureSpec));
        }

        /**
         * Use this method instead of {@link Node#updateBounds(RectF)},which (this method) will properly initialize the root Node's bounds.
         * @param bounds The output bounds for the given Node.
         * @param node The input Node to get the bounds for.
         */
        public void updateBounds(final RectF bounds,final Node node) {
            bounds.left = bounds.top = 0;
            bounds.right = getWidth();
            bounds.bottom = getHeight();
            node.updateBounds(bounds);
        }

        @Override
        protected void onDraw(final Canvas canvas) {
            for (final Node leaf: tree.leaves) {
                tmpPaint.setColor(leaf.color);
                tmpPaint.setAlpha(selected == leaf? 255: 64);
                updateBounds(tmpBounds,leaf); //Not the most efficient (needs logN),but remember this is just a demo.
                canvas.drawRect(tmpBounds,tmpPaint);
            }
        }

        @Override
        public boolean dispatchHoverEvent(final MotionEvent event) {
            //This is required by ExploreByTouchHelper's docs:
            return delegate.dispatchHoverEvent(event) || super.dispatchHoverEvent(event);
        }

        @Override
        public boolean dispatchKeyEvent(final KeyEvent event) {
            //This is required by ExploreByTouchHelper's docs:
            return delegate.dispatchKeyEvent(event) || super.dispatchKeyEvent(event);
        }

        @Override
        protected void onFocusChanged(final boolean gainFocus,final int direction,final @Nullable Rect prevIoUslyFocusedRect) {
            super.onFocusChanged(gainFocus,direction,prevIoUslyFocusedRect);
            //This is required by ExploreByTouchHelper's docs:
            delegate.onFocusChanged(gainFocus,prevIoUslyFocusedRect);
        }

        /*
        @Override
        public boolean onTouchEvent(final MotionEvent event) {
            final int virtualViewId = delegate.getVirtualViewAt(event.getX(),event.getY());
            if (virtualViewId != ExploreByTouchHelper.INVALID_ID)
                selected = tree.nodes.get(virtualViewId);
            invalidate();
            return super.onTouchEvent(event);
        }
        */
    }

    public static class MyAccessibilityDelegate extends ExploreByTouchHelper {
        private final HierarchyView host;

        /**
         * This is used as the <b>parent</b> of each node that should be interactive. If null,then the
         * root should be interactive,otherwise if not null,then the its children should be interactive.
         */
        private Node last;

        public MyAccessibilityDelegate(final @NonNull HierarchyView host) {
            super(host);
            this.host = host;
            last = null; //Start with root.
        }

        /** Helper method to retrieve a read-only Iterable of the nodes that should be interactive. */
        private Iterable<Node> readVisibleNodes() {
            return last == null? Collections.singletonList(host.tree.root) : Collections.unmodifiableList(last.children);
        }

        @Override
        protected int getVirtualViewAt(final float x,final float y) {
            final RectF bounds = new RectF();
            for (final Node node: readVisibleNodes()) {
                host.updateBounds(bounds,node);
                if (bounds.contains(x,y))
                    return node.id;
            }
            return INVALID_ID;
        }

        @Override
        protected void getVisibleVirtualViews(final List<Integer> virtualViewIds) {
            for (final Node node: readVisibleNodes())
                virtualViewIds.add(node.id);
        }

        @Override
        protected void onPopulateNodeForVirtualView(final int virtualViewId,final @NonNull AccessibilityNodeInfoCompat info) {
            final Node node = host.tree.nodes.get(virtualViewId);

            //Just set all text to node#text for simplicity:
            info.setText(node.text);
            info.setHintText(node.text);
            info.setContentDescription(node.text);

            //Get the node's bounds:
            final RectF bounds = new RectF();
            host.updateBounds(bounds,node);

            /*Although deprecated,setBoundsInParent is actually what ExploreByTouchHelper requires,and itself
            then computes the bounds in screen. So lets just setBoundsInParent,instead of setBoundsInScreen...*/
            if (node.parent == null) { //If node is the root:
                info.setParent(host); //The View itself is the parent of it (or maybe not,I am not sure).
                info.setBoundsInParent(new Rect(Math.round(bounds.left),Math.round(bounds.top),Math.round(bounds.right),Math.round(bounds.bottom)));
            }
            else {
                /*To get the bounds of any node which is not the root,I simply subtract the parent's bounds
                with the current node's bounds. I kNow... not the most efficient,but it's just a demo Now.*/
                info.setParent(host,node.parent.id);
                final RectF parentBounds = new RectF();
                host.updateBounds(parentBounds,node.parent);
                info.setBoundsInParent(new Rect(Math.round(bounds.left - parentBounds.left),Math.round(bounds.top - parentBounds.top),Math.round(bounds.right - parentBounds.left),Math.round(bounds.bottom - parentBounds.top)));
            }

            //As I have found out,those calls are absolutely necessary for the virtual views:
            info.setEnabled(true);
            info.setFocusable(true);

            //These calls seem to not be absolutely necessary,but I am not sure:
            info.setVisibletoUser(true);
            info.setImportantForAccessibility(true);

//            info.setContentInvalid(false);
//            info.setAccessibilityFocused(last == node);
//            info.setFocused(last == node);
//            info.setChecked(last == node);
//            info.setSelected(last == node);

            if (node.parent == last) { //This is the way I am testing if the current node should be interactive.
                info.setClickable(true);
                info.setCheckable(true);
                //info.setCanopenPopup(true);
                //info.setContextClickable(true);
                //info.addAction(AccessibilityNodeInfoCompat.ACTION_SELECT);
                info.addAction(AccessibilityNodeInfoCompat.ACTION_CLICK);
            }
            if (!node.children.isEmpty()) {
                info.setCollectionInfo(AccessibilityNodeInfoCompat.CollectionInfoCompat.obtain(node.size.y,node.size.x,true,AccessibilityNodeInfoCompat.CollectionInfoCompat.SELECTION_MODE_SINGLE));
                for (final Node child: node.children)
                    info.addChild(host,child.id);
            }
            if (node.parent != null)
                info.setCollectionItemInfo(AccessibilityNodeInfoCompat.CollectionItemInfoCompat.obtain(node.index.y,node.index.x,false,false));
        }

        @Override
        protected boolean onPerformActionForVirtualView(final int virtualViewId,final int action,final @Nullable Bundle arguments) {
            if (action == AccessibilityNodeInfoCompat.ACTION_CLICK) {
                host.selected = host.tree.nodes.get(virtualViewId);
                last = host.selected.children.isEmpty()? null: host.selected;
//                if (host.selected.children.isEmpty()) {
                invalidateVirtualView(virtualViewId); //,AccessibilityEventCompat.CONTENT_CHANGE_TYPE_SUBTREE);
                sendEventForVirtualView(virtualViewId,AccessibilityEvent.TYPE_VIEW_CLICKED);
                host.invalidate(); //To redraw the UI.
//                }
//                else
//                    invalidateRoot();
//                    invalidateVirtualView(virtualViewId);
//                    host.sendAccessibilityEvent(AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED);
//                    host.sendAccessibilityEvent(AccessibilityEvent.TYPE_VIEW_CLICKED);
//                    host.sendAccessibilityEvent(AccessibilityEventCompat.TYPE_VIEW_CONTEXT_CLICKED);
//                    sendEventForVirtualView(virtualViewId,AccessibilityEvent.TYPE_WINDOW_STATE_CHANGED);
//                    invalidateVirtualView(virtualViewId,AccessibilityEventCompat.CONTENT_CHANGE_TYPE_SUBTREE);
                return true;
            }
            return false;
        }
    }

    @Override
    protected void onCreate(final Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(new HierarchyView(this),new ViewGroup.LayoutParams(ViewGroup.LayoutParams.WRAP_CONTENT,ViewGroup.LayoutParams.WRAP_CONTENT));
    }
}

这会导致扫描在点击时冻结（但至少它在树中更深一层），而实际需要的行为是让扫描继续进行，直到点击一个叶子，扫描应该结束。 几乎就像扫描软件键盘一样（即首先是一组键，然后点击，扫描每个键，依此类推）。

请注意，由给定代码构建的树与图像的树并不完全相同。 图片作为视觉示例，代码作为讨论的基础。

我基本上是想找出 SwitchAccess 中的哪些规则会使我的应用程序与之兼容。 理想情况下，我希望用户看到正在扫描的叶子组，而不是像提供的代码那样每次都看到单个节点，但我想这是一个不同的故事。

我认为我要问的是可能的，因为否则 setParent 之类的方法不会包含在 AccessibilityNodeInfo 类中。

我也在考虑用 drawing order 进行实验，但我根本不知道它是否相关。

我正在使用最低 SDK 版本 14（如果有的话）。

互联网上有一些关于单级深度虚拟树的例子，但我就是不知道如何使它们成为多级深度。

请注意，我们采取了几个步骤来缩短代码，因此它不会遵循有关面向对象编程的最佳实践、时间复杂度、内存使用等，因为它只是作为演示。

一些资源：

1. The corresponding Google I/O 2013 video（从 ExploreByTouchHelper 用例的介绍开始）。
2. TalkBack's source code。我看了很多遍，但我仍然不知道如何解决我的问题。
3. How to use SwitchAccess on your phone（用户视角）。

解决方法

我不相信你想要的是可能的。或者至少，我不相信您想要的具有支持它的“显式 API”。让我们来谈谈 Switch Access 将关注什么。基本上，它会关注任何

• 未明确标记为“无障碍不重要”
• 有某种类型的动作与之相关
  • 点按
  • 点击并按住
  • 自定义操作
  • 等

Switch Access 对事物进行分组的方式不会响应任何特定的 API，而是根据与标准用户体验相关的现有信息并基于 Switch Access 配置计算得出。您可能会根据行列扫描与组选择的不同thigns 来确定这一点。一般来说，重要的事情是：

• 在视图层次结构中排序
• 在屏幕上的位置

为“精确分组”操作这些将非常困难。每个版本的 Switch Access 都可以为所欲为。没有记录的 API 说“这是分组”。 Switch Access 只是尽最大努力理解标准的 Android API。

从用户的角度准确表达您所拥有的信息。

• 将事物组放入虚拟布局中。
• 确保可以与之交互的所有内容都标记为此类。
• 在您的虚拟层次结构中以合理的顺序排列您的视图

这就是你真正能做的。事实上，试图对其进行更多的操作可能会让用户感到困惑。