Canvas Accessibility Testing & Evaluation (CATE)

Accessibility is most often an ongoing goal rather than an unambiguously quantifiable state. Answering the question “is this product accessible?” is frequently affected by factors within a vendor’s control. However, it may also be dependent on users’ choices of assistive technologies, browsers, and operating systems, as well as less tangible factors such as user sophistication and assumptions.
One universal tendency is that accessibility tends to grow best in environments that share knowledge and promote cooperation among vendors and users. Towards this end, several members of the Access Technology Higher Education Network (ATHEN) recently conducted an objective accessibility testing & evaluation of Canvas from November 2015 to January 2016. Following a similar process to the LMS Accessibility Evaluation projects, the ATHEN group published the results at the CSUN 2016 conference.
We have presented a high-level overview of the results of our new evaluation and provided recommendations for design, interaction, and coding practices whenever possible. Implementation of these suggestions should improve the overall usability/accessibility of Canvas Instructure for all users including those with disabilities.
Instructure, the developer of Canvas, has consistently demonstrated a commitment towards delivering an accessible product, and we look forward to continued collaboration for mutual benefit.
To learn more about the results, please visit and study the following documents:

1. Canvas Accessibility Testing & Evaluation CSUN 2016 (PowerPoint file)
2. Canvas Accessibility Testing & Evaluation Report (full report HTML file)
3. Canvas Accessibility Testing & Evaluation Results (Excel file)

To utilize the full feature of interlinking between different sheets in this document, you should open the file in Microsoft Excel program and not in the browser.

A Comparison of Learning Management System Accessibility

Learning management systems have become the primary delivery platform in most higher education systems for course-related activities such as lecture presentations, readings and assignments, discussions, and quizzes. Until a few years ago, access for learners and instructors with disabilities was either poorly supported or not considered at all in many popular tools. Due to lack of, or limited, accessibility in learning management systems, students were not able to fully or independently participate in key course activities.

Thanks to the hard work of various LMS accessibility working groups and their vendor and open-source developer partners, many LMS vendors have begun to understand the need for universal usability of their tools. LMS vendors have begun providing accessibility features that allow users with disabilities access, but we are still far from achieving full accessibility of learning management systems.

Although significant progress has been made, intuitive and effective utilization of LMS by users with disabilities is still a future goal. We believe much more needs to be done. For instance, discussion boards need to be made more functionally accessible, built-in authoring tools need to be enhanced so that accessible content is created by default, the authoring process is more intuitive, and the authoring tools themselves become more accessible. On the whole, the accessibility of learning management systems is improving but they still pose significant challenges for users with disabilities.

We have tested and evaluated four major LMS namely Blackboard, Desired2Learn, Moodel, and SAKAI
for accessibility/usability in 2010 and 2012 and published the results in CSUN and Midwest Educause conferences respectively.

We have been observing that LMS vendors have continued to strengthen the accessibility and usability of their systems, while also working to incorporate some of the features suggested in our evaluation. In light of this recent interest and activity, we have revised and enhanced our evaluation criteria based on common features, streamlined our evaluation process, and rescored our LMS systems.

We have presented a high-level overview of the results of our new evaluation, along with side-by-side comparisons that support our recommendations for improving learning management systems for users with disabilities at CSUN 2013.

To learn more about the result, please visit and study the following documents:

j
1. A Comparison of Learning Management System Accessibility: Blackboard, Desire2Learn, Moodle, SAKAI (CSUN 2013 PowerPoint file)
2. A Comparison of Learning Management System Accessibility (full report HTML file)
3. Comparison data (full testing and evaluation results Excel file)Comparison data (full testing and evaluation results Excel file)
   To utilize the full feature of interlinking between different sheets in this document, you might want to open the file in Microsoft Excel program and not in the browser.

A Model for Accessible LMS Discussion Boards

We have been testing and evaluating major learning management systems for several years, and while we continue to document significant improvement in many LMS components, we have noted little or no noticeable progress in the accessibility of the discussion board/forum, a tool which remains one of the most complicated and difficult for learners with disabilities to use effectively. In response, we decided to create a rough prototype of a more accessible discussion forum, one that draws on our experience and knowledge as accessibility and instructional technology/design specialists, and uses our collective understanding of how various LMS have approached this component.

Our discussion board model is an outgrowth of our long-range comparative accessibility testing and ongoing evaluation of four major learning management systems—Blackboard, Desire2Learn, Moodle, and SAKAI. Our suggested design attempts to provide solutions for many of the perennial, cumbersome accessibility/usability issues that affect LMS discussion forums, such as organizing and grouping forums, topics, threads, and messages so that all users, including assistive technology users, can easily identify a desired object/location, navigate to it, and interact effectively with it. Here are some of the key questions we considered:

• What are the best ways to order lists of discussion topics?
• What level of detail should be evident when the user first encounters a discussion board?
• How should greater detail and specificity be progressively revealed as the user drills down to deeper threads?
• What is an optimal amount of information for any given message and how does that message appear in the context of other messages so that the content and place within the overall discussion board or thread are clearly evident?
• What HTML structures and what scripted interactivity provide a strong user experience without getting in the way of that experience?

We are not promising a drop-in solution, and we cannot claim to provide a perfect implementation. Instead, our presentation first examines how typical LMS discussion boards tend to work from an accessibility perspective. From that vantage point, we then offer design patterns and a rough prototype implementation for others to consider as a more accessible potential alternative. Our hope is that LMS vendors will borrow some of our ideas to improve their own systems’ discussion boards/forums.

Acknowledgements

I would like to thank and acknowledge the hard work of all my colleagues who helped me with this project; without their help I wouldn't have been able to design and deliver this new model for discussion board. An especial thank to Michael McKelvey who worked on the code until the last minutes of our presentation at CSUN Conference 2013.

Contributors:

• Hadi Rangin, IT Accessibility & Collaboration coordinator, University of Illinois
• Michael McKelvey, Coordinator of Engagement Technology & New Media, University of Illinois
• Marc Thompson, Instructional Designer, University of Illinois
• Dan Hahn, ELearning Professional, University of Illinois
• Ken Petri, Program Director, The Ohio State University

Highlights of the Suggested Model

Note: This design is optimized for Firefox and Chrome. We have not worked on cross-browser compatibility yet.
Use NVDA with Firefox to test for screen reader accessibility; do not use Jaws.

1. Full ARIA landmarks to provide semantic structure
   • Breadcrumbs [navigation landmark]
   • Forum Options [navigation landmark]
   • Search [search landmark]
   • Forum [navigation landmark]
   • Message [region landmark]
2. ARIA menu for intuitive keyboard navigation in the menu
   • Display, Filter, and Manage menu
   • Enter or space to open the menu
   • Up and down arrow to scroll through the menu
   • Right arrow to open a sub-menu and left arrow to close a sub-menu
   • Enter to select an item in the menu
   • Escape to cancel interaction with the menu
   • Tab key to move from one top menu to another top menu
3. Select box for selecting the desired forum
   • Up and Down arrow to scroll through the menu
   • Enter to select the desired Forum
   • Selecting a forum updates the Thread treeview
   • NOTE: select list keydown & change events must be modified so simply scrolling through menu does not update thread treeview.
4. ARIA tree view to browse and select desired thread
   • Up and Down arrow to go through the threads/posts
   • Right arrow to open a thread
   • Left arrow to close a thread
   • Enter selects the desired thread/message and moves the focus to the beginning of the thread conversation shown in the message landmark
5. HTML headings to structure thread depth
   • Heading levels 2-6 are used structure threads/messages
   • Each post has an ARIA action menu for applicable functions.
   • A Message Display Mode select box to filter threads/posts shown.
6. Keyboard shortcuts
   • Shortcut key 1 to move to the Forum select box
   • Shortcut key 2 to move the focus to the threads treeview
   • F6 key to switch focus between landmark regions (not implemented)

• A Model for Accessible LMS Discussion Boards (CSUN 2013 PowerPoint file)
• See our suggested Discussion Board model in action.

About United Arab Emirates

About the United Arab Emirates

By: Niki Rangin

Date: December 2012

Last Christmas break I had a great holiday. My dad and I went to the United Arab Emirates (UAE) for vacation. It was my first trip there and we visited my dad’s friends and family. On this trip I learned a lot about the UAE.

The United Arab Emirates is a small country located at south side of the Persian Gulf. UAE is a rich country with interesting places. The Population is about four million. One million people are native Emirati and three million are guest workers coming mainly from Philippine, Bangladesh, India, Lebanon, Egypt, and Jordan. The Capital of UAE is Abu Dhabi and their official language is Arabic. Other languages spoken in UAE are Persian, Hindi, Urdu, and Punjabi.

There are forty million palm trees in the UAE so that means each person in the UAE has about forty trees. The highways are well-lighted and you can see rows of date trees on each side of the roads and dividing islands.

The weather in UAE is very mild in winter but very hot and humid in summer.

The world’s tallest building in the world is in Dubai. It is called Burj Khalifa (Khalifa Tower). It is 828 meters or 2,717 feet. The UAE paid 1.5 billion dollars to build and it was opened in 2010. 

The UAE has lots of earth oil well. Oil was discovered in the UAE in 1950. People of the UAE sold the oil to other countries. With the money they got from the oil, they build very interesting places like the Arab tower, Palm Island, Burj Khalifa and Atlantis which is a hotel and has an aquarium with about three million gallons of water with over 65000 sea creatures including sharks, eels, seahorses, and piranhas, . Atlantis is on Palm Island. Palm Island is an island that is man made and it is shaped like a palm tree. 

All of these interesting places make visitors come to the UAE and they spend their money on air travel, hotels, and food. About nine million tourists come to visit the UAE every year.

Every Emirati receives free land and the government helps them to build their houses with low-interest loan. The education is totally free from Kindergarten up to higher education, they don’t pay for electricity or water, and the health care is totally free for them.

How to talk and discuss about accessibility with vendors?

Introduction

Frequently sales people and representatives of companies we purchase products from are not familiar with accessibility of their products. When you ask them about accessibility of their product, they usually refer you to official company’s accessibility statements which are usually a series of statements emphasizing their strong commitment to accessibility and referencing Federal government 508 Standards or W3C Content Accessibility guidelines and usually without any indication if they meet those guidelines or not.

Some companies hire accessibility consulting companies to review their product usually at the end of the development phase where the product has already been released or is about to be released. Testing for accessibility at this stage is usually too late to address potential accessibility issues. The result of accessibility testing of these companies are not usually published or discussed with customers, however vendors talk proudly about it on their accessibility page of their websites to emphasize their commitment to accessibility.

On the other hand, university administrators and those involved in purchasing a product have also very limited knowledge and understanding about accessibility to discuss accessibility of the product they want to buy with vendors. So none of the parties involved in purchasing a product is familiar with accessibility or have no organized means to verify accessibility claims. This is one of the reasons why accessibility is frequently neglected or jeopardized in such negotiations and deal-makings.

This brief handout should help you to give you some ideas how to talk with a vendor about accessibility and ask them the right questions while you are educating them about accessibility.

It is very difficult to come up with a recipe that would work for all products; however, the following accessibility related questions/concerns should be applicable to most products. In some instances you might want to have very specific accessibility questions about behavior of the product in question in certain situation or for certain disability group.

In the following sections we introduce and discuss some major accessibility factors and then provide a few questions that you might want to consider discussing with your vendors.

Accessible Design

The first and most important thing that we need to know is that the best time to address accessibility is when a product is being designed. Addressing accessibility issues after a product has been released is usually very limited, incomplete, wishy-washy, too expensive, and sometimes impossible.

Considering the accessibility at the design stage ensures that the product is designed with accessibility from ground up, the right set of technologies and techniques are used during the implementation, and the accessibility features are tested in-house before the product enters the market.

Many companies/developers see accessibility as limiting factors in their creativity and functionality of their applications. In most cases, with a little creativity and innovation, an application can be designed and made accessible if they consider accessibility at the design stage and choose the right technologies and techniques. If you are not familiar with accessibility, you don’t want to get involved in technical discussion with vendors; instead bring your campus IT and Accessibility experts together and possibly partner with other institutions to build a group and discuss the issues and potential solutions as a group.

Questions

• When do you think about accessibility of your product? At the design stage, during the implementation process or later on after it has been developed?



• Do you have accessibility experts in your design/development team or you hire outside accessibility consultant? What accessibility consulting company do you work with?

Accessibility Testing

The fact that some vendors work with accessibility consulting companies is good as long as they work together during the design and implementation processes and accessibility feedback can be addressed while the product is being developed. An accessibility evaluation just for the purpose of publicity at the end of development phase has no value.

The best way to test a product for accessibility is if it is tested by real users with disabilities. A mixed group of users with disabilities can provide extremely valuable feedback to product development team. This will give the development team an opportunity to see the real users interacting with their product first hand, see the shortcomings, and how users can benefit from accessibility features.

Questions

• How do you test for accessibility? Do you have accessibility testers in-house, for example, at the quality assurance department, or you hire outside accessibility consultant? If latter one, what consulting company?


• Have you used real end-users with disabilities to test your product? How many and from what disability groups?


• When do you perform accessibility testing? During the development phase or before the product is about to be released?


• What is the accessibility criteria/matrix for testing your product? Can we have a copy of the document?


• Can you share the last accessibility report on your product with us?


• And remember to ask the sales representative to detach his/her the computer mouse and show the product in action just with the keyboard during the product presentation.

Accessibility Education

Sometimes talking with a vendor could give you the impression as if they don’t care about accessibility. This might be true but it is very uncommon. Most people including company’s representatives, sales people, and even CEOs would care for accessibility once they learn about it the accessibility of their products.

There are many people who have never dealt with accessibility issues in their lives and understanding accessibility issues might be too abstract to them. It is very important to be patient and educate them in a nice and friendly way. There is a wrong perception among some developers and product managers as if accessibility is too much work, expensive, and limits the functionality of their products. We need to educate vendors about the benefits of universal design and how easy it is to achieve it if we think about it at the right time. And remember accessibility is free if we consider it at the design phase.

Questions

• Where do your designers/developers get accessibility training?


• Which accessibility related conferences do you go?


• Have you ever had a disabled employee or an outside disabled user showing how your product can be used with assistive software/hardware?

Accessibility Documentations

Some products indeed support selected accessibility features that can be beneficial to many users with disabilities. Without any proper documentation on the supported features, users might not be able to discover them easily and/or utilize them properly.

A good accessibility documentation where all the accessibility features are listed and explained can help users to learn more about the product and how to effectively use it. Such documentations should be publically available and easy to access and clearly they should be fully accessible.

Questions

• Do you have accessibility documentation about your product? Are the supported accessibility features in your product listed and explained?


• Are they publically accessible? Can I have the URL to the accessibility documentations? What format are they in?

Accessibility Support

Another problem that users with disabilities have been facing is that they can’t get any accessibility support for a particular product from the campus or from the vendor. Accessibility related issues often don’t get recorded or don’t reach the right people because there is no well-defined accessibility support by campuses or vendors and as the result those users are left frequently abandoned with their issues.

Training and educating the support staff on accessibility and how to use the product with the keyboard would enable the support staff to help the disabled users with the accessibility issue. It is not expected that support staff know all about accessibility but they should have at least some familiarity with accessibility and use their product without mouse. Support should have also a mechanism to record and direct accessibility issues and feedback. This is the only way that the accessibility of a product can be improved.

Questions

• Does your support staff get training on accessibility? Is there any accessibility expert in-house whom other support staff can contact for more advanced issues?


• Is there any mechanism that accessibility feedback and problems can be reported to the company? Can accessibility feedback/problem page be easily found at your website? How these feedback/problems are directed to the right people?


• Do you provide accessibility training to the Campus IT staff to handle accessibility issues locally?