Comparison Between ASR and RCA
1. When and where will I be completing the RCA?
When: 1,366 registrants wrote or attempted to write the RCA in 2015 and received a results report. If a registrant did not receive a results report, they will be required to complete the RCA in 2018. See FAQ #5 for RCA exemptions.
Where: The exam is taken online at a location of the registrant’s choosing (a quiet location with stable Internet access is strongly recommended).
2. What is the cost to registrants for taking the RCA?
The cost for the Quality Assurance Program is included in the annual registration fees. The annual fees were increased several years ago in part to finance the development of the RCA. Containing the cost, while developing and implementing the Quality Assurance Program, is a high priority for the College, and was one of the criteria used in choosing the format for the RCA.
There can be penalty fees if registrants miss deadlines (e.g. requesting accommodations after the deadline).
3. If I don’t meet the standard set by the RCA, will I have to stop practicing?
No. The RCA is intended to assess if you are meeting a minimal standard, and if not, to help support and direct how you can meet that standard. This approach means that if you are unsuccessful you will not lose your registration. The College will work with you to set up an individualized plan to assist you to fill the gap(s) identified through the RCA and improve your practice.
• If the registrant does not meet standard on the RCA, they will take the RCA a second time, approximately 6 months after the first attempt.
• If the registrant does not meet standard on their second attempt at the RCA, the College will help them to set up a practice support plan that will enable them to demonstrate competence.
• All full and limited registrants must complete the RCA once every six years. Full and limited registrants must complete the RCA during the year that they were selected to complete the RCA in order to renew their registration for the following year.
4. Is there an ‘expected’ pass or fail rate for the RCA?
The RCA is a ‘criterion referenced’ exam where all those who meet the standard pass. Information on how the standard is set is contained in an article in the Fall 2014 newsletter.
Performance assessment literature and the experiences recorded from continuing competence programs over the past 20 years – regardless of whether a written test, portfolio or onsite assessment is used – has revealed that somewhere between 90 – 95% of examinees are successful the first time, with almost all (i.e., 98-99%) being successful after a repeat assessment. Generally, only 1-2 % have been found to need additional remedial education or support.
5. Is anyone exempt from writing the RCA (e.g., new graduates)?
• Inactive registrants are not required to complete the RCA.
• Those who pass the entry to practice examination (i.e., Physiotherapy Competency Exam – PCE) will be exempt from writing the first RCA after they pass. For a look at the RCA timing relative to the PCE, see the table below:
|If you passed the clinical PCE in…||You will take the RCA in…|
|2011||2015 or 2018|
|2012||2015 or 2018|
6. Why did the College choose an online written exam format for the RCA?
Mandated by a change in the provincial legislation (Health Professions Act) in 2007, the College is now expected to provide the public with evidence of the competence of its registrants.
Periodic assessment of 3,000 physical therapists in a valid, reliable manner that allows for some objective ‘evidence’ of continuing competence is a challenging enterprise.
Over a two-year period, a comprehensive review of the educational and assessment literature was completed by the College committees and staff, with the assistance of experts in performance assessments. This review process included exploring the current best practices in assessing continuing competence across Canada and internationally. Many assessment approaches were considered including a variety of written testing formats, clinical tests, portfolios and onsite assessments. The QAP Backgrounder lists the assumptions, guiding principles and design features that were used as a guideline to make decisions about the program. It also explains other aspects considered in decision-making.
Upon careful consideration, a written test was selected as the best approach for the College to meet the regulatory requirements of a consistent, valid and reliable assessment of the continuing competence of physical therapists’ knowledge, knowledge application, clinical reasoning and decision-making skills in a cost effective way. The style of questions and corresponding answers for the written test is a flexible one – called ‘Key Features’ – to assess approaches to common situations encountered in everyday physical therapy practice.
The committee recommended a written test using a Key Features case-based format due to its strong validity and reliability, and also because this approach is economically sustainable and administratively feasible. The questions can be designed to test the application of knowledge to practice – i.e., what clinicians actually ‘do’ in their area of practice.
Although there is no perfect tool for measuring continued competence, the College is confident that a written exam will satisfy the legislative requirements and also be fair to British Columbia physical therapists. The Board approved the decision to deliver the RCA online to best satisfy logistical, administrative, and cost implications for both registrants and the College.
7. What assessment tools did the College consider before choosing a written exam for the RCA?
Following an exhaustive research process of environmental scanning and review of key literature to understand issues, options and alternatives for best practices in continuing competence, a framework for the Quality Assurance Program from the College was established, overseen by a committee of physical therapists, other health professionals and public members.
The approved framework acknowledged that there is no perfect assessment tool to assess continuing competence. The Quality Assurance Program Advisory Team explored many assessment tools in detail before recommending that a written test was a very cost-effective, efficient baseline assessment tool for the periodic assessment of all physical therapists in British Columbia.
Assessment tools that were carefully considered, along with a ‘top line’ summary of the limitations that defined them as unsuitable, are outlined below:
- Portfolios – A portfolio is a collection of documents, reflections and experiences in hard or electronic form that reflect the practitioner’s career and demonstrates evidence of ongoing learning and the maintenance of competence. The assessment literature concerning portfolios suggests that, although useful for development and feedback, limitations include subjective and often cumbersome scoring if there are multiple raters/scorers. Operationally the administration and tracking of portfolios is very costly (i.e., much more costly per person than a written test).
- Continuing Education Credits – While collecting Continuing Education credits have some appeal, their use as an assessment tool for competence was not supported by the assessment literature. They are a ‘support’ to competence but not a tool to assess competence.
- Clinical Examinations – Simulated case examinations, such as Objective Structure Clinical Examinations, have very positive features for accurate assessment. An important factor, however, is that the cost per person for the clinical examination approach is 2-3 times the cost of a written test. An additional consideration was that many physical therapists in British Columbia have not had experience with such a format. Administering this form of assessment across a large province such as British Columbia would not be cost effective, requiring either travel to many small locations by assessors or travel to one central location by registrants. Due to administrative reasons and foreseen acceptability challenges from registrants, this format was deemed not acceptable to the College.
- On-Site Visits – The feasibility of on-site visits was explored, as well as the experiences of the College of Physiotherapists of Ontario. The assessment literature regarding on-site visits is not well developed but many similar features to the portfolio approach have been noted (e.g., subjectivity of scoring, administrative complexity, costs for large numbers of assessments). Administering assessments across a large province such as British Columbia would be very costly (i.e., more costly per person than a written test).
- Written Assessment – A written assessment option was explored and the assessment and administrative features proved to be very appealing. While it was noted that the written assessment would only effectively evaluate ‘some’ of the facets of a physical therapist’s practice, it was viewed as a reasonable approach to regular screening of important aspects of practice – physical therapy knowledge, application of knowledge, clinical reasoning and decision-making.
8. What are other health professions colleges in British Columbia using for their quality assurance programs?
In a recent inventory of a sample of eight other colleges, there appears to be a variety of approaches being used across the health professions in their quality assurance programs. Most are using multiple approaches to assessment including a regular reporting or self assessment of some aspects of competence (i.e., as is done with the College’s Annual Self Report) and a more thorough objective assessment within a defined period (e.g., 5, 6 or 10 years). Two other colleges are planning or using written tests (College of Dental Hygienists of British Columbia, College of Occupational Therapists of British Columbia).
9. Are there published protocols used to guide the development of the RCA?
There are established standards for developing and administering assessments. Writing questions (i.e., also known as ‘items’) is both an art and a science. The College has established consistent examination standards and best practices. This includes (but is not limited to):
• Following a defined blueprint;
• Identifying appropriate clinical scenarios that are realistic and relevant to physical therapy in British Columbia;
• Identifying specific key features that are to be addressed in the scenario (both regulatory and clinical);
• Following a 12 step process to write the items (cases, questions, answers): Case Development Steps, Roles and Responsibilities;
• Key validation takes place after each administration to ensure items are functioning appropriately; and
• Reviewing and revising of items and cut-score using data from each administration.
10. What did registrants tell us about the RCA when they ‘test drove’ sample questions?
The Quality Assurance Program and RCA were presented at the Annual General Meeting in 2010 and, subsequently, at a number of educational sessions around the province. The educational session included a presentation as well as workshop activities where physical therapists had an opportunity to ‘test drive’ three sample cases. The following section briefly summarizes central messages received from attendingphysical therapists who submitted formal feedback forms.
In total, 263 feedback forms were submitted. Approximately 608 individuals attended the various presentations between April – June 2010, resulting in a response rate of approximately 43%. This is a high response rate for this type of feedback report.
The demographic information indicates that the respondents were similar to the registrants of the College in most categories: gender, age, years of practice, practice settings and clinical practice areas. Respondents tended to work either with adults, seniors or children, while almost half of the College members report working primarily with all ages. Given the high response rate, and the strong representation across the demographics of the College membership, the results of the feedback can be accepted as representative of the membership.
Three sample cases were presented – Musculo-skeletal – MSK (geriatrics), Neurology and Cardiorespiratory. Using a five-point Likert scale from Completely Disagree to Completely Agree, respondents rated the cases on realism, whether they reflected current practice and whether they were suitable cases to assess physical therapy practice.
All three cases were rated highly with the majority of respondents Agreeing or Completely Agreeing that the cases were realistic (84-88%), reflected current practice (72-75%) and were appropriate for the assessment of physical therapy practice (67-72%).
For each specific question, they were asked to evaluate the difficulty of the question on a three-point scale: Too Difficult (beyond essential practice), Fair (part of essential practice) or Too Easy (below essential practice). Questions were almost universally rated as Fair (81-98%).
Other feedback on the RCA from the consultations with registrants:
- Overall, the data is highly positive regarding the Key Feature cases.
- The data indicated that the physical therapists who answered the sample questions found that they were quite fair.
- Suggestions and comments received about specific points in the sample cases have already been used to refine or improve the final questions and answers that will be presented in the RCA.
- Most of the registrants’ questions and comments indicated that they wanted more information about how the program will roll out. This desire for more information has been a College focus and has resulted in additional presentations at the Annual General Meeting as well as the updating/expansion of information on the College website and more RCA-dedicated information in the UPDATE newsletter.
11. What will the RCA test?
The RCA will focus on core competencies that arise in everyday practice.
Registrants are required to demonstrate continuing competence in one of four practice contexts in the blueprint for the RCA.†
The four practice contexts are:
- Adult + Older Adult Musculoskeletal
- Adult + Older Adult Neuromuscular
- Adult + Older Adult Cardiorespiratory
- Children and Youth
Each case has a scenario and three to five questions associated with it. Questions will cover both regulatory and clinical aspects of the case.
As an example, a case might describe a clinical situation where a physical therapist makes a care plan for his/her patient and wants to assign a particular task to a support worker. Questions will test regulatory knowledge about whether it is appropriate to assign a specific task to a support worker, and how to do so. Another regulatory question might address infection control principles relevant to the particular patient described in the case.
Clinical questions will ask you to identify key questions in a subjective assessment, key components of the physical assessment, or to identify the most relevant treatment approaches. All clinical questions are based on currently accepted practices and must be referenced.
†Registrants who participated in the pilot test of the RCA were able to choose two practice contexts (i.e. half of the questions in one context and half in the second context). This option will not be offered in future due to concerns with reliability. The Spring 2015 newsletter expands more upon this and can be found here.
12. How can I prepare for the RCA?
• The Blueprint for the College’s Quality Assurance Program provides a detailed outline of the purpose, definition and standards of the program and can be accessed here.
• The RCA will include 65% (+/- 10%) of questions relating to the essential competencies. 35% (+/- 10%) of questions will be on regulatory topics. The Resources tab (at the top of each page throughout the site) includes all of the key documents about regulatory topics.
• The RCA tests clinical reasoning so it is not expected that registrants need to study clinical content in order to be successful. However, for those interested in reviewing references utilized by case writers to develop the content, a frequently used reference list can be found here.
• Samples of Key Features cases are available on the website here.
• RCA Code of Conduct is here.
13. Why can’t I choose to write the RCA with two contexts?
For the pilot test in 2014, registrants were invited to choose to complete the RCA in either one or two of the four practice contexts. Those who chose two contexts did half of the RCA in one context and half in the other. Some important concerns due to this approach surfaced in the pilot test results. First, projections based on the results revealed that reliability for RCA forms containing two practice contexts may be too low to assign scores or make pass/fail decisions about either of the two practice contexts with the degree of statistical confidence recommended for this type of examination. According to established standards for test development, exams containing two different practice contexts should be thought of as two separate exams – one for each practice context. Therefore scores and reliability should be calculated and reported for each of the two contexts separately. The pilot test results also provided a strong indication that in order to keep the total testing time within a desirable time limit it is not possible to include enough cases and questions in each of two contexts to reach the degree of reliability necessary for each of them.
In addition, the number of registrants who chose to complete the pilot test in two practice contexts was low. Projections based on these numbers indicated that there would not likely be enough registrants completing the RCA in two contexts to allow for a large enough sample size at each sitting of the RCA to complete essential statistical analyses that are dependent on sample size.
In order to maximize reliability, keep the RCA within a reasonable time limit so as not to over-burden registrants, and ensure large enough sample sizes for proper statistical analyses, the Board of Examiners recommended that each RCA consist of a single practice context. The Quality Assurance Committee and the Board of Directors agreed with this recommendation. The Spring 2015 newsletter expands more upon this and can be found here.
14. I wrote the RCA in 2015 and the content didn’t reflect my practice. How can you assess my competence with an exam that doesn’t reflect my practice?
The RCA is not designed to reflect individuals’ practice in the same way an annual performance appraisal with your employer does. The RCA is designed to reflect day-to-day practice for the majority of physical therapists practicing in BC in each of the four practice contexts. Designing an exam that is as individual as each of the registrants practicing in BC is not the intention. The content is guided by the Quality Assurance Program Blueprint, a document that was developed in 2008 with the guidance of experts in the field through a lengthy exercise that included consultation with registrants. The Blueprint was revised in 2011 and will be reviewed regularly to ensure that it remains current, but it will not be changed significantly between the first two sittings of the RCA.
The RCA is not designed to assess expertise. The mandate of the College is to protect the public from registrants who do not meet a minimum standard of competence in terms of practice that is safe, ethical, and effective. Therefore, the RCA is not a means to demonstrate the full extent of your knowledge and ability to apply it; rather, it is an opportunity to demonstrate that your practice meets a minimum standard. While it’s not exciting for our high achieving profession to reflect on competence at a minimal level, it is what is required by the mandate of our College towards the public, and it is what is possible given the practice variety present among our 3700 registrants.
Finally, as per the QAP Blueprint, the standard for the RCA is the Essential Competency Profile for Physiotherapists in Canada (2009). Each question must be mapped to either a key/enabling competency or a specific regulatory topic. This ensures that the exam focuses on competencies that have been agreed to be essential to the practice of physical therapy. While there may be some physical therapists whose practice largely falls outside of that which is represented on the RCA, the College expects that the number will be very small.
15. What are the differences between the ASR and the RCA?
|Purpose is for registrant to demonstrate competence through assessment||Purpose is for registrant to maintain competence through self-assessment|
|Completion is a requirement for registration renewal||Completion is a requirement for registration renewal|
|Must be completed in one 3-hour session||Can be completed over multiple sessions|
|Session must be pre-booked||No need to pre-book session|
|Registrants must identify a proctor who will invigilate||No proctor is required|
|Cannot consult resources during RCA (not ‘open book’ format)||Can consult resources during ASR (‘open book’ format)|
|Online platform||Online platform|
|Cases embedded within registrant’s choice of one of four practice contexts:
• Adult and Older Adult Cardiorespiratory
• Adult and Older Adult Musculoskeletal
• Adult and Older Adult Neuromuscular
• Child and Youth
|Cases embedded within registrant’s choice of one of four practice contexts:
• Adult and Older Adult Cardiorespiratory
• Adult and Older Adult Musculoskeletal
• Adult and Older Adult Neuromuscular
• Child and Youth
|20-25 cases (100-110 questions)||4 cases (12 questions)|
|Questions cover regulatory topics and essential competencies (Essential Competency Profile for Physiotherapists in Canada, 2009)||Questions cover regulatory topics|
|Cases/questions are written by volunteer registrants||Cases/questions are written by volunteer registrants|
|Criterion referenced: pass score is determined by the level of difficulty of all questions||No pass score|
|The College knows individual results||The College does not know individual results|
|Individual results are not disclosed to employer||Individual results are not disclosed to employer|
|Questions can be answered using a mouse or touch pad; typing is not required.||Questions can be answered using a mouse or touch pad; typing is not required.|