From Practice Analysis to Test Redesign: Looking Ahead to Phase Three
Fall is in the air, and for most people that means football. For the Testing Task Force, it means we’re preparing for Phase 3 of our study, when we will develop recommendations for redesigning the bar exam. (Don’t worry, we’re still managing to make time for a little football, too.) In this blog post, we discuss how the results of the practice analysis from Phase 2 of our study and stakeholder input from Phase 1 will support the work of Phase 3.
When considering what is necessary for newly licensed lawyers (NLLs) to be minimally competent to practice law, it is important to note that from a testing perspective, there are two sides to the minimum competence coin: (1) content standards guide what is tested and how it is tested, while (2) performance standards help determine what level of proficiency or performance is required and where the passing score should be set. During Phase 3 of our study, we will be focusing on the content standards side of the coin. Decisions about what should be tested will be informed by the results of our practice analysis, while considerations about how content should be tested (issues of format, timing, delivery mode, and frequency) will be guided by the input we received from stakeholders during Phase 1, supplemented by additional stakeholder input that will be gathered during Phase 3. Only after our bar exam redesign work is completed can the question of performance standards be considered (this is typically done through a process known as standard setting ).
Research into the competencies needed by NLLs is nothing new and has been performed by several groups and organizations over the years, including NCBE . So why is the Testing Task Force conducting more research by doing yet another practice analysis? The knowledge, skills, abilities, and other characteristics (KSAOs) needed for minimum competence licensure may change as the occupation or profession changes. Therefore, it is considered a best practice to conduct a practice analysis every 5–10 years to confirm whether there have been significant changes in the work activities of the profession for which the licensing exam is used. A current practice analysis is particularly relevant now for the legal profession given the potential impact of technology and evolving societal expectations about legal services.
In developing the 2019 practice analysis, we took a different approach than was used for our 2012 study. The 2012 practice analysis was structured around practice areas, meaning that the survey asked about job tasks in several different practice specialties (e.g., Family Law, Criminal Law). For the 2019 practice analysis, we have instead grouped job tasks into four broad categories that are practice-area neutral. Currently, in every jurisdiction, a license to practice law is a general license, so we need the practice analysis data to highlight the general set of KSAOs that all NLLs should have regardless of their practice areas or specialties. This change to the structure of the 2019 survey is also consistent with input from stakeholders who told us during our Phase 1 listening sessions that the bar exam should emphasize testing skills of general application and core legal knowledge.
One of the benefits of a practice analysis survey over other methods of determining the content to be tested is the ability to conduct a variety of data analyses using the job task and KSAO ratings and the demographic characteristics of the respondents, including but not limited to practice areas, practice settings, location of primary office, race, ethnicity, gender, years of practice, and size of firm or organization. The rich data set we are collecting will permit analyses by subgroup, such as comparing ratings of job tasks and KSAOs as a function of work setting, geographic location, years of experience, and specialty/practice area. For example, subgroup analyses will help us to differentiate core job tasks applicable across many practice areas from those tasks performed primarily by specialists. Data analyses by race, ethnicity, gender, and other characteristics will also permit evaluation of any potential differences in perspectives among subgroups of respondents. Where the data shows convergence in perspectives among the thousands of respondents to the survey, it will provide a strong argument that a job task is (or is not) important to consider for inclusion (or exclusion) as possible exam content.
While the Task Force regards the quantitative practice analysis survey data as being vital for providing current information on entry-level competencies for the legal profession, we also value the types of in-depth qualitative information that can be obtained from focus groups and panels of subject matter experts (SMEs). Our practice analysis survey was crafted using SME focus groups of NLLs and seasoned attorneys who currently have, or have had, direct experience working with or supervising NLLs. NLLs know best what job tasks they perform and how frequently, while seasoned lawyers provide a more experienced perspective regarding the criticality or importance of the job tasks and KSAOs to practice effectively. For this reason, both groups are also being asked to answer the 2019 practice analysis survey. Because the purpose of the survey is to gather data about the work performed by NLLs and the KSAOs they need to perform that work, all respondents are instructed to answer the survey from that perspective: NLLs are instructed to answer based upon their own practice experience, and seasoned lawyers are instructed to answer based upon the practice of NLLs with whom they have had direct experience working or supervising, not upon their own practice.
We are excited about the wealth of information the results of the survey will reveal. And we want to close by noting that the practice analysis will undoubtedly identify some KSAOs that are not appropriate for inclusion in the bar exam, for various reasons, but might be appropriate for training of new lawyers by other stakeholders, such as law schools, bar associations, CLE providers, and employers. We look forward to publishing a detailed report of the results of the practice analysis survey early next year so that the findings can benefit the profession as a whole.
1. Standard-setting studies have been done by various bar admission agencies in the past (and more recently by California in 2017) to systematically determine where to set their passing scores by applying a policy definition of minimum competency. See, e.g., Michael T. Kane, Ph.D., et al., “Clearing the Bar: How to Set the Standard,” 70(4) The Bar Examiner (November 2001) 6-20, for information about past studies in Ohio, Florida, Minnesota, New York, and Pennsylvania.
2. To name just a few of the studies, see, for example, Robert MacCrate, et al., Legal Education and Professional Development – An Educational Continuum (Report Of The Task Force On Law Schools And The Profession: Narrowing The Gap) (1992); Marjorie Shultz & Sheldon Zedeck, Final Report—Identification, Development, and Validation of Predictors For Successful Lawyering (2009); NCBE’s 2012 practice analysis, A Study of the Newly Licensed Lawyer; Neil Hamilton, Law Firm Competency Models and Student Professional Success: Building on a Foundation of Professional Formation/Professionalism, 11 U. St. Thomas L.J. 6 (2013); and Alli Gerkman & Logan Cornett, Foundations For Practice: The Whole Lawyer and the Character Quotient (July 2016). To our knowledge, NCBE’s 2012 practice analysis was the first ever done for the legal profession. The IAALS Foundations for Practice survey in 2016 gathered information about the “skills, characteristics, and competencies” that new lawyers need “to launch a successful career” and some limited information about job tasks that new lawyers perform in their first year of practice.