Senior Product Manager, Scoring & Assessment Systems
National Conference of Bar Examiners, Madison, WI, United States
The Senior Product Manager, Scoring & Assessment Systems will own the product strategy, vision, and evolution of scoring across NCBE's exam products. This senior product leadership role provides clear decision authority and is responsible for defining how scoring works at NCBE—from human grading models to the responsible adoption of automated scoring. The role requires strong product judgment, industry perspective, and communication skills to evaluate tradeoffs and make recommendations that shape the future of high‑stakes legal licensure assessment.
This Senior Product Manager will serve as the single accountable product leader on scoring decisions, working across test development, psychometrics, operations, and technology. When priorities conflict or perspectives differ, this role will guide teams toward decisions grounded in best practices, data, and product outcomes.
Essential Duties and Responsibilities Own Scoring Product Strategy & Decisions
Define and drive NCBE’s scoring strategy across multiple‑choice and constructed‑response item types.
Own the product vision and roadmap for scoring, aligning it to organizational priorities.
Make informed product decisions on scoring approaches, balancing validity, reliability, operational feasibility, and grader and examinee experience.
Serve as the decision‑making authority when tradeoffs arise across stakeholder groups.
Lead Evolution of Scoring Models (Human & Automated)
Evaluate and guide the adoption of automated scoring, including where and how it should (or should not) be used.
Articulate the benefits, risks, and limitations of different scoring approaches, including hybrid human/AI models.
Partner with psychometric and research teams to ensure appropriate validation, calibration, and evidence frameworks.
Bring Industry Best Practices to NCBE
Apply industry experience and best practices from leading assessment organizations to inform scoring design, workflows, and policies.
Ensure NCBE’s approach reflects current best practices in large‑scale, high‑stakes testing.
Introduce improvements to rubrics, training materials, and scoring processes (e.g., training papers, calibration methods).
Manage staff in the day‑to‑day performance of their roles, including mentoring, setting performance goals, reviewing performance, training, professional development, recruitment, and onboarding. Foster a high‑performing, accountable team environment aligned to NCBE’s product strategy and organizational goals.
Translate Strategy into Product Requirements
Define clear product requirements for scoring systems, workflows, and tools.
Partner with technology teams to deliver scalable, reliable scoring platforms and integrations.
Maintain a forward‑looking roadmap for scoring capabilities.
Influence & Align Cross‑Functional Stakeholders
Work closely with test development, psychometrics, operations, and external partners to align on scoring approaches.
Work with senior stakeholders to make decisions and align teams when priorities conflict.
Communicate and defend product decisions with clarity and credibility.
Build alignment across teams, especially in areas of ambiguity or disagreement.
Represent Scoring Internally and Externally
Serve as a key point of contact for scoring‑related questions, including with external partners (e.g., jurisdictions such as JX).
Present strategy, decisions, and outcomes to executive leadership, governance boards and committees, and external stakeholders.
Clearly explain complex scoring concepts to both technical and non‑technical audiences.
Qualifications Required Experience
5–10+ years of experience in large‑scale assessment, licensure, or certification testing.
Direct experience owning or leading an assessment scoring product, capability, or major component end‑to‑end (not solely supporting, coordinating, or analyzing).
Experience working at a recognized or high‑quality assessment organization.
Demonstrated ability to make, own, and defend product decisions in complex, cross‑functional environments.
Experience writing product requirements and working closely with technology teams.
Scoring & Assessment Expertise
Strong understanding of constructed‑response scoring and rubric design.
Rater training, calibration, and monitoring (including training papers).
Inter‑rater reliability and why it matters.
Core concepts of validity and scoring quality (no advanced psychometric analysis required).
Practical knowledge of scoring operations at scale, including tradeoffs between quality, speed, and cost.
Automated Scoring Familiarity
Working knowledge of automated or AI‑assisted scoring approaches, including strengths and limitations of different models.
Validation and monitoring considerations.
Appropriate use cases in high‑stakes environments.
Product & Leadership Skills
Strong product management mindset with the ability to translate strategy into execution.
Proven ability to influence without authority and navigate competing perspectives.
Excellent communication skills, including the ability to explain and justify decisions to senior and external audiences.
Able to communicate confidently in high‑stakes discussions.
The Senior Product Manager may work remotely. NCBE will consider applicants from the following states: FL, IA, IN, MA, MD, ME, MI, MO, MN, NE, NJ, NM, NV, OK, PA, WA, and WI.
This position may include up to 10% travel for meetings with NCBE’s Board of Trustees, committees and/or vendors, and/or for conferences, workshops, and/or outreach.
NCBE is proud to be an equal employment opportunity organization. We are committed to providing equal employment opportunity to all applicants and employees.
#J-18808-Ljbffr
This Senior Product Manager will serve as the single accountable product leader on scoring decisions, working across test development, psychometrics, operations, and technology. When priorities conflict or perspectives differ, this role will guide teams toward decisions grounded in best practices, data, and product outcomes.
Essential Duties and Responsibilities Own Scoring Product Strategy & Decisions
Define and drive NCBE’s scoring strategy across multiple‑choice and constructed‑response item types.
Own the product vision and roadmap for scoring, aligning it to organizational priorities.
Make informed product decisions on scoring approaches, balancing validity, reliability, operational feasibility, and grader and examinee experience.
Serve as the decision‑making authority when tradeoffs arise across stakeholder groups.
Lead Evolution of Scoring Models (Human & Automated)
Evaluate and guide the adoption of automated scoring, including where and how it should (or should not) be used.
Articulate the benefits, risks, and limitations of different scoring approaches, including hybrid human/AI models.
Partner with psychometric and research teams to ensure appropriate validation, calibration, and evidence frameworks.
Bring Industry Best Practices to NCBE
Apply industry experience and best practices from leading assessment organizations to inform scoring design, workflows, and policies.
Ensure NCBE’s approach reflects current best practices in large‑scale, high‑stakes testing.
Introduce improvements to rubrics, training materials, and scoring processes (e.g., training papers, calibration methods).
Manage staff in the day‑to‑day performance of their roles, including mentoring, setting performance goals, reviewing performance, training, professional development, recruitment, and onboarding. Foster a high‑performing, accountable team environment aligned to NCBE’s product strategy and organizational goals.
Translate Strategy into Product Requirements
Define clear product requirements for scoring systems, workflows, and tools.
Partner with technology teams to deliver scalable, reliable scoring platforms and integrations.
Maintain a forward‑looking roadmap for scoring capabilities.
Influence & Align Cross‑Functional Stakeholders
Work closely with test development, psychometrics, operations, and external partners to align on scoring approaches.
Work with senior stakeholders to make decisions and align teams when priorities conflict.
Communicate and defend product decisions with clarity and credibility.
Build alignment across teams, especially in areas of ambiguity or disagreement.
Represent Scoring Internally and Externally
Serve as a key point of contact for scoring‑related questions, including with external partners (e.g., jurisdictions such as JX).
Present strategy, decisions, and outcomes to executive leadership, governance boards and committees, and external stakeholders.
Clearly explain complex scoring concepts to both technical and non‑technical audiences.
Qualifications Required Experience
5–10+ years of experience in large‑scale assessment, licensure, or certification testing.
Direct experience owning or leading an assessment scoring product, capability, or major component end‑to‑end (not solely supporting, coordinating, or analyzing).
Experience working at a recognized or high‑quality assessment organization.
Demonstrated ability to make, own, and defend product decisions in complex, cross‑functional environments.
Experience writing product requirements and working closely with technology teams.
Scoring & Assessment Expertise
Strong understanding of constructed‑response scoring and rubric design.
Rater training, calibration, and monitoring (including training papers).
Inter‑rater reliability and why it matters.
Core concepts of validity and scoring quality (no advanced psychometric analysis required).
Practical knowledge of scoring operations at scale, including tradeoffs between quality, speed, and cost.
Automated Scoring Familiarity
Working knowledge of automated or AI‑assisted scoring approaches, including strengths and limitations of different models.
Validation and monitoring considerations.
Appropriate use cases in high‑stakes environments.
Product & Leadership Skills
Strong product management mindset with the ability to translate strategy into execution.
Proven ability to influence without authority and navigate competing perspectives.
Excellent communication skills, including the ability to explain and justify decisions to senior and external audiences.
Able to communicate confidently in high‑stakes discussions.
The Senior Product Manager may work remotely. NCBE will consider applicants from the following states: FL, IA, IN, MA, MD, ME, MI, MO, MN, NE, NJ, NM, NV, OK, PA, WA, and WI.
This position may include up to 10% travel for meetings with NCBE’s Board of Trustees, committees and/or vendors, and/or for conferences, workshops, and/or outreach.
NCBE is proud to be an equal employment opportunity organization. We are committed to providing equal employment opportunity to all applicants and employees.
#J-18808-Ljbffr