Program Management: A Program Manager (PM) will be assigned to you, and will be responsible for managing the overall process for any or all of your certification exams. Your PM will not only manage the processes, but will be your primary point of contact for questions about these processes, as your exams are developed and published. Your PM will coordinate with all of the resources needed during the development process, including your SMEs, Psychometricians, Facilitators, and your Test Delivery Vendor. They will also coordinate scheduling, document handling, and obtaining of approvals through out the various stages.
Global Job Task Analysis:
The global JTA, sometimes called a Certification
Program Development Workshop, is a workshop that focuses on
the purpose, goals and needs for developing your certification
program. It is a big picture view of your entire certification
program. CCS facilitators will lead your executive team through exercises
designed
to fully define and investigate the need
for investing in your certification program prior to launching it,
or help to revise an existing program. The deliverables that come out of
a global JTA are the
preliminary test definition documents, and the
preliminary task expansion for each exam identified.
These will
be finalized during the exam-level JTA.
Exam Job Task Analysis: The exam-level JTA, sometimes called Competency Modeling, is the workshop that focuses on defining the complete testing domain for that given exam. CCS facilitators will lead your Subject Matter Experts (SMEs) through exercises designed to fully define the domain of information and the corresponding exam. Typically there are three main deliverables that come out of an exam-level JTA. The first deliverable will be the completed and finalized test definition document. The second deliverable will be the task expansion document. The third deliverable is
Exam Blueprinting: The exam blueprint is a document that lists all of the objectives for a given exam, and the number of items (questions) that need to be written for each objective. It is a document that tells the writers exactly how many items to write. The test objectives that are defined during the JTA, are evaluated and weighted during the JTA, and the total number of items per objective are identified. This is completed as part of the exam level JTA. Some clients choose to also validate those numbers by developing a blueprint survey that is sent to a larger group of subject matter experts, anywhere from 20 to 50 people. CCS will compile and analyze those survey results, and produce a final blueprint that will be used during the item writing.
Item Development: The items for the exam are developed during a week-long Item Development Workshop. This is a five-day concentrated workshop attended by the Subject Matter Experts (SMEs), where all of the items are written, edited, and technically reviewed during that week. CCS facilitators will instruct SMEs in appropriate, psychometrically sound item writing methodology. This workshop is specifically designed to produce a single exam that meets the highest standards for reliability and validity, that is edited for language and grammar to meet psychometric standards, and that is reviewed by the SMEs for technically accuracy and content validation, and it is all completed in five days.
Exam Data Analysis: We provide a complete suite of data analysis processes designed to help in the selection of statistically sound items for your certification exams. CCS Psychometricians will provide complete, comprehensive reports through each phase of analysis to help document, validate and protect your certification program.
The three main data analyses are:
|
|
Item Selection:
Our Psychometricians will analyze item level statistics gathered
during your defined beta exam period. They will provide information on how each item performed, and provide suggestions on which items to keep or delete from the final exam. |
|
|
Forms Build: Our Psychometricians will analyze difficulty and performance timing for all items chosen during the item selection analysis, and build equal and balanced exam forms to be used on the final exam. Thus helping ensure a consistent and fair testing experience to all candidates,regardless of which form of the exam they take. |
|
|
Inservice Analysis:
Our Psychometricians will analyze the item level statistics
gathered while your exam has been available. This is done to
evaluate how existing exams are performing, as compared to how they were expected to be performing when they were first developed. |
Cut Score Development:
Our Psychometricians will consult with
you regarding the particular needs, and limitations of your program. Based
upon that information, they will suggest the cut score setting
methodology (Borderline, Angoff, or Bookmark) that is best for
your program. They will then take the data gathered from that methodology,
analyze it, and recommend to you a cut score for your exam that
will be fair, valid, and legally defensible.