A BCBA candidate recently asked a study group a question that sounds simple: what does “believability” mean in data collection, and how does it relate to IOA, accuracy, validity, and reliability? They kept seeing the term on mock exams and couldn’t find a clear explanation.
The short answer is that these five terms are related, but they are not interchangeable. Cooper, Heron, and Heward describe believability as how well you can convince yourself and others that the data are trustworthy and worth interpreting, and they identify IOA as the index most often used to support it. But IOA is evidence for believability. It is not the whole of it, and by itself it says nothing about whether the data are accurate or valid.
That gap is exactly where the BCBA exam likes to work. A tempting answer choice often sounds generally “good for data” while missing the specific measurement problem the stem describes.
High IOA can increase confidence in a measurement system, but it does not prove that the data are accurate or valid. Two observers can agree perfectly and both be wrong.
| Concept | The question it answers | What it does not establish by itself |
|---|---|---|
| Believability | How much confidence should we place in these data and the measurement system that produced them? | It is a judgment, not a single number. IOA is the most common index, but not the only evidence. |
| Validity | Are we measuring the behavior or dimension that is relevant to the purpose of measurement? | Agreement between observers does not make an irrelevant measure valid. |
| Accuracy | How closely does the observed value match the true or accepted criterion value? | Agreement between observers does not reveal accuracy unless there is a criterion for comparison. |
| Reliability | Does measurement remain reasonably consistent when it is repeated under comparable conditions? | Consistency does not guarantee that the thing being measured is the right thing. |
| IOA | To what extent do independent observers agree in their records of the same events? | High agreement does not prove accuracy, validity, or the absence of shared bias. |
The BACB 6th Edition Test Content Outline includes task C.8, “Evaluate the validity and reliability of measurement procedures.” That makes these distinctions fair game for questions that ask what a measurement result actually demonstrates, and what it does not.
Believability is the broadest term in this group. It refers to the confidence we have in the data as a trustworthy representation of what was measured. Think of it as the conclusion you are trying to earn, not a statistic you calculate.
Several features of a sound measurement system can strengthen it: a clear operational definition, appropriate observer training, a procedure matched to the behavior and the dimension that matters, independent observation, evidence of agreement, and, when a criterion value is available, evidence of accuracy.
If a question asks what would increase the believability of observational data, IOA may well be part of the answer. But if the stem asks whether the measurement is accurate or valid, “high IOA” is not automatically enough.
Validity concerns whether the measurement procedure captures the behavior or dimension that is relevant to the purpose of measurement. A procedure can be implemented consistently and still fail this test.
A team wants to know whether a student is academically engaged. Two observers record whether the student is sitting in the chair, and they agree on almost every interval.
The agreement may be excellent. But sitting is not necessarily the same thing as academic engagement; a student can sit quietly while doing no work at all. If the procedure does not adequately represent the behavior or dimension relevant to the decision, the core problem is validity, not observer agreement.
This is why “the data were collected consistently” and “the data answer the clinical question” are not interchangeable statements.
Accuracy requires comparison with a true or accepted criterion value. In behavioral observation, that criterion might come from a calibrated system, a carefully constructed reference record, or another defensible standard against which an observer’s record can be compared.
Suppose a criterion record shows that a response occurred 20 times during a session. Two independent observers both record 10 responses.
The observers agree perfectly with one another on the total count, yet both records substantially underestimate the criterion. Agreement is high; accuracy is poor.
Evaluating accuracy requires a criterion, which is why research that evaluates observational measurement against known values, such as Mudford et al. (2011), compares observers’ records with a criterion rather than only with one another. IOA compares observers with each other.
Reliability concerns consistency of measurement. If the same measurement process is applied under comparable conditions, a reliable system should not produce wildly inconsistent results merely because of the measurement process itself.
Behavior-analytic literature has historically used terms such as interobserver reliability when discussing agreement between observers, and that history is one reason students sometimes collapse reliability and IOA into a single concept. For exam purposes, it is safer to keep the levels of analysis separate:
A system can also be reliably wrong. A scale that adds five pounds every time may produce highly consistent readings while remaining inaccurate. Behavioral measurement can show the same problem when observers make a stable, systematic error.
Interobserver agreement (IOA) evaluates the extent to which two or more independent observers agree in their records of the same behavioral events. It is extremely useful because disagreement can reveal problems with operational definitions, observer training, complex response topographies, recording logistics, observer drift, or the measurement procedure itself.
But IOA is evidence about agreement. The strongest exam questions will tempt you to make it prove more than it does.
When a question gives you information about data quality, ask what comparison is actually being made.
That sequence is a heuristic, not a replacement for reading the stem carefully. A real measurement system may have strengths or weaknesses in several of these areas at once.
Suppose a BCBA collects IOA on a classroom observer’s duration recording of out-of-seat behavior, and agreement comes out at only 58%. Before interpreting the data, the analyst should look at the measurement system.
A sensible sequence is to inspect the operational definition and recording procedure, clarify or retrain as needed, and then collect additional independent observations. Averaging two discrepant records does not repair the measurement problem; it hides it.
High IOA does not mean “measurement problem solved.” If both observers are using the same flawed definition or making the same systematic error, they may agree while the data remain inaccurate or invalid.
Accuracy requires comparison with a criterion, not merely with another observer.
Agreement can occur while both observers consistently measure the wrong behavior or an irrelevant dimension.
Disagreement tells you the records differ. It does not identify the correct observer by itself.
A measure can be consistent without measuring the right thing.
Believability is a broader judgment about the trustworthiness of the data and the measurement process. IOA is the index most often used to support it, but it is not the same thing.
Apply the framework to three new scenarios. In each one, ask what comparison is actually being made before you look at the answer choices.
A criterion video record shows 18 instances of a response. Two independent observers each record 12 instances, so their total counts agree exactly.
Which conclusion is BEST supported?
The best answer is B. The observers agree with each other, but both fall well short of the criterion value of 18. IOA and accuracy answer different questions: IOA compares observers with one another, while accuracy compares the records with a criterion.
Why the others fail. A treats agreement as proof of accuracy. C goes too far: a shortfall against the criterion shows the records are inaccurate, but it does not by itself show the definition targets the wrong behavior. Missed instances, observer error, or an unclear definition are all possible, and the stem does not say which. D is false because IOA can be calculated whether or not a criterion record exists.
One more caution: matching totals show that the observers counted the same number of responses, not that they scored the same instances. Total-count IOA is a coarse index. Even perfect instance-by-instance agreement, though, would not reveal the shortfall against the criterion.
A school team wants to measure independent academic engagement. Observers record whether the student is seated, and IOA is consistently high. The student is often seated while looking away from materials and not responding.
What is the primary measurement concern?
The best answer is C. The observers may apply the procedure consistently, but the selected measure does not necessarily represent the behavior relevant to the team's question. That is primarily a validity problem.
Why the others fail. A invents an undercounting problem; nothing in the stem suggests the observers are miscounting seated behavior, and high IOA would not show that anyway. B is a blanket claim about interval recording that the stem does not support. D sets a standard that does not exist: believability is a judgment about the data and the measurement system, not something that requires perfect agreement.
A BCBA observes large changes in event-recorded behavior across sessions. During a simultaneous observation, a second observer obtains a substantially different count.
What is the BEST next step before concluding that the client's behavior is truly variable?
The best answer is A. Observer disagreement makes the measurement system a plausible source of the apparent variability. Evaluate agreement, diagnose the source of the disagreement, correct the procedure if needed, and then recheck it before drawing conclusions about the client's behavior.
Why the others fail. B blends two discrepant records without finding out why they differ, so the underlying problem stays. C changes the measurement procedure on the strength of a single disagreement; disagreement points to a problem to diagnose, not proof that event recording is the wrong method. D adds more data from the same unchecked system, and a stabilizing trend would not reveal which observer was correct.
Two observers can agree perfectly and still both be wrong.
BA Prep’s measurement questions are built around this same discipline: naming what comparison a stem is really making before you choose an answer. If a question has ever tempted you to treat agreement as proof of accuracy, that is exactly the discrimination our practice questions are designed to train.
Start Free Trial