An Empirically Validated Metric of Student Growth

A primary difficulty using data from early literacy assessments is understanding how the various components fit together to inform student progress and next steps for instruction. Through the application of an Item Response Theory (Rasch modeling), we are able to empirically address these questions. By examining individual difficulties of key items and concepts, we can see how these components "map" for individual students.

Rasch modeling explanation
Under the Rasch analysis, each item has an estimated difficulty which places it in a unique position on the overall scale. If STEP functions as a developmental scale, items should cluster within the scale by sub-component. That is, the Concepts about Print items, for example, should tend to be easier than the Phonological Awareness items which in turn should be easier that than most of the reading accuracy ratings. Similarly, within each sub-component, the items difficulties should also order in theoretically predicable ways, e.g. the rhyming items within Phonological Awareness should generally be easier than items that examine a child's skill at segmentation.

STEP Developmental Literacy Scale
Please download the "STEP Developmental Literacy Scale" graph now (available in PDF). It is explained below.

Concepts about Print / Letters Names and Sounds
As expected, the easiest items within the STEP Developmental Literacy Scale focus on concepts about print and letter names and sounds. Within Concepts about Print, the easiest item is "knowing that you read the text" (and not the pictures). Next, students learn about directionality - reading left to right. The most difficult concept about print is understanding 1-to-1 matching of voice to words on the page.

Looking at the next boxplot, representing letter names and sounds, we see that as students gain more understanding of Concepts about Print, they are also learning to identify letters. Children tend to learn upper case letters first (-6.1 mean difficulty for 26 items for upper case; -5.2 mean difficulty for 28 lower case including print and type face fonts for 'a' and 'g') and to eventually associate letters with the sounds that they make (-3.0 mean difficulty for 26 items.)

Phonological Awareness
The item difficulties for the Phonological Awareness sub-component indicate that these skills are developing in tandem with Concepts about Print and Letter Names and Sounds. Rhyming words comes first (-4.3 mean item difficulty), followed by matching words that begin with the same first sound, (-3.4 mean item difficulty) and finally learning to segment words into individual phonemes(-1.6 mean item difficulty). Note that being able to consistently segment words into phonemes does not occur until students are accurately reading text between Step 3 and Step 4.

Text Reading Accuracy
Next come the results for Text Reading Accuracy, which follows a monotonic progression. The jumps in item difficulty across Steps 2 through 5 are relatively large. Students at this stage in learning to read are acquiring many new strategies and skills, and correspondingly, the ability to read accurately the next level text represents a significant developmental gain. A child with a scale score of -4.1 has a probability of 0.90 for reading a Step 2 text with mastery. To achieve a similar likelihood of mastery at Step 3 implies a scale score of -3.2; the corresponding jumps to reading accurately at Steps 4 and 5 translate into scale scores of -1.6 and -0.1 respectively. In contrast, for Step levels 5 and beyond, the distance between item difficulties become significantly smaller. Although students are continuing to learn new strategies and skills to decode more complex texts, the actual amount of development occurring here appears less than at lower step levels.

Reading Rate
Reading Rate item difficulties and Text Reading Accuracy levels display an interesting relationship with one another. As expected, the ability to read a text accurately precedes being able to read the same text with appropriate speed or rate. That is, the item difficulty associated with the reading rate at each step is significantly higher than the corresponding reading accuracy for that step. Of significance, these gaps become larger as students progress into reading the more complex texts involved in the upper levels of STEP. For example, the gap between accuracy and rate at Step 5 is approximately 0.5 logits on the scale. By Step 10, the gap has increased to 2.0 logits. These results are consistent with published research that the ability to decode accurately more complex texts represents modest developmental gains, whereas to read such texts with fluency represents substantially greater improvement. This is highly significant because fluency in turn links most directly with comprehension.

Comprehension
The boxplot for Comprehension displays the median item difficulty for the 5 to 8 comprehension questions associated with each text at each step level. Through Step 5, the difficulties for accuracy, rate and comprehension are fairly similar at each respective step. This implies that students at these levels are learning in tandem to read accurately, with a reasonable rate, and to comprehend these simple texts. As the text become more complex, however clear skill differentiation begins to emerge. As we saw with reading rate, a substantial gap appears at Step 6 and beyond that separates the comprehension item difficulties from a simple accurate decoding of text. To decode accurately these more complex texts represents an improvement, but the big developmental gains are associated with doing this with fluency and comprehension.

Developmental Spelling
Finally, we see that items difficulties for Developmental Spelling also follow the expected pattern moving from writing initial and ending letters to short vowels and blends to long vowels and so on. Moreover, these item results align in a reasonable fashion with the other sub-components in STEP. For example, we see that students who are accurately reading text at Step 5 are also likely to be writing short vowels in developmental spelling (i.e., Step 5 accuracy and writing short vowels have approximately the same item difficulty.) Similarly, students who are reading a step 9 text accurately are spelling long vowel and r-controlled vowel patterns as well.

Taken overall, the item difficulty alignment of the sub-components within STEP (as well as the sub-tasks within each sub-component) is fully consistent with established scientific research on reading skill acquisition. These empirical results add credence to the construct validity of STEP as an integrated development scale.