The Method Comparison Argument

Date: 2015-10-07; view: 335.

As noted earlier, each side has claimed victory in method comparison studies. Skill-building advocates claim that children in skills-based classes learn to read better, while whole language advocates claim that whole language is superior, as long as it is defined correctly. I discuss here a recent contribution to this debate.

Jeynes and Littell (2000) reviewed 14 studies and concluded that overall, low SES children do not benefit from whole language instruction, but "there may be some advantages to the whole language approach in its purist form" (p. 21). Of the 14 studies, only four were listed as published in journals or books. Of the ten studies that were listed as unpublished, two, it turned out, were in fact published in the Reading Research Quarterly. I was able to locate five others through ERIC, and one other that I believe to be identical to a study on Jeynes and Littell's list. My interpretation of these studies is quite different than Jeynes and Littell's interpretations in most cases.

As usual, the definition of whole language is at issue. Jeynes and Littell classified two studies as "pure" whole language. They satisfied the following criteria: (1) no adapted texts; (2) no whole class, teacher sponsored assignments, (3) "integrated language experiences as opposed to direct instruction in isolated skill sequences" (p. 23). The two studies in this category (Manning, Manning and and Long, 1989; Morrow, 1992) showed the strongest advantage for whole language. Less "pure" versions of whole language resulted in weaker and negative results.

Jeynes and Littell also classified several "language experience" treatments as whole language, considering language experience to be a "precursor to whole language," p. 27. The core of language experience consists of students dictating stories to teachers; these stories are transcribed by the teacher and used as reading material.

I object to Jeynes and Littell's definition of whole language. (1) In my view, the issue is not whether texts are adapted or modified but whether they are interesting and comprehensible. (2) There are some instances when a whole class teacher sponsored assignment or activity is appropriate in a whole language class. (3) Some phonics knowledge can help make texts comprehensible (Smith, 1994). While most whole language proponents prefer to teach phonics in context, I know of no reason why integrated versus isolated teaching of phonics should be part of the core definition of whole language. The real issue is whether texts are comprehensible.

A similar analysis, limited to published studies, appeared in Krashen (1999), examining studies that compared "whole language" and "skills." This analysis focused only on one characteristic: The amount of real reading for meaning done by the children. As noted earlier, comprehensible and interesting reading is not the only characteristic of whole language, but it is at the core of whole language. The conclusion was that children in classes with more real reading tended to do better on tests of reading comprehension, read more, liked reading more, and did just as well as "skills" students on skills tests (reading nonsense words). This analysis included some of the published studies that Jeynes and Littell included.

I present here comments on those studies in Jeynes and Littell I was able to obtain. As in Krashen (1999), I focus primarily on performance on tests of reading comprehension. Reading comprehension is, after all, the goal of reading instruction.

The impact of whole language/language experience was measured by Jeynes and Littell using effect sizes. Effect sizes are usually calculated by subtracting the mean of the comparison group from the mean of the experimental group, then dividing the result by the pooled standard deviation. They are weighted for sample size (see e.g. Wolf, 1986). Effect sizes can also be computed from other statistics, such as F, t, and r. Following Jeynes and Littell, in this paper, positive effect sizes indicate an advantage for whole language or language experience, and negative effect sizes indicate that comparisons did better.

1. Jeynes and Littell included an unpublished dissertation by J.R. Hoffman, which was not available to me. I did, however, find a study by Carline and Hoffman (1976) with a nearly identical title that was obviously the same study. Although Jeynes and Littell calculated an effect size of -.23 favoring the conventional reading approach over language experience, Carline and Hoffman (1976) concluded that "teachers who use the language experience approach to reading more often ... showed an increase of 2.9 raw score points more on English reading standardized test scores than those teachers who use it less frequently (p < .05)" (p. 43). Carline and Hoffman reported a correlation of .32 between the amount of language experience used and student gains in English reading, equivalent to an effect size (d) of .64.

2. McCanne (1966) compared the impact of a basal reader approach, an audio- lingual approach, and language experience on Spanish speaking children learning English as a second language. McCanne noted that the language experience approach was not designed for students who are acquiring English as a second language (p. 75). Some modifications in language experience were made in this study; nevertheless, the use of language experience clearly required considerable speaking competence before substantial listening and reading took place, a procedure that violates what is known about language acquisition (Krashen, 1985).

McCanne's results depend on the kind of statistical procedure used. When reading test scores were adjusted for factors such as listening ability in English, measures of cognitive development, SES, measures of teacher competence, and pupil attendance, the basal method was better (d = -.65, according to my calculations; based on standard scores presented in McCanne's table 12 (hypothesis six). When raw means were used, the language experience students were better (d = .36). It is not clear to me how Jeynes and Littell arrived at their figure of -.34.

3. Lamb (1972) earned a substantial -.75 effect size in Jeynes and Littell, in favor of the comparison group over whole language. Lamb noted however that all five teachers who used the language experience approach had not used it before and participated in monthly training sessions. The entire duration of the study was only four months.

Contrary to Jeynes and Littell's findings, the results were not clearly in favor of the comparisons. Lamb did several different analyses: Analyses of covariance, controlling for IQ scores, teacher experience, and teacher background showed language experience to be superior (for boys, d = .47; for girls, d = .41) but a one way ANOVA with no control for potential confounding variables found the basal method to be better. I was able to calculate effect sizes based on unadjusted (ANOVA) scores for girls (d = .29) but not for boys. From inspection of the F ratio, it was clear that the effect was less for boys. It was not clear to me how Jeynes and Littell calculated an effect size of -.75 in favor of the basal method. Apparently, Jeynes and Littell used only the simple ANOVA results. Note that in McCanne (1976), language experience did better on with raw scores (ANOVA) and worse on adjusted scores (ANCOVA), the opposite pattern, but in that study, Jeynes and Littell appear to have utilized the ANCOVA result. In my summary below I use the adjusted results.

4. Jeynes and Littell included Ewoldt (1976), a comparison of Follow-Through and Non Follow-Through classes on a story retelling task and reported an effect size of .05 favoring language experience. The Follow-Through Model advocates the language experience approach. A serious problem, however, was that only eight of the Follow Through subjects (out of 36) actually came from classes in which the basic program was language experience (see table 3, page 9). Nine others came from classes that used some language experience as supplemental activities. No information was available about the classes eight of the students participated in, and eight others came from classes that included no language experience. Clearly, this study should not have been included.

5. In Usova and Usova (1993), the number of students in the whole language class was small (n = 8) and the treatment was a combination of a wide variety of activities, emphasizing the combination of art activities with language arts, and also including reading, hearing stories, writing and direct instruction. We are provided with no information whatsoever on the activities of the comparison group, and we have no idea if experimentals and controls differed with respect to the amount of reading done. Neither experimentals nor controls made much progress over the academic year in reading comprehension, experimentals gaining less than one point on a standardized reading test, and controls actually getting slightly worse.

6. Stallings (1975) was awarded a -.79 effect size for comparisons over whole language, but it is difficult to determine how Jeynes and Littell arrived at this figure. This study was not a comparison of methods, but sought to determine predictors of achievement in first and third grade classrooms based on one day of observation. Amount of reading done was not one of the predictors considered.

7. Jeynes and Littell included Harris, Serwer and Gold (1966). I obtained Harris and Serwer (1966a), which appears to cover exactly the same data. Harris and Serwer (1996b) is a shorter version published in the Reading Research Quarterly.

Jeynes and Littell reported an effect size of -.51 in favor of comparisons ("skills-centered") over language experience. From Harris and Serwer's table 26, I computed an effect size of -.18 in favor of comparisons, based on the Stanford Paragraph Meaning test. What is crucial, however, is that Harris and Serwer report that children in the basal group actually spent more time in reading activities than did the children in language experience. Children in the skills classes spent 56% of the instructional time in reading activities, while children in language experience classes spent only 39.5% of the time on reading activities. It is thus quite likely that the comparison children did more real reading. Thus, both the sign and size of the effect size should be changed for this study. Moreover, Harris and Serwer (1966b) reported positive correlations between the amount of time spent in reading activities and scores on the reading tests (r = .47 for paragraph reading).

8. Dahl and Freppon (1994) (also available as Dahl and Freppon, 1995) earned an effect size of .67 in favor of whole language. This figure represents a combination of six different measures of literacy development, including tests of concepts about print, the alphabetic principle, story retelling, and concepts about writing. None were measures of reading comprehension.

A closer look at the results showed that the only significant difference between the whole language and skills children was on a task in which the child pretended to read a picture book without words. The story was rated for the presence of aspects of the written narrative register. I calculated an effect size of 1.79 in favor of whole language for this task.

9. Jeynes and Littell calculated an effect size of .50 for Morrow, O'Connor, and Smith (1990) in favor of whole language. Based only the standardized test used in the study, I calculated an effect size of d = -.18 in favor of the skills group over whole language. The test, however, did not contain a measure of reading comprehension. Literature-based students showed more interest in reading: they could name more authors, took more books home to read, reported more reading at home, and named significantly more kinds of reading material. They were also significantly better on an "attempted reading" test, showing more reading-like behaviors.

10. For Morrow (1992), Jeynes and Littell calculated an effect size of 1.24. For the two reading comprehension tests included in the study, I calculated effect sizes of 1.84 and .62, with a mean of 1.23, nearly identical to Jeynes and Littell's results. It is clear that the whole language ("literature-based") group read more. Children in the literature-based group spent about 3.5 hours per week with basals and about four hours with literature. They were read to daily, engaged in at lease three "literacy activities" per week (e.g. retelling and rewriting stories, book sharing, keeping track of what they read), and had at least three sessions per week in a comfortable "literacy center" for 30 minutes at a time, during which time they read, wrote and performed stories. Comparison students were read to no more than twice a week and focused nearly entirely on the basal and workbook. Free reading was allowed only when children had finished their basal seatwork.

11. Manning, Manning and Long (1989) lasted three years (k to grade 2), and researchers made biweekly visits to classes to "verify the continuity of the two different literacy programs" (p. 5). Both first and second grade whole language students were significantly better in reading comprehension (d = 1.97 in both cases, effect sizes were calculated from p values resulting from Mann-Whitney U's). Nine out of 11 whole language children could name a favorite author at the end of grade 2. None of the skills taught children could.

The small sample size (n = 11 in each group) is an obvious weakness of this study, but the care for fidelity of treatment and long duration are obvious strengths.