Source: Excerpts from A. Harry Passow,
Harold J. Noah, Max A. Eckstein, John
R. Mallea, The National Case Study: An
Empirical Comparative Study of Twenty-One
Educational Systems. New York: John
Wiley and Sons, and Stockholm: Almquist and
Wiksell, 1976. International Studies in
Evaluation VII: 12-14, 299-295.
Reprinted by permission of John Wiley and Sons.
NATIONAL CASE STUDY REPORT
[From Chapter 1: Context and Content of the National Case Study Report]
The IEA Six Subject Study and Comparative Studies in Education
IEA achievement studies constitute the largest multinational and international study of the cognitive outcomes of schooling ever attempted. The present series of IEA publications, of which this volume represents but one small part, surveys and analyzes school achievement and its correlates in 21 countries and in six school subjects. IEA work thus constitutes a massive contribution not only to the literature of pedagogy, in general, but also to international and comparative studies in education.
Comparative studies in education have been developed out of a variety of motives. Simply the need to know what is happening in the educational systems of other lands was the earliest, and remains one of the strongest, supports. This has been reinforced by the intense interest of policy makers in foreign examples, in the hope that there might exist abroad some useful lessons to improve domestic practice. Thirdly, comparative studies of education held the promise of enriching both theoretical and empirical understanding of the processes taking place within schools, and the interconnections between schools and society. In response, comparative education has developed in a number of characteristic ways. First, there has been heavy concentration on the collection of data: what is going on abroad? and how best might the data be organized to demonstrate the similarities and differences among countries? Second, there has been great emphasis on the cross-national study of "problems" in education and between school and society, in the hope that policy makers wrestling with intractable problems of, say, Church-State relations, Teacher Training Curriculum, School Budgets, and so forth, might have their decisions informed by an understanding of the variety of conditions and attempted solutions abroad. Third, comparative educatin has proceeded in the direction of testing hypotheses about school processes and their relation to societal phenomena against the evidence drawn from several countries. On occasion, such research hypotheses have been derived from single-nation (case) studies; alternatively, the propositions tested have emerged only "after the fact," following extensive, sometimes even global, efforts at data collection and analysis.
IEA studies represent progress along all these dimensions. A vast amount of descriptive data has been amassed to describe what is in fact going on, but this was done with some well-defined aims in mind. Attention was focused upon the task of measuring levels of school achievement cross-nationally, and the related problem of investigating variations in levels of school achievement, particularly in those countries and among those students in all countries where achievement levels were unusually low. Moreover, from its very inception, IEA work was intended to provide the opportunity of testing the validity of a number of clearly enunciated hypotheses, using data from many countries. Thus, the format of IEA work is a series of replicated studies, for which special instruments were devised, appropriate to all the cases (countries) studied, to collect data on school achievement and on as many variables as were thought to be useful for explaining achievement.
The National Case Study and the IEA Six Subject Study
IEA studies are all concerned with identifying and explaining sources of differences in school achievement. The six subject studies approach this problem from the perspective of students and schools within each country. They are less concerned with the general characteristics of the nations -- their societies and school systems -- than with analysis of differences in the characteristics of the students tested, the schools they attend and the teachers. On occasion, the subject studies refer to national factors, such as economic level, school system structure, selectivity and the like, but these factors do not play a central role in the subject study analyses.
The National Case Study (NCS), on the other hand, puts the structure of the nations' school systems and the leading dimensions of their social-cultural-economic systems at the very center of the analysis. Thus, we may characterize the approach of the subject studies as "micro," while the NCS uses a "macro" approach, employing the country rather than the student or school, as the unit of analysis.
In this way, the two sets of studies constitute complementary, strategies for answering the basic questions: What factors are associated with what differences in school achievement, and to what extent?
Countries differ in important respects with regard to their culture, socio-economic systems, and their educational systems. It is reasonable to suppose that at least some of these features will affect the way schools operate, the way teachers teach, and the way students learn. In other words, there is some basis for supposing that some factors of a society and an educational system can be used to predict and/or explain particular features of a country's pattern of educational achievement.
More than this, there has now passed into the realm of "conventional wisdom" a host of preconceptions about the relationship between societal and educational structure on the one hand and educational achievement on the other. For example, comprehensive schools are said to "lower standards;" centralized educational administration is believed to raise educational achievement across a country; student achievement is supposed to suffer when the social status of teachers is low, and to gain from a high innovative capacity of the school system. Such is the conventional wisdom: the job of the National Case Study inquiry was to try to find out if it is indeed supported by the facts.
[From Chapter 4: Conclusion]
Review and Summary of the Findings
....The National Case Study data do extend knowledge about the socio-cultural, economic, and pedagogical context of the participating IEA countries but not sufficiently to provide firm generalizations about organizing and implementing an educational system. What we are left with insofar as the policy-planner is concerned, are a number of hints and possible implications which need to be examined on both a within-country and a cross-national basis. The correlations of the indicators with achievement, it must be repeated, deal with achievement in Science and Reading Comprehension. An educational system has broader goals than achievement in these subjects, goals of both a cognitive and affective nature, goals dealing with individuals and the society. These were not considered in the NCS analyses, and the policy-maker must perforce consider these other goals, and make decisions on a wider basis than that of measure school achievement alone.
There are four distinct kinds of implications to be drawn from the NCS data. Some
findings suggest that the policy-makers might take "positive action" if increased
achievement in Science and Reading Comprehension is desired. Some findings suggest
that certain aspects of the so-called "conventional wisdom" are not borne out
cross-nationally, and policy-makers would do well seriously to query such wisdom. Some
findings suggest that policy makers should seek to make qualitative rather than quantitative changes and recognize that "more" alone may have little bearing on the question of "better" or "worse." And, finally, some findings suggest that even though there is a strong relationship between achievement and specific variables, other values should lead policy makers to look to different means of affecting achievement. For instance, a strong inverse relationship between achievement and economic development would not imply that nations reduce efforts to raise the GNP but rather they should study ways of improving the quality of school life, using an increasing level of affluence.
Implications for Further Research From its inception, the National Case Study gave promise of fewer implications for educational policy, than did the six subject studies. These focus much more than does the NCS on within-school variables, which are more susceptible to manipulation by school administrators and policy makers. The NCS deals typically with broad system characteristics of countries that are both more difficult to change and which reveal change only after quite long passages of time. The above discussion on policy implications has emphasized that the best that the NCS exercise could be expected to show is a number of rather gross associations, factors to be considered when attempting to reach conclusions about decisions on school policy. Commonly held assumptions might be questioned and previously neglected factors might receive more attention. Proof of definite associations and assertions of causal connections, leading directly to choice among policy options are not to be expected.
In the IEA study of Mathematics achievement, an effort was made to describe the several national educational and social contexts that should be taken into account, when considering the implications of the results. The NCS attempted to build on this example, and to go beyond it by looking at a wider range of country system characteristics; by including factors that are not easy to quantify and scale, as well as those that are; and by seeking time-series as well as cross-sectional data. It will have been evident that in all these respects a great deal of improvement is possible. Future research should try to cover far more comprehensively than was possible in the NCS such matters as differences by country in home environments, the political context of schooling, the connection between jobs and educational credentials, and the style of student-teacher "transactions" inside and outside the classroom-to name only a few.
Collection of time-series data on many variables proved to be very difficult, exceeding the statistical resources of most countries. Yet a good model of the factors influencing school achievement must be cast in terms of past inputs and processes. Insofar as social and educational systems change very slowly it is possible to use cross-sectional data as proxies for conditions in the past, and the NCS (and the other IEA studies) have relied upon the validity of this assumption. However, it should be recognized that it is an assumption, and research should continue to expand efforts to collect and use longitudinal data in explanatory models of cross-section achievement differences.
In international research in education, as in other social studies, cultural bias can be a serious pitfall. The multi-country collaborative nature of the IEA enterprise went far to eliminate such bias, and the instruments developed to measure student achievement appear to be, so far as is possible, neutral, or, if not neutral, at least not systematically biased with respect to national systems of education. Yet the very selection of school achievement as the criterion variable may be culturally biased, for it is a concept generated within and commanding particular attention in developed, Western-type societies. Even though achievement may include non-cognitive dimensions, such as associated skills and attitudes, even though it may be a value thoroughly acceptable to the educators of forward looking, but as yet less-developed countries, the IEA studies, and the National Case Study in particular, assume that school achievement is considered to be equally desirable (vis-*-vis other school and social objectives) among all people and in all countries. However, it may be that some people, countries or cultures simply value other outcomes of schooling (or of the use of young people's time and other national resources) more than they do achievement. And, while achievement appears to be a Western-generated criterion, it may not be an equally prized objective of schooling even among developed countries.
Although the subject committees gave varying amounts of attention to non-cognitive achievement in their respective areas (civic attitudes, literary tastes, science attitudes, for example, were particularly important and potentially illuminating), the NCS did not exploit the possibilities generated by these results. Future research should expand the analysis to investigate relationships between countries' system characteristics and such non-cognitive outcomes of schooling. Moreover, it is important to estimate the extent to which countries differ in the priority they accord to school achievement, compared to other outcomes of schooling.
For the most part, the statistical models employed in the subject studies explained only quite small fractions of the total observed variance in scores, between students and between schools. The reader will have noted also that the analyses employed in the NCS discovered few close fits between social, cultural, political and educational system characteristics and country achievement differences. There are a number of possible reasons for these, on the whole disappointing, results, reasons that apply as much to the explanatory models used in the subject studies as to the NCS approach.
First, measures on both the input and output sides, may contain large amounts of error. Error in the measurement of variables severely limits the explanatory power of statistical models, and in this respect the NCS probably suffered more than did the subject studies. Thus, the plight of the NCS Committee might be compared with that of a marksman condemned to shoot at a poorly defined target with rifle and bullets deficient in a number of unknown respects. Future work should give higher priority to improving the quality of measures, particularly on the input (independent variable) side of the equations.
Secondly, low explanatory power can arise from misspecification of the model. There are various aspects of this problem. There may have been omission of important variables. However, in view of the large number of variables used in both the subject studies and in the NCS, this is unlikely to have happened. A more damaging misspecification may have been the implicit assumption in the NCS study that, in general, the several explanatory variables can be used as if they are associated with achievement in a straightforward, additive and essentially non-interactive manner. If this assumption is incorrect, if in fact differences in the patterning of inputs are important for explaining differences in achievement, then the low values of correlation coefficients reported are not unexpected. Future NCS-type work, then, should try to assemble patterns of countries' system characteristics, and relate these to differences in school outcomes. If this is to be done, however, it will require a larger number of nations than was available from the Six Subject Survey, as well as the use of more sophisticated statistical techniques.
The third aspect of possible misspecification resides in the possibility that the significance (for outcomes) of a particular factor may not be its quantity, or even its role in some overall pattern, but its timing. Two countries' systems may supply equal amounts of several resources, but in one country the timing of these inputs may suit children's growth and achievement more than in the other. An analogy from agriculture is perhaps apposite. Two farmers may labor an equal number of hours on very similar fields and with similar equipment, seeds, fertilizer, and so on. If one farmer times his operations consistently better than the other, the size of their harvests can be expected to differ sharply. The NCS did not probe such questions in the context of school achievement, and future research might well try to do so.
The most interesting, and perhaps the most useful, approach to cross-national research
proceeds not in terms of existing country-wide units, but on the basis of sub-national
units. This means that it may be more interesting (for comparative work) to inquire
about the correlates of achievement within, say, metropolitan areas across several
countries, or among the children of the poor, or among girls, each group taken together
across nations, than it is to regard individual countries as the logical, or only,
units of analysis. This is not to deny, of course, that policy makers and researchers
within each country will place first priority upon knowing the contributions made by
such factors as metropolitan location, parental poverty, or sex to achievement in their
own country. But that is by no means the end of the story, and comparisons across
countries are inevitable. Here, the technique to be used is not simply a comparison of
system-wide derived regression coefficients (although that can be valuable, especially
if care has been taken to standardize definitions, measurement and scaling of variables
across countries), but rather a cross-country pooling of data, partitioned by such
factors as location, parental characteristics, teacher characteristics, levels of
school finance, and so forth. This would enable the researchers to answer the
question: How far, across all country units, among girls, are certain characteristics
of homes, students and schools associated with achievement; how, if at all, do these
associations differ among boys? Cognate questions could be asked about the production
of achievement in metropolitan areas compared with rural areas, in poorly financed
schools compared with well financed ones, and so on.
The NCS Committee did attempt a pilot investigation along these lines, but problems connected with the way schools were sampled defeated the attempt. It remains as a most fruitful line for crossnational research.
A final point concerning the value of partitioning samples for regression analysis
bears on the utility of the results to policy makers. Because, in general, IEA
regression analyses have taken the entire country sample as the universe to be
explained, policy makers may find the results much less useful than they might wish.
The results of linear multiple regression analyses done on aggregated country samples
conceal a great deal of the information that they need. The reason for this becomes
clear if one reflects on the nature of the information that a partial regression
coefficient conveys. It tells the reader the average value of the strength of the
association between the dependent variable and a particular independent variable, all
other independent variables being held constant at their average values. Such an
explanatory approach is able to clarify the connections between achievement and other
factors, and to assign relative weights to the importance of one factor compared with
others, on average, over the entire country. Policy makers, however, usually
require something more detailed than such average indications. They need to know the
effect upon achievement of varying a particular item under rather specific
circumstances. Policy considerations are directed not at influencing achievement
levels in "average," country-wide settings, but typically they deal with the problems
of achievement in, say, rural vis-à-vis urban settings; of girls vis-à-vis boys; and of students in poor neighborhoods as distinct from wealthier neighborhoods. What is the incremental achievement value of increasing expenditures for schools in, say, poor neighborhoods? In wealthier neighborhoods? What provides the largest increments to achievement for low achievers? For average achievers? For high achievers? Thus, the policy makers' questions require analyses that do not hold the values of the other independent variables constant at their average country level, but which partition the total sample, in such a way that the associations between a dependent variable (say, achievement) and a particular independent variable (say, current expenditure per student) can be investigated separately for specified groups -- e.g., poor children and rich children; urban center children and rural children; poor urban center children and rich urban center children.
In a sense, then, the NCS study can be regarded as an inevitably weak substitute for within-country regression analyses done on partitioned samples. Comparing, say, India, Iran and Thailand with the United States, England and Germany does underline the contrast between the way home, school and student variables are associated with achievement among poorer people, as compared with more affluent people. But there are severe problems in relying upon aggregate cross-national analysis to fulfill this role: the number of dimensions for partitioning countries soon exhausts their number, so that cell sizes become very small, or even zero; the results are always subject to challenge on the ground that national idiosyncrasies are at the root of all observed results; it is obviously not expedient to have to deal with all the difficulties involved in cross-national measurement and scaling of variables, if within-country research can answer some of the important policy questions as well, or better; and, finally, it is often more difficult to see how results obtained from cross-national research can be applied to the problems of a given country.
The role of an NCS-type study in the future, then, becomes one of trying to use national system characteristics to assess and explain problems and paradoxes that remain after as much variance as possible has been explained using partitioned national samples and partitioned pooled data.
This report has, it is hoped, reconfirmed the potential of crossnational studies of schooling, based upon broad cultural, societal and educational measures. However, the caveats attached to findings are numerous, and the study is, perhaps, best regarded as an interim report. The goal of a coherent, reliable, persuasive explanatory model of school achievement differences is still far distant; and many problems of measurement, scaling, and comparability of variables remain intractable. The present state of the art in empirical comparative educational analysis leaves much to be desired. For we are all as yet in the position of one of W.S. Gilbert's comic opera criminals, endlessly expiating his misconduct, serving as a cautionary example to others:
And there he plays extravagant matches
In fitless finger stalls,
On a cloth untrue
With a twisted cue
And elliptical billiard balls.