Scientific Inquiry and Educational Research

I had the opportunity to sit in, as a guest lecturer, on a doctoral seminar in the Instructional Systems program at Florida State University in 2007. Prof. Norbert Seel, who led the seminar, asked students to indicate what each regarded as the single most important research question to be addressed in the next five years in the domain of instructional systems, broadly and loosely defined to include analysis, design, development, evaluation, management and technology. Answers reflected topic areas rather than research questions. He then asked each student to indicate an appropriate research methodology to address [part or all of] the indicated question – this proved to be problematic since topic areas rather research questions had been indicated. I was struck by two things. First, the notions of ‘science’ and ‘research’ seemed to vary considerably from one person to another. Second, specific responses indicated a strong tendency to only consider those aspects of instructional systems with which a particular individual was engaged, with the implicit assumption that what each was doing represented the most critical research issue in instructional systems.

What is science? What is the nature of scientific inquiry? What distinguishes scientific research from other forms of research? What do scientists do? There are many answers to such questions. They can be found in books on the philosophy of science and in nearly every introductory text to a particular scientific discipline. I found myself generating such questions during Prof. Seel’s seminar as various doctoral students provided their responses. I settled on a rough and ready representation of inquiry in physics as a starting point. For centuries, physicists have been asking such questions as these: (a) what kinds of things are there in the universe? (b) where did these things come from? and, (c) how do these different kinds of things affect each other? My first thought was that the basic questions within a discipline have remained fairly stable over the years; what have changed are the instruments and tools used to develop answers, which have led to new answers and some variations on the basic questions. Of course research methods and perspectives have also evolved, partly based on new answers to the basic questions. The basic research questions, though, change very little from one generation to the next, in spite of what we might want to believe our how innovative and radical our own approaches are. What changes are the answers to those basic research questions. Moreover, interpretations of the basic questions have changed considerably over the years; new interpretations of the basic questions might be regarded as representing a new approach, or possibly even a paradigm shift.

For example, Empedocles, (a pre-Socratic physicist who lived circa 492-432 BCE) believed that there were only four basic things - earth, air, fire and water – and that the physical world and our experiences could be completely accounted for in terms of these four elements and their interactions. Aristotle further elaborated this view of matter and argued that all earthly substances contained mixtures of these four elements, with the particular distribution of the basic elements determining the nature and appearance of a particular object. For example, a rock contained much more earth than air, fire or water, according to Aristotle, which is presumably why rocks are hard, not readily combustible, and not easily transformed into liquid or gaseous forms. Aristotle then identified four kinds of causes: (a) material cause – the basic composition of an object; (b) formal cause – the inherent or underlying structure of a thing; (c) efficient cause – how the thing came to be in its current state; and (d) final cause – the purpose of an object.

We do not think about the world in the same way as did Empedocles and Aristotle. Physicists no longer accept their account of the physical world. In spite of dramatic advances in physics in the last two thousand years, much has not changed. What has not changed are the basic questions: What kinds of things exist, where did they come from, and how do they interact? Scientists are still attempting to elaborate adequate answers to these basic questions. Modern answers are that there are some 118 or so elements – a few more than four – and these elements are comprised of more basic building blocks – with leptons, quarks, and bosons being the basic categories for these sub-atomic building blocks. Where did these things come from? Big bang … or is that just someone knocking on my office door?

My basic line of thought while listening to various doctoral students was that this framework might be applicable to Prof. Seel’s questions. Someone will tell me not to waste my time because such an account applies only to the hard sciences – the physical sciences, such as astronomy, biology, chemistry, and physics. This person says that the soft sciences (including the social sciences, the learning sciences, and what Herbert Simon called the sciences of the artificial) are different. I understand those distinctions, I think, but there are some common concerns that I believe apply across all the sciences. Basically, what scientists want to know is what exists – the building blocks – and how these things interact to bring about the things we observe, predict or would like to create. While causal interactions might be more difficult to establish in the social sciences, there is still strong interest in understanding, explaining, and predicting critical human interactions, including the development of knowledge and expertise. While the things that social scientists investigate might not be as precisely and narrowly defined as those investigated by physical scientists, there is still strong interest in identifying the basic elements that explain and can predict what we have observed and are likely to observe with regard to human learning and development. Moreover, if we wish instructional design knowledge and practice to be cumulative, it would seem to be in our interest to have a small set of basic questions and ways to explore those questions that others could examine with regard to credibility of methods and veracity of findings.

Perhaps this is a biased or naïve interpretation of science. Nonetheless, I am going to look for the basic elements and their interactions in the domain of instructional systems. What are the basic building blocks of an instructional system? What comes to mind immediately are students, instructors, things to be learned, and instructional resources. This might be an earth-air-fire-and-water kind of answer, though. Each of these elements might be further elaborated in terms of more discrete components which are more informative with regard to explaining interactions that are observed or desired.

What are the essential interactions or causal influences in an instructional or learning context? Outcomes common to most instructional and learning systems include improved understanding and performance with regard to some body of knowledge or set of skills. This implies that there should be reliable ways to assess relative levels of understanding and performance (relative to past performance or understanding or relative to a desired standard or goal). Other outcomes might be identified, and these might be further elaborated in terms of types of outcomes (e.g., affective, cognitive, psycho-motor or … there are many ways to cluster outcomes) and their relationship to knowledge, beliefs, attitudes, habits of minds, mental models, skills, and such.

Regardless of the sophistication and granularity of the components and interactions, we want to understand the various things that comprise an instructional or a learning system and how they are related, especially with regard to efficacy in achieving desired outcomes. Maybe. Well, I seem to recall Gagné saying that our job was to help people learn better. What can we do at a systems level to fulfill that responsibility? How can we measure and foster success?

Lastly, there is the notion of research issues central to progress in a domain. The students who responded to Prof. Seel each had a favorite area or method of inquiry. Why believe that one’s favorite area or method of inquiry is critical to progress in instructional and learning systems research, however? What evidence can one bring to bear to defend such a view? How might one identify critical areas and methods of research inquiry in educational technology?

One might think beyond oneself and beyond one’s own training and set of predispositions. One might look at what distinguished researchers in related areas have said. The Book of Problems (see the 2002 events archive at www.learndev.org) would be a good starting point, I would think. I recall the advice I was given when selecting a dissertation topic by Ed Allaire, my dissertation advisor at the University of Texas in Austin: Pick the central domain and then pick a central unresolved issue within that domain. Allaire’s perspective was that writing a dissertation was a part of an ongoing process of developing knowledge and competence, and, further, that familiarity with central issues was an essential part of that process. Of course there is some subjectivity in this – there will be different views about the centrality of domains and issues. I suspect, however, that a small number of alternatives can be identified. What might these alternatives be for instructional and learning systems?

That is where I thought the discussion might have gone in Prof. Seel’s seminar, or at least that is how it was going in my mind during the seminar. What I have not touched upon yet is a core issue involving values – values that are at the core of scientific inquiry. The starting point of scientific research is a desire to understand a phenomenon or situation or sequence of events. This implies that one admits to a state of relative ignorance: “I do not understand this.” One might say that humility (“I know that I do not know”) is the starting point of every scientific inquiry. What do leading instructional systems researchers admit to not knowing or not understanding? Again I recommend looking at Visser’s Book of Problems (www.learndev.org) to gain insight in to this question. There is too much that I do not know about learning and instruction for me to include the list here. However, one thing that I do not understand well is how particular people develop competence in solving complex, challenging and ill-structured problems. We do learn by doing, so engaging in and reflecting on such problem-solving and decision-making activities seems to be a necessary ingredient in the mix. It also seems woefully insufficient, however. More challenging still, from my perspective, is how we can measure individual progress in developing such skills, which I think is needed in order to develop and implement robust and reliable training programs. We could of course leave such matters to chance and coincidence, but there does seem to be an increasing demand for people who are skilled in solving very complex problems.

A second value to add to humility with regard to scientific investigation concerns being open to alternatives. “This alternative you or I have devised is only one possible approach or explanation; perhaps there are others.” The inability to imagine alternative explanations does not mean that alternative explanations do not exist. Alternative explanations always exist (this is a remark about the logic of scientific explanations). Humility is the starting point, and openness to alternative explanations is required for sustained inquiry.

Finally, one ought to engage in a kind of inquiry that will contribute to progress of human knowledge in and about the domain of inquiry. Scientific inquiry at its best is cumulative and progressive. This requires using reliable instruments and methods that others might also use. If every study is about a unique situation and every instrument used is unique to that study, then generalizing findings becomes a significant challenge – that is one of the many challenges confronting educational research, I think.

Perhaps no one mentioned such things because they are so obvious. I find myself requiring such reminders, though.

Mike Spector
AECT President