Scholarship in the Digital Era

An issue that arises in many tenure and promotion review meetings concerns what should count as credible and substantial contributions in the area of scholarship. Many universities base promotion, tenure and merit pay on three areas: scholarship, teaching and service. Emphasis varies from institution to institution; teaching colleges and universities typically stress quality of teaching over the other two areas; research institutions typically stress scholarship over the other two areas. In all cases, criteria and associated measures are quite important. These remarks focus primarily on the area of scholarship, although there are many important issues to consider in the other two areas.

What is research and what should count as scholarship? Answers vary from program to program and from discipline to discipline. In the performing and fine arts, performances and exhibitions at prominent venues count significantly, as one would expect. In applied programs (e.g., computer and information science, media studies, etc.), the development, evaluation and deployment of innovative programs count significantly. Traditionally, publications in refereed journals and invited presentations at national and international conferences are considered a mark of distinction. In computer and information science programs, ACM and IEEE proceedings are typically considered top-tier publication venues, whereas in educational technology and instructional design research programs, journals such as ETR&D are considered a preferred publication venue.

Issues that frequently arise concern the order of authorship, co-authoring with students, variety of publication venues, whether or not the journal is indexed and refereed, the impact factor of the journal, and so on. In some instances, publishing in online journals arises as a concern. In our discipline, there are reputable indexed, peer-reviewed journals (e.g., Education, Technology and Society).

In addition to online journals, non-traditional publication venues are now available. Examples include blogs, wikis and podcasts. My experience serving on tenure and promotions committees suggests a strong bias for traditional, peer-reviewed publications. Faculty who submit blogs, wikis and podcasts as publications are sometimes viewed as trying to pad their resumes with publications that do not reflect traditional research contributions. Those who write and read blogs and wikis and create and listen to podcasts argue that one often finds legitimate scholarship in those venues that is just as credible as many journal articles. If one accepts that this is sometimes true and likely to increase with time, what is a reasonable approach to recognize new forms of scholarship?
As with traditional forms of scholarship, there ought to be reasonably well-defined criteria and guidance for faculty. I believe that many institutions lack well-defined criteria and guidance for the traditional forms of research and scholarship. New forms of scholarship only make the task of recognizing scholarly contributions more challenging.

What is research? My sense is that research involves a systematic process of investigating a problem and answering questions where existing knowledge falls short or does not exist. Those who conduct research, then, recognize a gap in knowledge – their own knowledge and that of others – and are open to alternative approaches and answers to unresolved questions. This view draws a distinction between advocacy and research, although the boundaries are admittedly fuzzy and advocacy at its best relies on research and evidence. In addition, openness to alternative approaches recognizes the value of peer review. Double blind peer review in journals such as ETR&D is considered a critical part of assessing research significance. When reviewers critique a journal article without knowing the author or the other reviewers, and when authors revise and respond, again not knowing the reviewers, the presumption is that awareness of prior work and alternative approaches and explanations has occurred.

While a blog may allow for comments, anonymity is lost, which means that comments might be biased; in any case, the comments left by readers of blogs are certainly selective. While a journal editor selects the reviewers for a submission, the reviewers are generally obligated to offer a fair and detailed critique. A blog is more akin to comments presented to a class or at a meeting and quite different from a research paper critiqued by peers for a journal. My sense is that blogs should not be considered in the same category as refereed journal publications. This does not imply that blogs should not be counted at all. Rather, for programs that stress new forms of information and communications technologies, blogs, wikis, and podcasts may well count as a form of scholarship. However, blogs are more akin to presentations than to journal articles. Various programs should then decide how much significance to place on presentations and publications and other forms of scholarship (e.g., the creation of innovative instructional programs, learning environments, etc.). If one wants to create a legitimate place for blogs, wikis, and podcasts in the context of scholarly contributions, I believe it does not make sense to argue that they should count just like refereed journal publications, as that tactic is likely to meet serious resistance and create unnecessary disputes.

It does seem reasonable to argue that new technologies are changing the nature of research and scholarship. New ways of conducting research are available. Powerful tools for planning, implementing, disseminating and evaluating research are available. Online journals are becoming increasingly prevalent, and many good papers have been published in online journals. What we should not lose, however, in our rush to integrate technology into research and scholarship is the value of peer review. I suppose this puts me on one side of the debate of what to count as a significant research contribution in this area of networked technologies and social computing. Actually, there are probably many sides to this debate. Regardless of how this debate evolves, I hope that we do not lose sight of the distinction between advocacy and research, that we continue to value peer review and recognize that research essentially involves a willingness to be shown that one is wrong (see Popper’s Conjectures and Refutation), and that we find reasonable ways to encourage faculty and graduate students to engage in a variety of scholarly activities so as to ensure that knowledge is cumulative and relevant. I worry that fascination with new technologies may result in losing sight of the many relevant contributions from prior generations of technology. I worry that fascination with new technologies may result in devaluing the notion of the accumulation of knowledge – this amounts to a devaluation of science and engineering, in my less than humble opinion. I also worry that over-emphasis on traditional forms of scholarship will disincline many bright young scholars from pursuing careers at some of our best research institutions. I worry too much (I have that on the good authority of my better half).