My colleagues Bill Savage, Dick Wheeler, and I recently published an opinion article in the journal Frontiers in Research Metrics and Analytics (https://doi.org/10.3389/frma.2022.812312). This paper is meant to accompany the expansive dataset on publishing patterns across disciplines that we recently made public (https://osf.io/myaut/). In short, this article is meant to highlight that academe’s growing use of bibliometrics to evaluate academic units must be done judiciously, and even the best-intended bibliometric indicators do not always consider myriad factors specific to the discipline in question, as well as the individual who authored those publications.

Equally important to our thesis (that bibliometrics, when used, should complement expert review and should be modulated to account for discipline and career stage) is that scholars in many disciplines do not produce quantifiable “bibliometric artifacts” that are archived in databases. Assigning a DOI or other identifier to a particular class of work is uncommon in some disciplines, while it is common in others. Conference proceedings, for example, are often archived and catalogued in engineering and computer science fields, but this is not as often the case in the humanities and other fields. Across-discipline comparisons must consider the archiving traditions of different fields before meaningful comparisons can be made.

There are also modes of knowledge production that are not amenable to archiving. “Some works are ‘small circulation ephemera’ (e.g., zines as described by Brown et al., 2021), and archiving and cataloging these works may be antithetical to the ethos in which they were created.” As the pace of scholarly publication (at least in STEM areas and the Social Sciences) continues to accelerate along with the evaluation culture in which these publications are a de facto currency, it may be best to resist the temptation to include some types of research in bibliometric archives. Rather than expanding the quantification of scholarly outputs to include non-journal article modes of dissemination, we might encourage generative dialogue across disciplines about different knowledge generation and dissemination processes. Academe has an opportunity to move from “are we including all the relevant journals?” to questions that enhance interdisciplinary communication, such as “what knowledge dissemination types are pertinent to these disciplines?” and “do the types of knowledge production common in this discipline become quantifiable artifacts?”

 

Anthony J. Olejniczak, Ph.D.

Director, Academic Analytics Research Center

 

References

Brown, A., Hurley, M., Perry, S., and Roche, J. (2021). Zines as reflective evaluation within interdisciplinary learning programmes. Front. Educ. 6:675329. doi: 10.3389/feduc.2021.675329

Olejniczak AJ, Savage WE and Wheeler R (2022) The Rhythms of Scholarly Publication: Suggestions to Enhance Bibliometric Comparisons Across Disciplines. Front. Res. Metr. Anal. 7:812312. doi: 10.3389/frma.2022.812312