Abriendo la caja negra de la investigación académica: evaluación, publicación y perspectivas críticas

What does it mean today to take a critical stance on scientific knowledge, its production and its dissemination? And what tactics should one adopt for this purpose? These questions are relevant to all disciplines; but for STS they are crucial, because the processes by which knowledge is produced and institutionalized are among the main themes of the entire STS field of study. This paper tries to answer the above questions by outlining the main features of the scenario in which the production and assessment of scientific knowledge today take place. It then concentrates on a concrete case in the field of open-access scientific publishing in order to show some of the tactics useful for pursuing a critical perspective on both the production and dissemination of knowledge at academic level.


Introduction
What does it mean today to take a critical stance on scientific knowledge, its production and its dissemination?And what tactics should one adopt for this purpose?
These questions are relevant to all disciplines, but for STS they are crucial, because the processes by which knowledge is produced and institutionalized are among the main themes of the entire STS field of study.
Opening up the black-box of academic research and scientific publishing, this paper tries to answer the above questions.It does so first by outlining the main features of the scenario in which the production and assessment of scientific knowledge today take place.It then concentrates on a concrete case in the field of open-access scientific publishing to show some of the tactics useful for pursuing a critique on both the production and dissemination of knowledge at academic level.
The concept of 'black-box' is well known in STS (Latour, 1987), as it indicates the closure of a project, forgetting all the ambiguities and conflicts that took place during its construction and making it appear coherent and 'already done'.In our case, we will imply it in processual terms, as a 'sensitizing concept' (Blumer, 1954).To quote Latour (1999: 304), blackboxing is "the way scientific and technical work is made invisible by its own success.(…) Thus, paradoxically, the more science and technology succeed, the more opaque and obscure they become.".In what follows, to paraphrase Latour, we will see that the more academic research is assessed, the more scientific publishing becomes black-boxed.In our view, a critical perspective is needed in order to bring light to the scientific and political assumptions embedded within this scenario.

The scenario: assessment, merit, competition
Leó Szilárd was one of the physicists who worked on construction of the atomic bomb.He was one of the few scientists in that project to oppose the dropping of the atomic bombs on Hiroshima and Nagasaki.After the war, he became a convinced pacifist and an advocate of the international control of atomic energy.Thereafter, he abandoned research in physics and devoted himself, among other things, to writing short stories.In one of these stories, a billionaire asks a researcher how the progress of science can be slowed down.This was the answer: You could create an institution with annual funding of forty million dollars.Researchers who need capital could turn to this institution, if they present convincing projects.Appoint ten committees, each composed of twelve scientists, with a mandate to examine these projects.Take the most active laboratory scientists and make them members of these committees.[...] Firstly, in this way the best scientists would be removed from their laboratories and engaged in the work of these committees responsible for the allocation funding.Secondly, scientific researchers in need of capital would focus on issues considered promising and likely to lead se-curely to publishable results.For some years, there would be a sharp increase in scientific production; but at a guess, this could be exactly the right system for withering science.[...] It would be a matter of fashion.Those who follow the fashion obtain funding.Those who do not follow it, no; and you will see that they, too, will soon learn to follow the fashion.
The passage just quoted is used ironically by Francesco Sylos Labini 3 (2016: 239) to draw attention to the darker side of the current system of financing scientific research, assessment, and career advancement in academia.This system is increasingly based on three keywords: 'assessment', 'merit', and 'competition'.Although entirely acceptable in abstract (who would not agree that research should be assessed and that the most meritorious researchers should be rewarded through a process of competition among peers?), the way in which the meaning of each of these notions is declined creates problems.
Let us begin with assessment, which is also essential for understanding how merit and competition are defined.The philosopher of science Donald Gillies (2008), has conducted a detailed critique of the Research Assessment Exercise (RAE -the evaluation system introduced in the UK in 1986 by the Thatcher government).Gillies shows how this system has had considerable effects in terms of both the allocation of funds and influencing the decisions of researchers (as well as entire departments and universities).Since there are no 'objective' criteria with which to judge whether, over the past three decades, the RAE has improved or worsened the quality of research in the UK, Gillies imagines the effects of implementing the RAE criteria in some cases of scientific excellence.For example, if Wittgenstein had been evaluated according to the RAE parameters, he would probably have been judged unproductive, and a system of evaluation like 'publish or perish' would hardly have had a positive influence on his intellectual development 4 .The same applies to Frege (who lived in the second half of the nineteenth century and is considered the father of modern mathematical logic), whose works were considered of little interest by his contemporaries, even though they are the basis of modern computer systems.More recently, Peter Higgs (the physicist who gave his name to a boson) does not believe that a university would employ him in today's academic system 5 .After his pioneering work of 1964 (in which he identified the mechanism by which subatomic material acquires mass), Higgs published fewer than ten papers, which in the age of 'publish or perish' would likely lead to his dismissal as 'unproductive'.
The main problem with the evaluation of research today, therefore, is that it is bound to a purely quantitative logic of 'publishing prestige' which considers how and where a paper has been published, but not its content.As Sylos Labini (2016) _____________ 3 Francesco Sylos Labini is an Italian astrophysicist, founder and editor of the Return on Academic Research website (www.roars.it).In recent years, as a result of radical changes made to the Italian academic system, this website has become the main forum for critical discussion of policies on higher education and research in Italy and abroad.points out, this is clearly reflected in the frantic race to publish as many articles as possible in the 'best' journals 6 , the insatiable hunger for citations, the addiction to bibliometric parameters.The publish or perish philosophy has thus generated an exorbitant increase in scientific articles and their citations.The databases which collect them have proliferated, with a worrying increase in the misuse of data on the quantity of articles and their citations (Baccini, 2010).Not coincidentally, in the recently issued Leiden Manifesto (Hicks et al., 2015) concerns are expressed about the misuse of bibliometric parameters precisely by those who have done most to popularize the use of indicators and to make bibliometric a global business 7 .Many doubts have been raised about the quality of bibliometric data themselves, and especially regarding commercial databases: it is impossible to verify the provenance of the raw data and validate their reliability (Rossner, Van Epps and Hill, 2007).
A second issue concerning assessment is the manner in which it is implemented, i.e. through peer review.A researcher writes a scientific article, sends it to a journal, and the journal's editor chooses anonymous reviewers who decide whether or not the article is publishable.This does not mean that what is written in the article is 'true'; rather, that it is plausible in its reasoning and argument (Ioannidis, 2015).It is obviously much easier to convince the reviewers of the appropriateness of one's arguments by adhering to the more traditional and widely accepted canons of research, rather than propounding radically innovative assumptions, interpretations or methodologies.A famous historical example may again help to evidence the fallacies of this system (Gillies, 2008;Sylos Labini, 2016).
The reference is to Albert Einstein, who fell out with his doctoral thesis supervisor and was unable to obtain a doctorate either at the Polytechnic or the University of Zurich (he acquired his doctorate a few years later, in 1905).Being unable to obtain a post in academia, Einstein found a job at the federal patent office in Bern.He devoted his spare time to writing articles on theoretical physics, which he managed to publish in the leading German physics journal of the time, Annalen der Physik 8 .Einstein had no academic affiliation, and in presenting the theory of special relativity he did not cite any article or book on the subject by his contemporaries.What is the likelihood today that an article with no bibliographic references and submitted by a researcher without a doctorate and university affiliation would successfully pass the peer review process 9 ?_____________ 6 The fact that one journal is considered better than another is often the outcome of further standardization work, i.e. the indexation of journals and calculation of their impact factors.

7
For example, the number of scientific papers in the largest database, Scopus, rose from around one million per year in 1996 to three million per year in 2013.In recent years, the academic publishing industry has grown to a similar extent, and today there are thirty thousand journals, some of which fuel a situation that risks to resemble more like the Wild West than academic publishing (Bohannon, 2013;Sylos Labini, 2016). 8 This was possible because Einstein had already published five articles in Annalen der Physik, whose editorial policy was that only the initial publications of an author were to be examined by the editor or another member of the editorial board (subsequent articles by the same author could be published without further assessment).9 The biologist Randy Schekman, for example, just before receiving the Nobel prize in physiology, announced that his laboratory would no longer send research articles to journals such as Nature, Cell or Science in order to break their tyranny on scientific progress (Sylos Labini, 2016).This is not to claim that the peer review process is useless (indeed, we regard it as important); rather, it is to acknowledge the limitations and risks of peer review, given that there are no 'neutral' evaluation models.What we want to emphasise are the fallacies associated with the assessment of merit based solely on quantitative standards and parameters, and oriented to selecting as much as possible, in order to reward what is presumed to be excellence.As noted by Gillies (2014), there is a fundamental misunderstanding in regard to competition: positive competition in science is that among research programmes, not among individual researchers.What until around thirty years ago was a purely intellectual competition among different theories and research projects has changed into a struggle for scarce resources among individual researchers (Sylos Labini, 2016).Strong competition for resources among individual researchers has primarily negative effects on themselves, on the quality of their work, and on their relationships -effects that are rarely considered.According to some scholars, competition fosters the adoption of unethical behaviour, a drop in the sharing of information and methods, a sabotaging of the ability of others to use one's work, interference with the evaluation process, the deformation of interpersonal relations, and conduct of research in a controversial manner (Anderson et al., 2007).
As pointed out by Sylos Labini (2016), the mixture of peer review, a shortage of resources, and concentration of the available funding on a few groups (when not individual star-researchers), is generating the 'Matthew effect' described by Merton and well known to those interested in STS: evaluation processes tend to reward 'excellent researchers'; the more researchers are excellent, the more they have access to resources; the more resources they have, the more they can devise research programmes fulfilling the criteria defined by the evaluation parameters.In other words, a perfect system to reward past excellence, but not that of the future.

What kind of alternatives for 'critical' scholars and journals?
Before going on with the critical attitudes that can be enacted in scientific publishing, it is worth it to make explicit, even briefly, some theoretical remarks on the meaning of "critical" about scientific production.First, our adoption of the adjective "critical" in the realm of scientific journals relies on a Foucauldian perspective, affirming that knowledge production is not a neutral activity, but on the contrary a work that actively contributes in producing the realities it claims to just report.This stance is also a core part of the STS perspective, which believes that knowledge production is performative, and that practices and methods of scientific production enact our reality at the same time they describe it, as it has been highlighted by John Law (2008) reflecting on method.From this standpoint, looking at scientific publishing in a "critical way" implies a reflexive attitude about our own scientific practices.Setting up and running a journal, thus, is not just a way to share ideas; it is a performative action that can alter the scientific context we inhabit and the kind of science we produce.Thus, reflecting critically on scientific and publishing practices results to be inherently political, because it puts under scrutiny the relationship between scientific production, the role of science in society and the rules and methods scientists adopt in their work.
How does the scenario presented thus far implicate and affect the work of scholars involved in producing and building scholarly journals?In recent years, a small but intense debate has grown about the need of a critical perspective in scientific publishing, especially among those journals and scholars that reclaim some kind of critical standpoint within the contemporary scientific landscape.For example, a few years ago the editors of the journal Organization questioned in an editorial article what it means to assume a critical standpoint as a journal in a situation characterised by increasing commodification of scientific journals, knowledge and work, largely dominated by global private corporations (Parker and Thomas, 2011).Martin Parker further developed his reflection on the role of scholars who contribute with their work to this system based on a dominant configuration of academic publishing.This publishing system "actually has some very negative consequences for taxpayers, academics and students" because it "encourages the overproduction of academic output" turning scientific outcomes "into a commodity which is traded, whilst simultaneously tending to discourage forms of knowledge production that fail to fit into the boxes which have already been established for them, whether in terms of content or style" (Parker 2013, p. 461).
However, the reflections by Parker go beyond description of the consequences of this publishing system on the scientific environment.It also addresses the role of those scholars who contribute to the maintenance of such a configuration, i.e. all of us who are also part of the boards of journals owned by global academic publishers.Hence, if "extraordinarily high profits of the big publishers are dependent upon a double appropriation that exploits both academic labour and universities' financial resources" (Beverungen et al. 2012) and "the publishers of journals are engaging in hyper-profitable activities which extract value from university and state budgets", then the editors of these journals should recognize that they are like marionettes, sorts of puppets manipulated by the same publishers that we criticize: "the editors of these journals are helping them to do it, even if they can't see the strings."(Parker 2013, p. 471).
Even though this may seem somewhat dramatized, this situation is more than realistic and therefore we should be increasingly pressed to question which alternatives exist for critical scholars, especially if they are involved in science & technology studies, in which it is expected that analysis of the production of scientific knowledge is part of the bigger picture.
Beverungen, Böhm and Land (2012) outlined and addressed four alternatives available to critical scholars to create alternatives to this system: the strengthening of existing open-access repositories; a fair trade model of publishing regulation; the revitalization of university presses; and the construction of alternative models of publishing, which should move away from private, for-profit publishing companies and toward independent and autonomous journals, owned and maintained by editorial boards or academic societies.It is this fourth model -self-organized open publishing -that Tecnoscienza embraced in 2010, when it started its journey in an uncertain and fast-changing landscape of academic publishing.

Tecnoscienza: a tool for critical reflexivity on academic publishing
Today there are several tools with which to make self-organized publishing more simple and viable: for instance, the work of the Public Knowledge Project (PKP), which provides the Open Journal Systems platform, a free software developed specifically to make open-access publishing more efficient and that has been adopted by Tecnoscienza as well as by thousands of new journals around the world.This model, based on self-organized open publishing, represents the newest and in some sense most radical alternative to traditional publishing because it not only relocates control of the scientific publishing system to those who should have it -the scholars on the editorial boards -but it also reshapes the possibility of access by both authorship and readership.
The debate on what shape this model should assume is ongoing in several areas.This alternative and more radical kind of publishing has been recently defined by Fuchs and Sandoval as the "Diamond Open Access Model": a model in which "not-for-profit, non-commercial organizations, associations or networks publish material that is made available online in digital format, is free of charge for readers and authors and does not allow commercial and for-profit re-use."(2013, p. 438).Several definitions and attempts to develop these alternative models are emerging.They include the decision taken by the board of Tecnoscienza, which in 2010 invested in developing an independent online peer-reviewed journal, released under the Creative Commons license and free for both authors and readers.As we also stated in the introduction to the very first issue of the journal (AA.VV., 2010), Tecnoscienza can be conceived as a useful tool with which to perform our critical and reflexive view on our work in STS and on the transformations of the current scientific publishing system.Publication of Tecnoscienza came about as the result of the commitment and energy of an emerging generation of researchers, who expanded the scope of their interests by drawing on different areas and research fields.From the outset, the subject areas on which Tecnoscienza focuses have involved both 'classic' STS topics (such as laboratory studies and public communication of science) and more cross-sectional ones (such as post-feminist debates, cultural studies, design and media studies).In fact, we are interested in expanding connections and intersections with areas most affected by innovations and transformations: economy, organizations, design, art, and everyday life.The aim of Tecnoscienza has been to carry out this work by pursuing two parallel goals.
On the one hand, the journal has wanted to contribute to the already-existing and today flourishing STS debate by drawing a transversal line across the existing categories and boundaries, by questioning fields, objects and methods involving a heterogeneous set of knowledge, disciplines and topics.The purpose of establishing a new STS journal, more than just topics and contents, has been to reflect on the evolving geography of STS at global level.With regard to the journal's intellectual policy, we can say that it especially targets the wider process whereby the cultural geography of STS is reconfigured.Since the STS landscape arose in specific countries (the UK, France, Netherlands, the USA), it has been characterised by the growth of newer, increasingly international and globally interconnected networks, journals, and research.Today, the presence of STS scholars has expanded in many different countries around the world.In this scenario, one of the aims of Tecnoscienza is to relocate the geography of the global STS community by giving resonance to the importance of the local embeddedness of STS perspectives.Thus, not only is Tecnoscienza an attempt to draw attention to a relatively new, 'indigenous' Italian STS community; it also more generally supports a revaluation of the role of smaller national communities and alternative perspectives in the STS domain.This policy is centred on new appreciation of the multiple, locally embedded, alternative STS perspectives, as well as the local trajectories of researchers, communities and countries where STS have in the meantime developed.
On the other hand, another target of the journal has more directly related to affirmation of a political and critical view on the publishing industry and, more in general, on the dominant scientific model that we have outlined and criticized in the first part of this article.Firstly, Tecnoscienza has been conceived for distribution in the way most dynamic and accessible today: as an online open-access publication with no fees for authors or for readers.Thanks to a non-profit platform like Open Journal System, we have been able to work without the support of a traditional publisher.We have managed everything by ourselves and attended to all aspects of the publication process.This independent practice and open access policy distinguish Tecnoscienza from most of the current STS journals, providing an example of alternative scientific publishing practices away from the existing oligopoly of international academic publishers.The journal's specific features include governance and decision-making about publishing options and choices -what a proper publisher usually does -as well as the more mundane and basic work: management of the peer-review process, copyediting, maintenance of the web platform, and promotion.This is of course hard work, often with little visibility, and only partially rewarded in academic terms, but nevertheless crucial for the journal's independence and autonomy.For all these reasons, Tecnoscienza is an attempt to develop and advance a reflexive discourse on where to go today with scientific publishing and a distinctive way to conduct science & technology studies by relying on a distinctive 'sociomaterial configuration': a specific configuration that arranges a specific model of (self)governance and a horizontal division of labour, a set of technological devices and platforms, and political stances regarding the current landscape of scientific production and distribution.
To return to our original question -what does it mean today to have a critical attitude toward scientific knowledge and which strategies do we have to undertake?-the experience of Tecnoscienza together with that of similar self-published journals help to envision the path to follow: toward a scientific environment where control over the crucial infrastructure of scientific publishing starts to return, al least in part, to the scientists and researchers themselves.

4 From
1930 until the end of his academic career (in 1947) Wittgenstein published nothing.He spent his time editing the notes that then became the Philosophische Untersuchungen (Philosophical Investigations), published in 1953 and considered among the foremost expressions of twentieth-century philosophy.5 Peter Higgs, "I Wouldn't Be Productive Enough for Today's Academic System", The Guardian, 2013, December 6.