Open Peer Review Vol. 13 N. 2 | Teknokultura. Journal of Digital Culture and Social Movements

Quantitative Metrics for Generative Justice: Graphing the Value of Diversity

Brian Robert Callahan, Charles Hathaway, Mukkai Krishnamoorthy

Reviewer A:

Reviewer did not want comments to be published

Recommendation: Revisions required

Reviewer B:

Please, comment on the most relevant aspects (positive points and areas to improve) of the reviewed article.

This is a very unusual piece. I really like the way the article assesses Generative Justice, and offers OSS as a kind of ideal or model example of Genrative Justice. The careful approach to assessing whether “diversity” somehow means justice is quite cogent; I also appreciated the attempt to offer both advocates of diversity in OSS, and the OSS community at large, a way to assess the value of contributions by a range of diverse members.
Critique: I believe this is a qualitative analysis disguised as a quantitative one, or perhaps better, I think this is a preface to an article that actually attempts the quantitative analysis dicussed. Applying the analysis to actual data, even a small simple, would make this an interesting piece. In addition, taking on the “problem space” of OSS is itself a complex undertaking. There are many types of groups within the huge OSS umbrella term, and again, attempting to apply the analysis to a specific OSS community would perhaps address this problem.
I’ve attached a brief edit of the article, and I also have an edit by a content expert in the OSS world that I am happy to share.

Would you suggest any changes or make any recommendations to improve the quality of the article?

As I’ve indicated, I think the article needs to do what it proposes: apply this notion of diversity and the Power Law graphing to an actual group within OSS. Or absent that, I think the article could outline the qualitative elements of the core argument, which connects generative justice aims with a presumably desired aim of valuable contributions from a community diverse in gender, race, and so on. Is the argument that the latter creates, in and of itself, better software? Or that the value which the diverse creators of the software bring lies in a larger overlap with end users? I think this latter argument is implicit in the piece, but it might be teased out more.

Recommendation: Resubmit for review

Reviewer C:

Please, comment on the most relevant aspects (positive points and areas to improve) of the reviewed article.

Overall, I think is a very suggestive and interesting article, which develops a series of new and original contributions to the study of crucial and current phenomena (FLOSS communities and their work) developed on well-defined goals. Based on the idea of Generative Justice and a case study where it is observed or applied this approach aims to overcome the (very common to date) qualitative approaches by the calculation of a quantitative indicator that accounts for these processes and some similar (diversity , innovation , etc.) . The article proposes a quantitative index in particular ("Anonymous Complexity") and addresses how it should be measured and what results are expected. It really is a hypothetical or tentative proposal on how a quantitative measure of the type of this kind of communities we deal with should be performed.

All this presents some problems however. The Generative Justice, as such, does not appear well explained for my taste and is only defined in economic terms ("unalienated value through networks") or referred to previous works. I find problematic a model of justice based on a Marxist economic definition (labor theory of value) or it may be that I do not understand. In fact, the article starts as if the concept was completely taken and understandable (the first sentence of all the text is "How can we measure Generative Justice?"). Thus, the main concept is announced, it is prepared, is mentioned with a link to a wiki, but there isn’t a very concrete and detailed development of what it means in legal or moral terms and what are their differences with other regulatory models. It is not clear either whether relations with the idea of Generative Grammar (Chomsky) or anything like that or why that term is used in particular when there is a lot of previous terminology to refer to the online and open source communities. It would be necessary to introduce better this paradigm to justify the need for a quantitative metric and not just for the fact that so far all studies are qualitative (which is the justification given). The existence of some previous quantitative studies is also mentioned but not explicitly mention which.

Afterwards it is mentioned that the paradigmatic example of this type of Generative Justice are Free Software communities but we finde only that example or case which restricts a lot the understanding and evaluation of the given proposal. Finally, the indicator presented, which is also not well defined ("hypothetical measurement") and aims to measure diversity (and innovation) within a community does not respond to a specific formula but is displayed in a few graphics (number of participants who reached a certain level of contribution) and a power law but there is no other way of putting such measurement more generally. Nor is it clear why the "anonymized Complexity" must necessarily measure diversity (¿only because it is defined so?) and why that diversity is an indicator of effectiveness, organic composition or degree of innovation of the community. The author/s only mentions that there are numerous "utilitarian" studies (correlations) between diversity and innovation, that's all (and a study of diversity and connectivity in marine ecosystems). Nor do I understand how a measure of justice is done through the diversity of a community but it is suggested to be used as well to measure the degree of openness (openness). I think the author needs to define what is meant by justice, for diversity, etc. in these cases or contexts because I have the feeling that they are not consensual or shared assessments. Overall I feel that all over the text the arisen ideas are postulated or taken for granted and not sure if there is a lack of justification or that I am personally missing data.

Would you suggest any changes or make any recommendations to improve the quality of the article?

First, assuming the Generative Justice (GJ) is not a well-known term or consensus concept in academic field (something that does not take away legitimacy or interest), I do recommend a brief more concrete introduction about what brings and what it is meant by Generative Justice in several examples. And, especially, why it is necessary to coin a new term and what are their relations with other models of justice (Rawls, for example). Given that there are several theories of justice, it would be desirable to understand what this new concept brings in relation to the above (iusnaturalism, utilitarian, liberal, Marxist, feminist, anti-colonial) and why, for example, the ideas of Rawls, Habermas or Bobbio do not apply to the present time or examples. On the other hand, the examples presented have already been analyzed and quite theorized and previous terminologies have already been developed for such communities. Why should we stand for a new approach? For example, the work of Lin Yue who has written extensively on these issues. I put a couple of examples:
Lin, Y. (2006). Hybrid innovation: The dynamics of collaboration between the FLOSS community and corporations. Knowledge, Technology & Policy, 18(4), 86-100.
Lin, Y. (2007). Hacker culture and the FLOSS innovation. Handbook of research on open source software: technological, economic, and social perspectives, 34-46.
And some similar works: Belussi, F., & Staber, U. H. (Eds.). (2012). Managing networks of creativity(Vol. 19). Routledge.
Second, the author should try to find a series of varied examples to display the characteristics of GJ more precisely since contrast and comparison help to understand it much better. Only through a unique example perhaps you cannot grasp the whole complexity of the concept. It would be desirable to propose different types or models of communities or expressions of GJ and try to show that there are different behaviors and therefore it makes sense to make measurements to levels of development, success, composition or evolution thereof. Here are several different types mentioned that you could try: http://www.hass.rpi.edu/pl/hass-events/?objectID=100003855

Finally, the proposal for a quantitative indicator requires a somewhat more elaborate and accurate operationalization. It is not absolutely necessary a mathematical formula but some kind of methodological development that can be applied to different cases and examples. It is not clear that such a concept as "anonymous Complexity" has to be measured by the idea of "diversity" and that only can be observed empirically by the number of contributions that reach a certain level (What level? In all communities the same level? What kinds of contributions are taken into account?), etc. And if the indicator is just that (amount of contributions / level of contributions), these are measures that have already been applied once previously and the author would have to explain to what extent it makes sense to call it "anonymous complexity" and why not take into account other factors (composition of users, types of contributions, communicative modes of expression, etc.).

Recommendation: Revisions required.

Reviewer D:

Please, comment on the most relevant aspects (positive points and areas to improve) of the reviewed article.

The the most positive aspects of the article is the attempt to quantify phenomena that have mainly be qualitatively described. Although potentially promising, the authors need to do more to describe the benefits of a quantitative metric - and, indeed, do more to quantify their proposed metric.

Would you suggest any changes or make any recommendations to improve the quality of the article?

1) The article is unclear and/or contradictory when it comes to spelling out its contribution:

   a) The introduction suggests that the metric can be used to measure the diversity of a population. However, the “diversity make-up” presented later on is not defined or explained. Indeed, diversity is added to a power law distribution, which itself is not really a measure of diversity so much but describes differences in the distribution of contributions (at least for the case of OSS). The solution to this is to clearly spell out the logical connections (as of now unstated) connecting diversity and distribution of participation to an overall notion of generativity. It would be straightforward to present something like a hypothetical mathematical model: e.g., generativity = diversity * distribution function * something else? Furthermore, given the importance of inequities in the distribution of expertise to the proposed metric, I would also expect some references to the literature about the harms of expertise being overly specialized or inaccessible to lay persons (see Frank Fischer and others) and other relevant issues.

   b) Much is suggested about the limits of qualitative approaches and the benefits of quantitative metrics but the only real support for that claim comes at the top of what I believe is page 6 (please add page numbers!): quantitative measures may be more persuasive. What exactly was “previously unavailable” and now rendered visible by the quantitative metric? To this point, the earlier critique of Coleman’s relative neglect of diversity issues (though fine in one respect) only demonstrates that earlier qualitative researchers have not paid enough attention to diversity in OSS, not that qualitative research methods are insufficient for the job. Hence, the authors need to add a much stronger argument regarding the relative merits of quantitative metrics. Is it that it enables “finer” distinctions or is it that it illustrates something difficult to capture in words? Here (as is true elsewhere) a brief search of the literature would probably make the authors’ job easier. I’m sure other smart people have debated the relative merits of qualitative and quantitative metrics. Reading and citing those will free the authors up from having to reinvent the wheel.

   c) Are the authors really providing a “metric?” The paper does not seem to suggest that the output of this kind of analysis would be a number (or even a set of numbers). Rather, as it appears in this version of the text, they are offering a visual representation of diversity/distribution of participation. This could be easily fixed by suggesting possible diversity metrics (perhaps the authors could draw up one of the many diversity indices already developed and in use in ecology, or perhaps sociologists have something ready-made). Certainly some kind of metric could be devised to combine the different parameters of the power law distribution with a diversity index to come up with a diversity/participation function. Furthermore, another small addition that could help in this regard would be to discuss the different parameters of the power law distribution function: p(x)=A*e^(n*x)+k, and relate each part (A,n,k) to participation. Doing the above would help sell this as a metric, rather than simply a visual representation, unless the latter is what you mean to do. If it is the latter, I would recommend changing the title.

2) There are number of places in the text that are currently confusing for the reader.

   a) What exactly the analogy between binary and source-code and sheet music and sound does for the argument in the first section is unclear. Sure musicians read sheet music, but is it not the case that only the computer (and not the programmer) that reads binary? Musicians may only share the sounds (and even then only sometimes freely) but not the sheet music, but the analogy implied (that coders share the binary but not the source code) does not immediately make sense to me. Perhaps the discussion of source-code and binary is not really necessary for the reader to follow the argument about sharing? The analogy between music and code might be clearer without it. In any case, I would give this section some revision with an eye toward improving clarity.

   b) On page 2, what do the author’s mean by “the value created by the software artisan never leaves her hands?” What is “value” in this case? Ownership? Something else?

   c) Why cite a wikispaces page for the definition of generative justice when there are already published works out there that define it (e.g., Eglash and Eglash & Garvey)?

   d) The topic sentence of the 2nd paragraph on page 3 doesn’t fit with the following material. The authors state that OSS is “open in a true sense” but the rest of the paragraph deals with how difficulties in obtaining (or inequities?) in expertise may render an ostensibly “open” system effectively closed. Here and elsewhere, double check topic sentence alignment with paragraph contents.

   e) Usage of “we” is often confusing and sometimes borderline dishonest. It is fine to use “we” when it is the authors speaking to the reader. Elsewhere it is vague: “we can share music” and “we need to be able to identify software.” Who is “we?” All of humanity? STS scholars? OSS enthusiasts? In many of these cases, the “we” can be dispensed with by rephrasing: e.g., “Being able to identify software matters because…”

   f) Other phrases, like “realities of our understanding” or “doing tech” are similarly unclear and/or awkward.

   g) “Diversity make-up” and “power category” need defined and explained in “anonymized complexity” section.

   h) On the next page, the authors refer to having someone “hired…to sit in a chair and fill a quota.” This comes out of left field and can be too easily read as a dismissive remark about affirmative action policies. I’m not sure if the author’s want to be so blasé. This paragraph needs some reworking so as to make the critique seem more thoughtful and circumspect. Moreover, I suspect there is a contradiction between the authors’ own perspective and the connection to shallow and deep recursion. The authors’ seem more concerned with a vague notion of “encouraging innovation” while Eglash and Banks seem more concerned with the degree to which people get to participate in substantial ways. There might be a connection between the two but, as it stands right now, the discrepancy between the authors’ ostensibly “product-centric” view and the “people-centric” view guiding generative justice detracts from the article.

   i) When discussing figures, alert the reader to them (e.g., “See figure 1”).

   j) Typically one avoids contractions like “don’t” and “doesn’t” in academic writing.

   k) Reference to ranking work by how “technical” it is needs some justification. How would someone do that in anything but an arbitrary way? Again, perhaps someone else has already done this in the literature.

3) Finally, the authors end with a lamentation about the “difficulty of measuring the social dynamics of the OSS community” that comes off as fatalistic. It does not help sell the value of the author’s contribution regarding the measuring of generative justice in OSS communities by seeming to suggest that it is pie-in-the-sky in the final paragraphs. It would be far better to examine what the authors’ see as the primary barriers to better data gathering in OSS endeavors. What would it take to overcome these barriers? What currently provides the incentives and disincentives that stabilize the status quo? Is there any possible way to realign those incentives? The authors’ need not have a fully fleshed out answer, but at least considering these questions would do double duty: End the paper on a less fatalistic note and suggest more avenues for future research (something that needs looked at in its own right).