The Case: The Association between Salivary Cortisol and Mental Health in TRAILS

TRAILS is a multicenter study designed with the aim of finding the origins of mental health problems. Dysregulation of the hypothalamus-pituitary-adrenal axis was an interesting candidate, and several TRAILS papers were published on the relationship between cortisol and various aspects of psychopathology. However, it appeared that it was not possible for TRAILS to make a comprehensible synthesis regarding the potential role of cortisol in the etiology of psychopathology. Concerned by this observation, we analyzed the strategies used by the consortium to answer the questions on cortisol and psychopathology, and found that, although the strategies employed within the papers were usually correct, there were inconsistencies across papers. These inconsistencies concerned the operationalization of psychopathology (different questionnaires, informants, cutoff levels), the cortisol variables (different composite measures), and the use of statistical methods and included confounders. The end result was a rather confusing pattern of findings. For instance, self-reported oppositional defiant problems were cross-sectionally associated with high morning cortisol levels in girls, and parent-reported disruptive (i.e., oppositional defiant plus conduct) problems predicted high evening cortisol levels in boys (for details see Table 1). In general, the results could not be combined in an overarching model, and were thus disappointing with regard to scientific progress. In contrast, the end result in terms of publication output was quite positive: the majority of papers were presented at international conferences and published in highly cited journals (Table 1), and several students earned PhD degrees based on their work on the subject.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Publications on associations between (basal) cortisol measures and psychopathology in TRAILS.

https://doi.org/10.1371/journal.pmed.1001143.t001

We were interested in the processes that enabled the publication of incoherent papers. To identify these processes, we made an overview of all relevant studies on cortisol and psychopathology and discussed this at a meeting of TRAILS researchers, which yielded several potential causes of the problem. The identified causes were analyzed, and the resulting report was sent to all authors of the cortisol articles listed in Table 1, with the request to indicate which processes they experienced, which processes they did not experience, and whether the report missed any processes they thought were important. The authors' comments were incorporated into our analysis reported here.

The identified causes were analyzed using Hackman's normative model of group effectiveness [3],[4] as a general framework. This model is a theoretical framework incorporating social psychological knowledge derived from descriptive research on group performance, with the aim to guide improvements. Although we realize that there are other ways of looking at social interactions that might also apply here, two characteristics of Hackman's model make it particularly interesting for our case study. First, Hackman uses a multidimensional definition of group effectiveness, which includes not only task but also group outcomes. A highly effective group in this model is thus characterized by a tight balance between an appropriate amount of work-related criticism and good social relations. Second, Hackman suggested that the model can provide a basis for diagnosing the strengths and weaknesses of groups, and can thus be used as action guide in the world of practice [3]. Hackman hypothesized that group effectiveness is determined by an interaction between organizational context and group characteristics (Figure 1). While Young et al.'s paper discussed the organizational context, we will analyze these group characteristics that may influence effectiveness, in particular the design and synergy of the group.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Figure 1. Hackman's normative model of group effectiveness.

https://doi.org/10.1371/journal.pmed.1001143.g001

Group Design

Three important aspects of group design can be distinguished in Hackman's model: (1) the structure of the task, (2) the composition of the group, and (3) group norms about performance processes.

Group Synergy

Group synergy arises if group processes result in gains in energy and effectiveness that go beyond what would be expected given the organizational context and the task design. In groups with high synergy, members think of themselves as group members first and as individuals second. In groups that produce scientific papers within the context of large longitudinal cohort studies, group synergy is not easy to obtain. Most PhD students will leave the group after they obtain their PhD, so their primary goal is individual: obtaining a PhD. Senior investigators also tend to have individual goals, because in the end their own performance is being evaluated, not the group performance, and they all have their own responsibilities towards their faculties and grant funders. In short, the strong focus on individual achievements in science hampers group synergy, particularly in multicenter collaborations. On the other hand, for many of the current scientific questions, research consortia are necessary because of the amount of money, participants, or expertise needed. The fundamental problem is the way science manages cooperation versus competition.

The Task

Although current scientific practices sometimes seem to suggest otherwise, the overarching task of a research group is not to write papers, but to address scientifically relevant issues. This often involves multiple papers. To avoid that the writing of a particular paper starts to take on a life of its own, it is important that the overarching task also be considered a meaningful piece of work, and that its organization be appointed to a steering committee with clearly defined goals, rights, and responsibilities, which are agreed upon by all participating investigators beforehand. Explicit, and preferably written, general agreement on the overall strategy will increase the likelihood that individual writing teams act in accordance with the overarching goals. In addition, regular general research meetings can be organized to discuss specific papers in relation to the overall strategy. This strategy will increase not only the conceptual and methodological consistency within consortia, but also the shared commitment and possibilities to learn from each other.

The Writing Teams

For individual articles, the appropriate number of authors is, in general, not too many. Coauthors are the ones who design, analyze, and write. Other, more general contributions to studies such as grants, project management, delivering research subjects or materials, and laboratory work, which are often rewarded with honorary authorships, should be valued as important scientific achievements in another way, in order to take away the impetus to grant coauthorship to contributors who do not actually participate in the writing process. Writing a scientific article together requires a high level of task interdependence among team members. If individuals' work becomes intertwined with that of others, however, it is more difficult for them to determine a sense of personal achievement, and they will be unlikely to make extraordinary efforts unless they view their individual task as meaningful [5]. This has clear implications for the organization of teams of coauthors: it is important to start the writing process by clarifying the unique role, responsibility, and contribution of each coauthor, and how this role will be filled. An advantage of explicitly recognizing members' expertise and role within the group is that group members are thus mutually aware of each other's areas of expertise from the beginning onwards, which has been shown to increase use of members' expertise and information exchange [6].

Optimal teams consist of coauthors whose skills are, to a large extent, complementary rather than overlapping. Whereas heterogeneity of coauthors, providing it is within reasonable bounds, is likely to improve performance, too much overlap in skills will promote social loafing, i.e., the tendency to reduce individual effort when working in teams compared to when working alone [5],[7]. Non-overlapping skills will also increase the task visibility of the coauthors, i.e., individuals' belief that their efforts are seen as important by others [5]. All ideas should be evaluated thoroughly, regardless of the status and seniority of the author. Finally, team members should be encouraged to teach and learn from each other, and so increase the quality of the writing team, as well as of the research group as a whole: development of knowledge, skills, and talent is a scientific obligation.

The Methodological Strategy

Researchers who participate in large longitudinal studies should be aware of the necessity of building explicitly on prior work performed on the same dataset. In practice, this implies using similar methods, unless there are strong theoretical reasons to do otherwise (statistical significance is not among these reasons), which should be clearly outlined in the papers involved. Importantly, this means that the overall strategy should be agreed upon before the first paper is published. Post hoc or irrational choices about the inclusion of covariates can be prevented by choosing the covariates to be included before starting the analyses, e.g., based on causal graphs, which provide an excellent guideline for the selection of potential confounders (e.g., [8]).

We acknowledge that science is intrinsically unpredictable. It will be impossible to account for all possibilities in advance, since part of the scientific process involves coming up with new ways to measure phenomena. Nevertheless, even if no single best strategy can be specified in advance, it is possible to build group norms that increase the likelihood that members will develop task-appropriate performance strategies and execute them well. For instance, consortia could agree upon a number of standard methodological papers and evaluate every publication proposal and paper against the recommended methodological guidelines. It is essential that every consortium member explicitly agree with these standard papers, in order to ensure that the performance strategies become part of the fabric of the group. Once a strategy is agreed upon, group members tend to behave in accordance with it and enforce adherence to it. Thus, establishing group norms is important because they can foster self-regulation and reflection, and thereby bring science to a higher level.

A more rigid way to avoid methodological divergence could be to have all analyses performed by an independent statistician. Apart from the fact that this is not always financially feasible, an evident downside of this system is that PhD students would not be trained to do their own analyses, which we feel is an important aspect of their education as independent researchers. We do recommend, however, discussing the analytic plan and having all papers reviewed before submission by an independent statistician. Additionally, for reviewers evaluating a paper, it would be helpful if cohort studies generating multiple papers had a unique code in journal literature search systems (e.g., PubMed), so that it would be easy to find all papers published previously on a certain cohort to check for methodological consistency.