How easy is it to reproduce the results found in a typical computational biology paper? Either through experience or intuition the reader will already know that the answer is with difficulty or not at all. In this paper we attempt to quantify this difficulty by reproducing a previously published paper for different classes of users (ranging from users with little expertise to domain experts) and suggest ways in which the situation might be improved. Quantification is achieved by estimating the time required to reproduce each of the steps in the method described in the original paper and make them part of an explicit workflow that reproduces the original results. Reproducing the method took several months of effort, and required using new versions and new software that posed challenges to reconstructing and validating the results. The quantification leads to “reproducibility maps” that reveal that novice researchers would only be able to reproduce a few of the steps in the method, and that only expert researchers with advance knowledge of the domain would be able to reproduce the method in its entirety. The workflow itself is published as an online resource together with supporting software and data. The paper concludes with a brief discussion of the complexities of requiring reproducibility in terms of cost versus benefit, and a desiderata with our observations and guidelines for improving reproducibility. This has implications not only in reproducing the work of others from published papers, but reproducing work from one’s own laboratory.
Cognitive science has long shown interest in expertise, in part because prediction and control of expert development would have immense practical value. Most studies in this area investigate expertise by comparing experts with novices. The reliance on contrastive samples in studies of human expertise only yields deep insight into development where differences are important throughout skill acquisition. This reliance may be pernicious where the predictive importance of variables is not constant across levels of expertise. Before the development of sophisticated machine learning tools for data mining larger samples, and indeed, before such samples were available, it was difficult to test the implicit assumption of static variable importance in expertise development. To investigate if this reliance may have imposed critical restrictions on the understanding of complex skill development, we adopted an alternative method, the online acquisition of telemetry data from a common daily activity for many: video gaming. Using measures of cognitive-motor, attentional, and perceptual processing extracted from game data from 3360 Real-Time Strategy players at 7 different levels of expertise, we identified 12 variables relevant to expertise. We show that the static variable importance assumption is false - the predictive importance of these variables shifted as the levels of expertise increased - and, at least in our dataset, that a contrastive approach would have been misleading. The finding that variable importance is not static across levels of expertise suggests that large, diverse datasets of sustained cognitive-motor performance are crucial for an understanding of expertise in real-world contexts. We also identify plausible cognitive markers of expertise.
There is vigorous debate about the reproducibility of research findings in cancer biology. Whether scientists can accurately assess which experiments will reproduce original findings is important to determining the pace at which science self-corrects. We collected forecasts from basic and preclinical cancer researchers on the first 6 replication studies conducted by the Reproducibility Project: Cancer Biology (RP:CB) to assess the accuracy of expert judgments on specific replication outcomes. On average, researchers forecasted a 75% probability of replicating the statistical significance and a 50% probability of replicating the effect size, yet none of these studies successfully replicated on either criterion (for the 5 studies with results reported). Accuracy was related to expertise: experts with higher h-indices were more accurate, whereas experts with more topic-specific expertise were less accurate. Our findings suggest that experts, especially those with specialized knowledge, were overconfident about the RP:CB replicating individual experiments within published reports; researcher optimism likely reflects a combination of overestimating the validity of original studies and underestimating the difficulties of repeating their methodologies.
The Clinician’s Guide to Prevention and Treatment of Osteoporosis was developed by an expert committee of the National Osteoporosis Foundation (NOF) in collaboration with a multispecialty council of medical experts in the field of bone health convened by NOF. Readers are urged to consult current prescribing information on any drug, device, or procedure discussed in this publication.
Researchers have shown that people often miss the occurrence of an unexpected yet salient event if they are engaged in a different task, a phenomenon known as inattentional blindness. However, demonstrations of inattentional blindness have typically involved naive observers engaged in an unfamiliar task. What about expert searchers who have spent years honing their ability to detect small abnormalities in specific types of images? We asked 24 radiologists to perform a familiar lung-nodule detection task. A gorilla, 48 times the size of the average nodule, was inserted in the last case that was presented. Eighty-three percent of the radiologists did not see the gorilla. Eye tracking revealed that the majority of those who missed the gorilla looked directly at its location. Thus, even expert searchers, operating in their domain of expertise, are vulnerable to inattentional blindness.
Reading scientific articles is a valuable and major part of the activity of scientists. Yet, with the upsurge of currently available articles and the increasing specialization of scientists, it becomes difficult to identify, let alone read, important papers covering topics not directly related to one’s own specific field of research, or that are older than a few years. Our objective was to propose a list of seminal papers deemed to be of major importance in ecology, thus providing a general ‘must-read’ list for any new ecologist, regardless of particular topic or expertise. We generated a list of 544 papers proposed by 147 ecology experts (journal editorial members) and subsequently ranked via random-sample voting by 368 of 665 contacted ecology experts, covering 6 article types, 6 approaches and 17 fields. Most of the recommended papers were not published in the highest-ranking journals, nor did they have the highest number of mean annual citations. The articles proposed through the collective recommendation of several hundred experienced researchers probably do not represent an ‘ultimate’, invariant list, but they certainly contain many high-quality articles that are undoubtedly worth reading-regardless of the specific field of interest in ecology-to foster the understanding, knowledge and inspiration of early-career scientists.
People in Western cultures are poor at naming smells and flavors. However, for wine and coffee experts, describing smells and flavors is part of their daily routine. So are experts better than lay people at conveying smells and flavors in language? If smells and flavors are more easily linguistically expressed by experts, or more “codable”, then experts should be better than novices at describing smells and flavors. If experts are indeed better, we can also ask how general this advantage is: do experts show higher codability only for smells and flavors they are expert in (i.e., wine experts for wine and coffee experts for coffee) or is their linguistic dexterity more general? To address these questions, wine experts, coffee experts, and novices were asked to describe the smell and flavor of wines, coffees, everyday odors, and basic tastes. The resulting descriptions were compared on a number of measures. We found expertise endows a modest advantage in smell and flavor naming. Wine experts showed more consistency in how they described wine smells and flavors than coffee experts, and novices; but coffee experts were not more consistent for coffee descriptions. Neither expert group was any more accurate at identifying everyday smells or tastes. Interestingly, both wine and coffee experts tended to use more source-based terms (e.g., vanilla) in descriptions of their own area of expertise whereas novices tended to use more evaluative terms (e.g., nice). However, the overall linguistic strategies for both groups were en par. To conclude, experts only have a limited, domain-specific advantage when communicating about smells and flavors. The ability to communicate about smells and flavors is a matter not only of perceptual training, but specific linguistic training too.
Recent investigations into the neural basis of elite sporting performance have focused on whether cortical activity might characterize individual differences in ability. However, very little is understood about how changes in brain structure might contribute to individual differences in expert motor control. We compared the behavior and brain structure of healthy controls with a group of karate black belts, an expert group who are able to perform rapid, complex movements that require years of training. Using 3D motion tracking, we investigated whether the ability to control ballistic arm movements was associated with differences in white matter microstructure. We found that karate experts are better able than novices to coordinate the timing of inter-segmental joint velocities. Diffusion tensor imaging revealed significant differences between the groups in the microstructure of white matter in the superior cerebellar peduncles (SCPs) and primary motor cortex-brain regions that are critical to the voluntary control of movement. Motor coordination, the amount of experience, and the age at which training began were all associated with individual differences in white matter integrity in the cerebellum within the karate groups. These findings suggest a role for the white matter pathways of the SCPs in motor expertise.
to explore the views of expert midwives in Ireland and New Zealand of the skills they employ in expectant management of the third stage of labour (EMTSL).
BACKGROUND: When comparing a single-stroke dissection maneuver among surgeons with differing experience levels, there are major differences in the force applied to the instrument tip. It is difficult to explain to surgeons in training the appropriate force and for the surgeons to ascertain the force intuitively. We quantified the force pattern during single-stroke laparoscopic dissection maneuvers to reveal the factors related to expertise. METHODS: We recorded the force pattern of a single maneuver and measured the magnitude of vertical (VF) and horizontal forces (HF) on the instrument tip using a box trainer (ex vivo). We compared VF and HF among surgeons: experts (n = 10), intermediates (n = 10), and novices (n = 10). The dissection time of a single stroke (T), magnitude of the VF and HF, and the timing of the peak vertical force (TPV) and horizontal force (TPH) were evaluated as performance parameters. RESULTS: The dissection time of a single stroke (T) was shortest in the expert group (p < 0.05). The average maximum magnitude of VF and HF was smallest in the expert group. TPV occurred significantly earlier than TPH in all three groups (p < 0.05). TPV in the expert group occurred earlier than in the intermediate and novice groups (p < 0.05). With increasing experience, TPV occurred earlier. CONCLUSIONS: Expert surgeons apply the most efficient vertical forces to make an initial dissection point and then change to the horizontal direction to separate surrounding tissues from the target organ. Measuring instrument tip force could help in understanding and improving the safety margin in laparoscopic surgical dissection.