Scientific Design Sketch

This month’s topic came about from a suggestion for a session on power analysis and experimental design. As NABES members’ work covers both experimental and observational studies, the session was broadened to capture more of the challenges faced within the network, both technical and otherwise, e.g. in communication and collaboration.

What challenges might we encounter when designing studies?

Hayley Carr (Babraham Institute) led the first part of this session, sharing her perspective as an early career statistician with a more biological background. To prompt discussion, some aspects of study design and experimental design that she identifies as challenges include:

  • Dealing with unknowns – approaching situations where there are only poor estimates of variability
  • More complex designs / experiments – e.g. determining optimal sample sizes for high throughput omics experiments, balancing competing resource constraints, numbers of technical (single sample, multiple measurements) versus biological replicates (multiple distinct samples)
  • Communication – asking the right questions when designing studies for someone, how to convey the more statistical aspects and their importance, matching the statistics to biological expectations, encouraging engagement at the design stage rather than after data collection Additionally, later in a career, it might be challenging to know when / how to expand beyond familiar methods and approaches.
    With these points in mind, sharing examples of approaches and designs, useful tools or resources, and common pitfalls others in the network may have come across is welcome.

General good practice for designing studies

Having designed experimental trials and surveys over much of his career, at varying spatial scales and levels of complexity (e.g. with multiple trials or multiple experimental phases), Andrew Mead (Rothamsted Research, andrew.mead@rothamsted.ac.uk) offered some examples of good practice for approaching study design:

  • Ask naïve questions if needed to clearly identify research questions, hypotheses, resources and constraints
  • Find out what prior information exists about variability and effect sizes, whether from previous trials or literature
  • Standard designs won’t always be appropriate so avoid using a recipe book approach
  • Modern analytical tools let us combine strange treatment structures and arrangements of experimental units, so we can be more imaginative and address the research questions using the natural structure of the resources
  • Borrow approaches from other application areas
  • Control what you can and adjust post-hoc for what you can’t

Small group discussions covered challenges that others in the network face when designing studies, and approaches which might be helpful. For example:

  • Spending time in the lab / field can provide more understanding of the biology and help with communication
  • Benchmarking experiments can help with understanding the observation process
  • Try to keep hold of metadata that might provide information on factors that weren’t controlled in the design

How can NABES help?

We also talked about how we could use the network to help us develop skills in this area. Some of the points raised include:

  • Skills mapping and knowledge sharing to identify relevant expertise within the network and know who we might contact
  • Consultancy skills training
  • Guidance documents on e.g. what to think about rather than prescriptive actions
  • Hackathons to practice skills