Essential 10:
| ITEM | RECOMMENDATION | IMPC’s Approach | |
|---|---|---|---|
| Study Design | 1 | For each experiment, give brief details of the study design including: a. The groups being compared, including control groups. If no control group has been used, the rationale should be stated. b. The experimental unit (e.g. a single animal, litter, or cage of animals). | See Item 4 |
| Sample Size | 2 | a. Specify the exact number of experimental units allocated to each group, and the total number in each experiment. Also indicate the total number of animals used. b. Explain how the sample size was decided. Provide details of any a priori sample size calculation, if done. | As a high throughput project, the sample size is relatively low with a target number of knockout animals being processed of 14 (7 per sex) for the standard Early Adult phenotyping. The assessment of the optimal number of animals for the Late Adult phenotyping is still ongoing . This number was arrived at after a community wide debate that involved statisticians, biologists and project managers to find the lowest number that would consume the least amount of resources while achieving the goal of detecting phenotype abnormalities in a strain[1]. At times, practical issues might limit the number of animals it is possible to test, such as viability issues or the difficulty in administering a test. As such, each time data are shown, the number of animals phenotyped per sex per genotype is listed with the graphical visualisation of the data. In a high throughput environment, replication of individual lines is not cost effective. Instead, multiple IMPC centres generated and characterised the same six reference knockout lines that present a wide range of phenotypes based on previously published research. |
| Inclusion and exclusion criteria | 3 | a. Describe any criteria established a priori for including and excluding animals (or experimental units) during the experiment, and data points during the analysis. b. For each experimental group, report any animals, experimental units or data points not included in the analysis and explain why. c. For each analysis, report the exact value of N in each experimental group. | During experimentation individual animals or mutant lines can be held back from a particular test or withdrawn completely from phenotyping due to welfare concerns according to compliance with each institutes’ local regulations. If an animal undergoes a procedure, but the measurements cannot be reliably acquired, the data for that procedure or particular parameters can be excluded. During data quality control, performed at the Data Coordination Centre (DCC), all the data undergoes a centralised rigorous QC process with specifically implemented QC web interface which allows visualisation and tracking of issues. Potential issues are identified and reported back to the phenotyping centres where the data is confirmed as acceptable or removed. Measurements which cannot be obtained or fail QC either at procedure or parameter level require specified codes to indicate why data is not available. N numbers are reported for experimental groups (see Item 2). |
| Randomisation | 4 | Describe the methods used: a. To allocate experimental units to control and treatment groups. If randomisation was used, provide the method of randomisation. b. To minimise potential confounding factors such as the order of treatments and measurements, or animal/cage location. | As a high throughput project that is looking to generate a hypothesis about gene function for all genes, the IMPC consortium follows a general design of having seven knockouts per sex (for the early adult pipeline) that are compared to the control data, producing the following groups: wildtype males, wildtype females, knockout males and knockout females. In some cases (e.g. embryonic lethality for homozyogtes), heterozyogote mice are phenotyped and compared to wild-type mice. Within the analysis we consider the mouse as the experimental unit. Unlike most experiments, we cannot randomly allocate animals to experiment groups; rather we are relying on Mendelian inheritance to provide a natural randomisation of the data. However, there are still many other aspects of the experiment where planning is necessary to avoid bias (e.g. order effects). Whilst the general approach and procedures are captured and tightly defined in IMPReSS, we identified that implementation (e.g. blinding) could vary significantly from institute to institute depending on local resources and priorities. As an international consortium, discussions on study design and implementation identified inconsistency as a significant problem. We found we could be using the same language but not actually meaning the same activity in practice. Addressing this issue required the development of a standardised language used to describe various aspects of the experimental design and resulted in the Mouse Experimental Design Ontology (MEDO). The practical reality of experimental design is that there is no perfect solution but instead transparency is needed to capture how experiments were implemented to allow users to independently assess the potential strengths and weaknesses in the design. A solution for one institute might not be appropriate for another and this can arise from the balance between removing risk and practical considerations where we need to follow the KISS design principle (keep it simple and straightforward). Examples of variation in implementation included how institute manage the potential bias from instrumentation and differences in blinding strategies with different levels of stringency. This variation demonstrates that experimental design, whether on a small or high throughput scale, doesn’t have an obvious simple solution but rather a spectrum of solutions. Information on the experimental design at each centre is collected through the Experimental Design procedure. |
| Blinding | 5 | Describe who was aware of the group allocation at the different stages of the experiment (during the allocation, the conduct of the experiment, the outcome assessment, and the data analysis). | See item 4 |
| Outcome measures | 6 | a. Clearly define all outcome measures assessed (e.g. cell death, molecular markers, or behavioural changes). b. For hypothesis-testing studies, specify the primary outcome measure, i.e. the outcome measure that was used to determine the sample size. | See item 9 |
| Statistical methods | 7 | a. Provide details of the statistical methods used for each analysis. b. Specify the experimental unit that was used for each statistical test. c. Describe any methods used to assess whether the data met the assumptions of the statistical approach. | Ensuring that the appropriate statistical analysis is applied is a common problem in biology. The IMPC has devoted considerable resources to this, particularly in the context of high throughput phenotyping, with new approaches being investigated [2, 3, 4, 5]. An analysis pipeline for a resource with data from many sources is challenged by the number of variables, different data types, the data quantity for an institute, and variation in experimental workflow. An example of the variation in workflow includes different control strategies where one batch of knockout mice can be run with concurrent controls at some institutes whereas at others the knockout and control mice are collected in multiple batches but not necessarily on the same day. The analysis implemented is further complicated by the requirement for the analysis to be completely unsupervised with no user intervention. As such, an analysis pipeline has to be robust, it must process the data consistently, and it cannot be fine-tuned for all possible scenarios. There is a dedicated working group for statistical analysis within the consortium to ensure continuous development of the applied analysis and address any issues. This continuous review has lead to an implementation of an updated analysis pipeline, OpenStat, which is available as an R package on Bioconductor, replacing the previous analysis package PhenStat. As data can be analysed in many ways and each way has strengths and weaknesses, we have developed tools and procedures to enable users to download data for independent analysis by spreadsheet downloads, ftp access and automatic programmatic interfaces. |
| Experimental animals | 8 | a. Provide details of the animals used, including species, strain and substrain, sex, age or developmental stage, and weight. b. Provide further relevant information on the provenance of animals, health/immune status, genetic modification status, genotype, and any previous procedures. | The large scale of the project necessitated the development of an international tracking system which tracks the progress of the genetic modifications from all IMPC centres, starting at the planning stage for each centre, continuing through the generation of the knockout animals, animal breeding and verification of their genotype, to the end of phenotype data capture. With the current advances in the genetic techniques the knockout generation has moved from ES line to CRISPR and this has required an update from the original tracking system iMits to a new system GenTar to ensure information essential for CRISPR lines is captured. The coordination of production and phenotyping between all IMPC centres, and the ultimate repository of the genotypes and international strain nomenclature for each mutant allele produced by the IMPC is a critical component of the project output. Strain, mutation and production status information is fed forwards as needed to the various informatics tools needed to store, analyse and present the data. The IMPC informatics infrastructure has developed an automated method of data capture from all phenotyping centres. This method, is supported by a strict data standard that defines, in additional to the protocol from IMPReSS, data about each animal (including local ID, date of birth, strain, sex and centre). The volume of data on individual animals is vast, so we have worked with the community through extensive user testing to organise the information in as intuitive manner as possible. |
| Experimental procedures | 9 | For each experimental group, including controls, describe the procedures in enough detail to allow others to replicate them, including: a. What was done, how it was done and what was used. b. When and how often. c. Where (including detail of any acclimation periods). d. Why (provide rationale for procedures). | The experimental procedures in IMPC are the backbone to the project, so very early in the project and pilots, a standardised data format and underlying database (IMPReSS) were developed. This allows the capture and organisation of all the necessary information to define the protocols, ensure data reproducibility across centres, and enhance the data with the relevant meta-data for analysis. The IMPReSS database is based on a pipeline concept, which is a series of experimental protocols performed in order, a protocol that defines the method and the parameters and metadata to be measured. The information is organised such that each data point for a parameter of interest can be associated with the relevant procedural and mouse information.For each protocol the following are captured: The purpose of the experiment. The experimental design – number of animals, the age and sex tested. Procedure – the protocol, which can also be called the standard operating procedure, followed. Parameters – the measured variables. Those that are marked as positive for “annotation” indicate variables of interest which will be processed through a statistical analysis pipeline. Metadata – parameters that the area experts have determined as important for capture that could explain potential variation in the parameters of interest (e.g. equipment model). Those that are indicated as “required for analysis” are parameters that are used in data assembly for comparison. IMPReSS has been implemented in IMPC so that every animal (both mutant and control) is associated with a protocol which ensures that across the global project we can report what experiment was carried out on every animal and in what order. To provide trackability, IMPReSS also provides change history information on how the experimental definition has changed with time. IMPReSS thus provides the framework for not only transparency in the procedure and what data are required to be captured, but also stores information to facilitate subsequent analysis and thus is the backbone of the database and web portal. The development of this resource has required extensive collaboration with the institutes, area experts and those constructing the database. |
| Results | 10 | For each experiment conducted, including independent replications, report: a. Summary/descriptive statistics for each experimental group, with a measure of variability where applicable. b. If applicable, the effect size with a confidence interval. | The challenge with the analysis is not only in selecting the most appropriate analysis platform but presenting all the results and the concomitant information in a user accessible way. Data outputs used to make genotype-phenotype associations are visualised graphically and augmented with p value as a measure of biological significance. Accompanying the graphs are summary measures for each group: including number of animals, and appropriate summaries for the data type (e.g. for continuous data the mean and standard deviation). The weight data can be separately accessed for a line of interest by looking at the weight curve data. Data are rarely excluded from the analysis. |
Recommended Set:
| ITEM | RECOMMENDATION | IMPC’s Approach | |
|---|---|---|---|
| Abstract | 11 | Provide an accurate summary of the research objectives, animal species, strain and sex, key methods, principal findings, and study conclusions. | The information typically presented in a manuscript abstract is captured within the website and described within several of these items. |
| Background | 12 | a. Include sufficient scientific background to understand the rationale and context for the study, and explain the experimental approach. b. Explain how the animal species and model used address the scientific objectives and, where appropriate, the relevance to human biology. | The IMPC webpage, About the IMPC, details the goals of the project and the relevance to human biology. In summary, the IMPC aims to systematically discover and ascribe biological function for every gene by generating a knockout mouse line for each protein-coding gene and identifying phenotypic changes between the knockout and control animals. Essential to this effort is to establish collaborative networks that work together to standardise protocols and data analysis. |
| Objectives | 13 | Clearly describe the research question, research objectives and, where appropriate, specific hypotheses being tested. | See item 12 |
| Ethical statement | 14 | Provide the name of the ethical review committee or equivalent that has approved the use of animals in this study and any relevant licence or protocol numbers (if applicable). If ethical approval was not sought or granted, provide a justification. | The IMPC Consortium collects data from international member institutes who collect phenotyping data guided by their own ethical review panels, licenses, and accrediting bodies that reflect the national and/or geo-political constructs in which they operate. We have captured this data via an ethical and funding survey from each contributing institute. The IMPC international members institutes are committed to 3Rs’ principles of Replacement, Reduction and Refinement, and engage in their respective 3Rs communities. |
| Housing and husbandry | 15 | Provide details of housing and husbandry conditions, including any environmental enrichment. | To capture housing and husbandry information, the international community came together and based on the requirement of the ARRIVE guidelines, the Gold Standard publication Checklist reporting Guidelines and the Genetically Altered (GA) Passport constructed a series of questions and answers which was used in the construction of a housing and husbandry survey. Housing and Husbandry protocol has since been implemented to allow the collection of this data. The phenotyping pipelines have been designed to ensure that there are no welfare related issues for a normal mouse, however incidental welfare issues may arise. These will be driven by both the environment and the genetic background of the mice, and commonly include runting, malocclusion and hydrocephalus. Animals presenting with these incidental welfare issues will be assessed on a case by case basis to determine if the issue can be managed through remedial care, such as wet mash on the cage floor for runting or clipping teeth in the case of malocclusions, or whether our ethical obligation to the animals and the scientific endpoints are better served by euthanizing the affected animal and providing a replacement. In addition to these incidental welfare concerns, a welfare issue could occur with genetically altered mice. Two standard complementary approaches are used within the community to identify and manage the potential systematic welfare issues that could arise. First, during the generation of the first homozygous progeny there are a series of assessments considering the basic dysmorphology to identify welfare concerns. If significant welfare concerns are raised, then the breeding strategy is modified to avoid the generation of homozygous mice and heterozygous mice are phenotyped instead. In addition to this early assessment, ongoing monitoring is carried out through the lifetime of the colony. Should the animals present a welfare concern, either an intervention is made in the form of the Alternate Mouse Pipeline or, when applicable, remedial action is taken within husbandry (e.g. long water spouts or food on cage base) or within experimental procedures (e.g. an alternate anaesthetic is used or tests are dropped). The Alternate Mouse Pipeline is an adapted phenotyping pipeline where a bespoke pipeline is run where the age and protocols are adapted to manage the welfare of the line. These two complementary approaches allow us to maximise the value of information extracted from the lines while minimising welfare issues. If the mice/cryopreserved sperm are requested by the community, the welfare concerns are documented in a report when ordered (e.g. to the Genetically Altered (GA) passport) from the mouse repositories (discussed further within item 19). |
| Animal care and monitoring | 16 | a. Describe any interventions or steps taken in the experimental protocols to reduce pain, suffering and distress. b. Report any expected or unexpected adverse events. c. Describe the humane endpoints established for the study and the frequency of monitoring. | As discussed in Housing and husbandry (item 15), the pipelines have been designed such that a normal mouse would not be expected to present with adverse welfare indicators and systems are in place to minimise genotype specific welfare issues. Modifications required to ensure the welfare of a knockout line, are captured in the report issued with the mouse line if requested. An Alternative Pipeline has also been created to allow capturing of data at a time point deviating from that defined in the standard phenotyping pipeline due to welfare issues. A Welfare procedure has also been implemented to capture data on the welfare issues observed in the animals. |
| Interpretation / scientific implications | 17 | a. Interpret the results, taking into account the study objectives and hypotheses, current theory and other relevant studies in the literature. b. Comment on the study limitations including potential sources of bias, limitations of the animal model, and imprecision associated with the results. | The results are interpreted in an automated fashion following statistical analysis by the assignment of a mammalian phenotype ontology term when a significance threshold agreed upon by the community is reached. A mammalian phenotype term (MP term) is a standardised ontology for describing a phenotype developed by the Mouse Genome Informatics group. For example the MP term MP:0005634 – decreased circulating sodium level – is defined as less than the normal concentration of this ion in the blood and it is assigned to mutant mouse strains when blood sodium values for members of this strain are significantly lowered when compared to controls. Phenotype assignment can also occur for a particular sex of a mutant strain when the effect is only observed in one sex. The use of the standardised ontology is a critical step to allow comparison across studies and species. Data about animals, housing conditions, experimental design and metadata at procedure level is collected so these can be accounted for in the analysis or referred in interpretation of the results to reduce and assess any limitations, biases and imprecisions. These are continuously assessed. The size of the database itself does impose limits as the analyses need to be automated and reliable. |
| Generalisability / translation | 18 | Comment on whether, and how, the findings of this study are likely to generalise to other species or experimental conditions, including any relevance to human biology (where appropriate). | Mouse repositories play an important role in the generalizability of results by allowing researchers to perform follow up studies on genetically identical animals as those used in previous analysis. All strains generated as part of the IMPC project are available to the research community via the established mouse repositories such as MMRRC (Mutant Mouse Resource and Research Centres) and EMMA (European Mutant Mouse Archive). These repositories include details about allele structure, genetic background, pathogen exclusion list and any potential issues in husbandry and welfare that may result from carried mutations. This information increases the likelihood of reproducing previous results, the lack of which is seen as a detriment to translating mouse research studies to human biology [6]. Many animal experiments study only one sex, typically the males, to avoid potential issues with oestrogen cycles. There has been growing concern over the gender imbalance in biomedical research [7, 8, 9] and this had led to a policy shift within the US National Institutes of Health such that they now require all grant applicants to report their plans for balancing male and female animals in preclinical studies. Fortunately, IMPC members decided at the start of the project that phenotype data should be collected from both sexes and this design supports the generalizability of the findings. In addition, our continuous analysis pipeline is designed to detect sexual dimorphism and we associated a tag to the MP terms to classify the effect observed. |
| Protocol registration | 19 | Provide a statement indicating whether a protocol (including the research question, key design features, and analysis plan) was prepared before the study, and if and where this protocol was registered. | Strategy and governance information can be found on our About the IMPC page. |
| Data access | 20 | Provide a statement describing if and where study data are available. | All of the IMPC data is available in multiple formats in each data release, subject to passing quality control (See item 3). |
| Declaration of interests | 21 | a. Declare any potential conflicts of interest, including financial and non-financial. If none exist, this should be stated. b. List all funding sources (including grant identifier) and the role of the funder(s) in the design, analysis and reporting of the study. | Further details about the International Knockout Mouse Consortium (IKMC) including funding sources is available on the website, as well as details of the Knockout Mouse Programme (KOMP) which funds several centres. All individual members of the consortium are listed on our Consortium members page. |