Reviewing papers

From GO Wiki
Jump to: navigation, search

The purpose of this page is to support the members of the GOC by providing guidelines on how to review a paper which implements GO.

Assessing the scientific merit

(added some and moved general from the bottom ,val)


  1. Does the research question posed make sense
  2. Does the methodology outlined address the questions posed
  3. Do the data and results support the conclusions
  4. Is this work a significant advance over the previous literature
  5. Is there anything in the paper that seems unsure/controversial/inconclusive (Q what do we do in this case??)


Comment Val, perhaps the points below (which were under assessing scientific merit) should go in the relevant sections below. These were specific problems applicable to this particular paper, other papers wil have other issues and this document should be more general?:


  1. There should be a minimal number of simplifying assumptions (move to methodology? )
  2. They show an awaresness of the rigor curators apply when inferring annotations from sequence similarity (move to source data interpretation? applies only to papers which make some special case about ISS, for example assessing accuracy)
  3. They take into account the relative completeness or incompleteness of annotation, if this is relevant to their analysis and results (results)
  4. Blast scores should be used intelligently, not simply arbitrary cut-offs (move to methodology). An awareness that annotation providers do not use Blast with an arbitrary cut-off for making annotations inferred from sequence similarity.
  5. Are methods and/or results validated against examples if appropriate (e.g. incorrect annotations for a paper about annotation accuracy, other examples?). (move to methods/results)

Materials, that is, the data sets used as input

  1. Are the datasets used appropriate to answer the research question posed
  2. Is the paper explicit about
    • Which data sources it has used (version of ontology/ date of annotation files/ numbers and types of annotations used)
    • How it has partitioned these resources (which subsets of the data were used for which parts of the analysis)

Source Data Interpretation

  1. Do the authors take account of the DAG structure in their analysis, and show a consideration for the consequences of misuse of the DAG (direct vs. indirect annotations). It is CRUCIAL that any accurate analysis which compare GO annotations, or look for enrichment of GO terms, implement the DAG structure(extend)
  2. Do the authors display an understanding of how current curation best practices may affect their results and/or conclusions. This includes but is not limited to:
    • Understanding of procedures used by curators at contributing MODs for the evaluation of similarity/orthology for functional transfer
      • i) Functional transfer is only made between orthologs except in exceptional circumstances (protein family information is sometimes used)
      • ii) Blast hits alone are NOT routinely used for functional transfer of annotation
      • iii) Annotation transfer is sometimes necessarily more conservative than the annotation of the gene product it is made from. This is espeocially true when the annotation species has duplicate members of the gene product in question. In these cases the functions may be partitioned (subfunctionalization) or subtly changed (neofunctionalization) and more specific predictions cannot be made with confidence.
    • Evidence code (meaning and use) particularly with respect to inferring functions based on sequence similarity
      • i) ISS annotation are now only made from experimentally supported sources, reducing the possibility of transitive annotations. The source of the ISS annotation is always recorded in the 'with' column
  3. Awareness of the possible incomplete nature of the annotations due to the curation backlog
  4. Qualifier awareness (especially NOT), but also 'contributes to' and 'colocalises_with'
  5. Add something about 'unknown' (root node) annotations (supported by evidence code ND)

Methodology

  1. Is the reviewer familiar with the specific types of analyses used in the manuscript
  2. Are the analyses used relevant to the research questions posed
  3. Are algorithms developed and reasoning used to evaluate the data robust for the purpose
  4. Are any statistical tests used appropriate to evaluate significance
  5. Use of third party software -Are the version, parameters, cut-offs specified
  6. Reproducibility- Are the methods for making the conclusions fully described and reproducible

Results

Citations

Are all appropriate citations made to

  1. Support scientific statements
  2. Recognise previous work
  3. Describe input data sources

Miscellaneous

  1. Is the terminology used to describe aspects of GO (terms/annotations etc), and general description of GO correct
  2. Reviewers should provide specific citations of claims made in the review

Different types of papers

We should also probably have different guidelines for different types of papers:

  1. Technical papers which use GO
  2. Assessment of annotation accuracy
  3. Functional prediction using GO
    • Are these truly 'functional predictions' or are they annotation ommissions (i.e would the annotations be made from sequence similarity or other methods if the annotations were complete?)
  4. GO Tools
    • Do existing tools have this functionality?
    • What additional benifits does the tool provide?
    • How frequently will the resource be updated (ontology and annotations)
    • Expand from list to GO tool submitters
  5. Papers that evaluate biological results using GO enrichment
    • Have the authors considered enrichment of unknown terms (i.e those annotated to the root node), many tools (including GO term finder), do not handle unknown terms at present
  6. etc...

Notes

Perhaps we should also look at journals guidelines and criteria for assessing informatics papers, and especially multidisciplinary papers (as GO papers often increasingly are). A reviewer might be competent to review the biological or technical content, but not vice versa, which makes the review process more difficult