• Register
X
Forgot Password

If you have forgotten your password you can enter your email here and get a temporary password sent to your email.

X

Leaving Community

Are you sure you want to leave this community? Leaving the community will revoke any permissions you have been granted in this community.

No
Yes
X
Forgot Password

If you have forgotten your password you can enter your email here and get a temporary password sent to your email.

SciScore

SciScore

All raw classifiers for SciScore are higher than 70% F1, the harmonic mean of precision (quantification of false positives) and recall (false negatives). The F1 depends on how many examples, in what size of dataset were used to determine whether the classifier could detect a particular item. In general, the larger the number of examples that the algorithm can learn from, the better it can learn. 

Table S1 (direct link) of Menke et al, 2020 shows the specifics for each individual classifier, such as the institutional review board finder. All together there are over 30 classifiers that work in concert to identify each type of reagent, and rigor criterion. Some classifiers, such as the antibody finder, are composites of multiple subparts, e.g., a detector for the catalog number, or source organism for an antibody. Other classifiers detect paragraph types or tables. 


All SciScore Criteria for Version 2 of SciScore and the training set size:

Note the list of source checklists that address a particular criterion is not exhaustive.


Entity Type

Source

What is this?

Rigor Criteria (5 total points)

Institutional Review Board Statement 

MDAR

A statement (usually a single sentence) addressing IRB approval for biomedical research involving human subjects (or why IRB approval was not required).

Example: All human work was conducted under human subjects protocols approved by the Stanford Institutional Review Board (IRB), the University of Michigan UM-IRBMED, and the Ethical Committee of d’Ile de France II.

Example: The trial was approved by the NRES Committee London—South East. [less] [less]

Example: All human work was conducted under human subjects protocols approved by the Stanford Institutional Review Board (IRB), the University of Michigan UM-IRBMED, and the Ethical Committee ...[more]

Consent Statement 

MDAR

A statement (usually a single sentence) addressing subject/patient consent in human research (or why consent was not required).

Example: Written informed consent was obtained from parents of all participating children and oral assent was obtained from 7-year olds.

Example: All infants were enrolled with informed parental permission under a protocol that was reviewed and approved by the Institutional Review Boards of the respective study sites. [less] [less]

Example: Written informed consent was obtained from parents of all participating children and oral assent was obtained from 7-year olds.

Example: All infants were enrolled with informed parent ...[more]

Institutional Animal Care and Use Committee Statement 

MDAR, ARRIVE

A statement (usually a single sentence) addressing IACUC ethical approval for research involving vertebrate organisms.

Example: All animal experiments were performed in accordance with relevant guidelines and regulations and were approved by the University of Pennsylvania Institutional Animal Care and Use Committee (IACUC).

Example: All animals used in this study were treated in accordance with UK Animal (Scientific Procedures) legislation and under the appropriate project licenses, national and local ethical approval. [less] [less]

Example: All animal experiments were performed in accordance with relevant guidelines and regulations and were approved by the University of Pennsylvania Institutional Animal Care and Use Com ...[more]

Field Sample Permit

MDAR

A statement disclosing the relevant permits obtained (including the name of the permitting authority) for field studies (or why approval was not required).

Example: Permission to conduct field surveys on each location was given by the individual landowners concerned, and by the regulatory authority (Natural England) in those situations where the field site was afforded protected status (i.e. Site of Special Scientific Interest). [less] [less]

Example: Permission to conduct field surveys on each location was given by the individual landowners concerned, and by the regulatory authority (Natural England) in those situations where the ...[more]

General Euthanasia

AVMA Guidelines

The mention of culling or euthanasia for the animals used in a research experiment.

Example: Mice injected with CGG-NP23 were boosted on day 21 with the same inoculum and killed on day 28.

Euthanasia Agent

AVMA Guidelines

The mention of an agent or method (i.e. cervical dislocation or carbon dioxide inhalation) used in the euthanasia of research animals.

Example: Twelve hours after the final doses, the animals were euthanized by cervical dislocation.

Inclusion & Exclusion Criteria

Landis et al., 2013 (NIH), MDAR, CONSORT, ARRIVE

A statement or statements reporting the criteria prospective subjects must have (or must not have) in order to be included/excluded in a study.

Example: Exclusion criteria were pregnancy, severe medical conditions, abnormal laboratory baseline values, unstable psychiatric features (e.g., suicidal), a history of alcoholism or drug abuse, epilepsy, brain trauma with loss of consciousness, neurological illness, or a concomitant Axis I psychiatric disorder. [less] [less]

Example: Exclusion criteria were pregnancy, severe medical conditions, abnormal laboratory baseline values, unstable psychiatric features (e.g., suicidal), a history of alcoholism or drug abu ...[more]

Attrition

Landis et al., 2013 (NIH), MDAR, ARRIVE

A sentence reporting whether any sample or data point was omitted (participant drop out or intentionally excluded by author). This includes sentences that report no attrition.

Example: One participant withdrew from the yoga exercise group due to personal consideration.

Type of Replication

Landis et al., 2013 (NIH), MDAR

A description of the type of replication being performed (e.g. technical replicant or biological replicant - biologically distinct samples or repeated measures of the same sample).

Example: Each real-time PCR experiment included technical replicates, in a final volume of 15 µL.

Number of Replications

Landis et al., 2013 (NIH), MDAR

A brief mention of the number of times an experiment was independently performed.

Example: The experiment was replicated four times.

Randomization of subjects into groups

Landis et al., 2013 (NIH), MDAR, CONSORT, ARRIVE

Considered addressed when a statement describing whether randomization was used (e.g.  assigning subjects to experimental groups, positions in a multiwell device, processing order, etc.).  

Example: Animals were assigned to experimental groups using simple randomization.

Example: Communication with schools, and elicitation of willingness to participate, was conducted before the village-level randomization took place. [less] [less]

Example: Animals were assigned to experimental groups using simple randomization.

Example: Communication with schools, and elicitation of willingness to participate, was conducted before the v ...[more]

Blinding of investigator or analysis

Landis et al., 2013 (NIH), MDAR,  CONSORT, ARRIVE

A statement discussing the degree to which experimenters were unaware (or blinded) of group assignment and/or outcome assessment.

Example: Responses were then scored by an experimenter blinded to injection condition and experimental cohort.

Example: All the analysis was performed by a person unaware of the experimental question.

Power analysis for group size

Landis et al., 2013 (NIH), MDAR, CONSORT, ARRIVE

A statement addressing how (and if) an appropriate sample size was computed.

Example: Sample size was based on estimations by power analysis with a level of significance of 0.05 and a power of 0.9.

Example: Sample size calculation was done for the primary aim of this study, i.e. FMD, as reported previously.  [less] [less]

Example: Sample size was based on estimations by power analysis with a level of significance of 0.05 and a power of 0.9.

Example: Sample size calculation was done for the primary aim of this s ...[more]

Sex as a biological variable

MDAR, NIH, CONSORT, ARRIVE 

Reporting the sex of any and all organisms, cell lines, and human subjects.

Example: Six healthy adult rhesus macaques (Macaca mulatta) of Chinese origin (4–8 kg, three males and three females, 4–8 years old) were inoculated intramuscularly (i.m.) with 1,000 pfu of EBOV Makona strain.

Example: In each session, the behavior of each mother was recorded every 2?min. [less] [less]

Example: Six healthy adult rhesus macaques (Macaca mulatta) of Chinese origin (4–8 kg, three males and three females, 4–8 years old) were inoculated intramuscularly (i.m.) with 1,000 pfu of E ...[more]

Age

CONSORT, ARRIVE

A statement reporting the age (or stage of life) of an experimental subject or organism. 

Example: All mice were 8–16 weeks of age.

Weight

ARRIVE

A statement reporting the weight of an experimental organism. 

Example: One Thoroughbred healthy adult horse (540 kg body mass) from the Royal Veterinary College (RVC) participated in the study.

Cell Line Authentication 

MDAR, NIH

A statement detailing how the cell lines used were authenticated (e.g. short tandem repeat analysis). This is only required when cell lines are detected.

Example: MOLM-14 cells were authenticated by STR profiling and flow cytometry.

Example: All cell lines were obtained from ATCC, tested negative for mycoplasma, and their identity was verified by short tandem repeat analysis (Promega GenePrint 10 System). [less] [less]

Example: MOLM-14 cells were authenticated by STR profiling and flow cytometry.

Example: All cell lines were obtained from ATCC, tested negative for mycoplasma, and their identity was verified ...[more]

Cell Line Contamination Check 

MDAR, NIH

A statement addressing the mycoplasma contamination status of the cell lines used. This is only required when cell lines are detected.

Example: All cell lines were obtained from ATCC and tested negative for mycoplasma contamination.

Example: All cell lines were confirmed to be mycoplasma free using a PCR-based detection strategy with positive and negative controls. [less] [less]

Example: All cell lines were obtained from ATCC and tested negative for mycoplasma contamination.

Example: All cell lines were confirmed to be mycoplasma free using a PCR-based detection strat ...[more]

Protocol Identifiers

MDAR, CONSORT (clinical trial number required)

We use a series of regular expressions to find and link certain patterns (usually accession numbers) with their corresponding database. Protocol identifiers include registered clinical trials like clinicaltrials.gov and EU Clinical Trials Register and protocol repositories like protocols.io and protocol exchange. [less] [less]

We use a series of regular expressions to find and link certain patterns (usually accession numbers) with their corresponding database. Protocol identifiers include registered clinical trials ...[more]

Example: To study the effect of dutasteride on Abi metabolism, serum samples were collected from patients treated on a phase II clinical trial at Dana-Farber Cancer Institute (NCT01393730).

Code Availability

MDAR

A sentence disclosing the availability of any computer code (either newly generated or previously created) that is essential for replicating the main findings of the study.

Example: Image analysis was performed with ImageJ software macro (code available upon request).

Code Identifiers

MDAR

We use a series of regular expressions to find and link certain patterns (usually accession numbers or URLs) with their corresponding code repositories.

Example: All scripts used for the analyses in this paper are available at the Github repository (https://github.com/vplagnol/recursive_splicing).

Data Availability

NIH, MDAR, ARRIVE

A sentence disclosing the availability of any data (either newly generated or from a previous study) that is essential for replicating the main findings of the study.

Example: All other relevant data that support the conclusions of the study are available from the authors on request.

Data Identifiers

MDAR

We use a series of regular expressions to find and link certain patterns (usually accession numbers) with their corresponding data repositories.

Example: The complete results are uploaded in NCBI GEO as GSE75387.

Key Biological Resources (5 total points)

Antibody

MDAR, NIH, STAR, RRID

SciScore attempts to find all antibody entities within the methods section. “Identifiable” antibodies are reported with any metadata required to uniquely identify the antibody used such as vendor, catalog number, clone ID, batch number, or RRID. [less] [less]

SciScore attempts to find all antibody entities within the methods section. “Identifiable” antibodies are reported with any metadata required to uniquely identify the antibody used such as ve ...[more]

Example: ATF3 antibody (Santa Cruz Biotechnology) was used at 1:2000.

Example: Slices were then washed (3x) and placed in PBS containing the following; 1% (vol/vol) normal goat serum, 1% (vol/vol) BSA, 0.25% (vol/vol) Triton X-100, and mouse monoclonal anti-5.8S rRNA, clone Y10b at 1:500 (Abcam, ab37144, RRID: AB_777714) overnight at 4°C. [less] [less]

Example: ATF3 antibody (Santa Cruz Biotechnology) was used at 1:2000.

Example: Slices were then washed (3x) and placed in PBS containing the following; 1% (vol/vol) normal goat serum, 1% (vol/ ...[more]

Organism

MDAR, NIH, RRID, STAR, ARRIVE

SciScore attempts to find all organism entities within the methods section. “Identifiable” organisms are reported with any metadata required to uniquely identify the organism used such as vendor, catalog number, or RRID. [less] [less]

SciScore attempts to find all organism entities within the methods section. “Identifiable” organisms are reported with any metadata required to uniquely identify the organism used such as ven ...[more]

Example (mouse): Adult (10-12 weeks; 25-30g) male C57BL/6 and TH-Cre mice were group-housed until surgery.

Example (fly): To generate PIP821bp? the following sgRNA was generated 5’-GCAGGAGGAGGTACAGCGGG-3’ and cloned into pU6-2-BbsI-gRNA (DGRC #1363) and then subsequently injected into w1118; vas-Cas9 (RRID:BDSC_51324, Rainbow Transgenics).

Example (fish): The transgenic lines used in this study were Tg(kdrl:EGFP)s843 (Jin et al., 2005), Tg(lyve1b:DsRed2)nz101, Tg(lyve1b:EGFP)nz150 (Okuda et al., 2012), Tg(mpeg1:EGFP)gl22, Tg(mpeg1:Gal4FF) gl25 (Ellett et al., 2011), Tg(lyz:EGFP)nz117 (Hall et al., 2007), Tg(i-fabp:RFP)as200 (Her et al., 2004), Tg(UAS-E1b:nfsB-mCherry)c264 (Davison et al., 2007) and Tg(-8.mpx:KalTA4)gl28. [less] [less]

Example (mouse): Adult (10-12 weeks; 25-30g) male C57BL/6 and TH-Cre mice were group-housed until surgery.

Example (fly): To generate PIP821bp? the following sgRNA was generated 5’-GCAGGAGGAGG ...[more]

Cell Line

MDAR, NIH, STAR, RRID

SciScore attempts to find all cell line entities within the methods section. “Identifiable” cell lines are reported with any metadata required to uniquely identify the cell line used such as vendor, catalog number, or RRID. [less] [less]

SciScore attempts to find all cell line entities within the methods section. “Identifiable” cell lines are reported with any metadata required to uniquely identify the cell line used such as ...[more]

Example: The lung cancer cell line, H1299, was obtained from the American Tissue Culture Collection (Manassas, VA).

Example: J774A.1 murine monocytes and macrophages (ATCC, number TIB-67) were cultured at 37 °C in a humidified air/carbon dioxide (CO2) (19:1) atmosphere in RPMI medium supplemented with 10% (v/v) heat-inactivated fetal bovine serum, penicillin (100 IU/mL), streptomycin (100 µg/mL), and amphotericin B (250 ng/mL). [less] [less]

Example: The lung cancer cell line, H1299, was obtained from the American Tissue Culture Collection (Manassas, VA).

Example: J774A.1 murine monocytes and macrophages (ATCC, number TIB-67) were ...[more]

Plasmid 

STAR, RRID

SciScore attempts to find all plasmid entities within the methods section. Plasmids were not used in this analysis.

Example: The constructions were prepared using the vector pSpCas9(BB)-2A-Puro (PX459) V2.0, which was a gift from Feng Zhang (Addgene plasmid #62988; RRID: Addgene_62988).

Example: For expression in HEK293 cells, INF2 was first subcloned into pGADT7.3 (BspEI/XmaI-XhoI) and then into pEGFP-C3 (EcoRI-SalI). [less] [less]

Example: The constructions were prepared using the vector pSpCas9(BB)-2A-Puro (PX459) V2.0, which was a gift from Feng Zhang (Addgene plasmid #62988; RRID: Addgene_62988).

Example: For express ...[more]

Oligonucleotide 

STAR, MDAR

SciScore attempts to find all oligonucleotide entities within the methods section. Oligonucleotides do not impact score and were not used in this analysis.

Example: Activating Notch1 mutations in mouse models of T-ALL, Blood 2006 107:781–785), including one new oligonucleotide primer pair: Ex34B-f: 5?-GCCAGTACAACCCACTACGG-3?; Ex34B-r: 5?-CCTGAAGCACTGGAA-AGGAC-3?

Example: Primers used were GRHL2-1-424-F (TATATAGGATCCATGTCACAAGAGTCGGACAA), GRHL2-1-424-R (ATATAAAGATCTT­TTTCTTTCTGCTCCTTTGT), GRHL2-438-625-F (TAAATTAGATCTAAAGGCCAGGCCTCCCAA­AC), and GRHL2-438-625-R (TTATATGTCGACCTAGATTTCCATGAGCGTGA). [less] [less]

Example: Activating Notch1 mutations in mouse models of T-ALL, Blood 2006 107:781–785), including one new oligonucleotide primer pair: Ex34B-f: 5?-GCCAGTACAACCCACTACGG-3?; Ex34B-r: 5?-CCTGAAG ...[more]

Software Project/Tool

STAR, RRID

SciScore attempts to find all software tools within the methods section. “Identifiable” tools are reported with an RRID or are able to be uniquely identified through a distinct name/URL.

Example: Image J was used to process and analyze raw images (Extended Data Video 2, and 3).

Example: All simulations were performed using the NEURON simulation environment (Carnevale and Hines, 2006 ).

Statistical Tests

MDAR, CONSORT, ARRIVE

A statement reporting the statistical tests used during the experiment, ideally including a justification (i.e. whether the tests’ assumptions are met).

Example: Kruskal-Wallis test was used to compare three or more groups.




Table 2: Classifier Performance

Entity Type

F1

Precision

Recall

Training Set Size (# of entities/# of sentences)

Mean ± SD

Mean ± SD

Mean ± SD

Rigor Criteria (5 total points)

Institutional Review Board Statement 

81.41 ± 3.62 

84.45 ± 5.26

79.57 ± 8.83

340/78,170

Consent Statement 

94.75 ± 1.68 

96.29 ± 2.42

93.38 ± 3.63

373/78,170

Institutional Animal Care and Use Committee Statement 

81.30 ± 4.20 

89.30 ± 4.60

74.89 ± 6.12

591/78,170

Field Sample PermitA

76.6 

90.0

66.67

537

General EuthanasiaA

94.2

96.1

92.4

530

Euthanasia AgentA

65.5

82.6

54.3

350

Inclusion and Exclusion CriteriaA

89.4

86.6

92.4

224

AttritionA

44.7

50.0

40.4

940

Type of ReplicationA

92.9

100.0

86.7

150

Number of ReplicationsA

67.3

71.3

63.7

113

General ReplicationA

86.6

91.3

82.5

1,140

Randomization of subjects into groups

83.05 ± 3.04 

80.25 ± 5.05

86.45 ± 4.64

368/52,945

Blinding of investigator or analysis

78.96 ± 12.38 

77.74 ± 17.16

81.79 ± 10.32

183/52,945

Power analysis for group size

64.45 ± 29.37 

73.74 ± 34.13

59.50 ± 26.91

81/52,945

Sex as a biological variable

88.32 ± 3.91 

87.94 ± 6.03

88.93 ± 3.52

862/52,945

AgeA

76.5

83.2

70.7

1,470

WeightA

80.5

92.1

71.4

490

Cell Line Authentication 

54.08 ± 11.88 

85.70 ± 10.78

41.15 ± 12.82

155/14,792

Cell Line Contamination Check 

91.70 ± 5.24 

93.35 ± 7.15

90.65 ± 7.05

151/14,792

Protocol IdentifiersA

17 patterns (clinical trials (US & EU), protocol exchange, STAR protocols, JOVE, Bio-protocol, MethodsX, nature protocols, springer protocols, biotechniques, PROSPERO) 

Code AvailabilityA

0

0

0

10

Code IdentifiersA

4 patterns (github, google code, sourceforge, bitbucket)

Data AvailabilityA

75.6

77.3

73.9

230

Data IdentifiersA

28 patterns (doi, genomeRNAi, GEA, dbGAP, dbSNP, GEO, SRA arrayexpress, JGA, EGA, metabolights, peptide atlas, proteomeXchange, Flow repository, Biostudies, ClinVar, MassivE, pcddb)   

Key Biological Resources (5 total points)

Antibody

78.94 ± 2.62 

86.89 ± 3.78

72.46 ± 3.20

16,772/53,216

Organism

66.05 ± 4.70

79.91 ± 6.28

56.64 ± 5.75

4,439/45,500

Cell Line

70.07 ± 5.95 

86.48 ± 3.27

59.34 ± 8.03

1,763/45,500

Plasmid 

79.62 ± 3.35

92.53 ± 3.80

70.09 ± 4.85

2,568/63,400

Oligonucleotide 

83.03 ± 9.05

95.28 ± 3.13

74.94 ± 13.90

1,893/63,400

Software Project/Tool

89.03 ± 0.90

92.49 ± 2.08

85.84 ± 1.10

10,161/19,002

Statistical TestsA

97.5

97.8

97.2

4,600

A New to version 2.0


Search

Recent ASWG Tool Limitations

rTransparent

Seek 'n Blastn

ODDPub

ASWG Tool Limitations Tags

X

Are you sure you want to delete that component?