Issues to Consider About Significance Testing

1. When are the statistics calculated?

• Table printing phase
• Table building phase

2. What type of tables will be constructed?

3. Will the tables be weighted?
4. What is the statistical/percent base on the tables?
5. What Confidence level is needed?
6. Will all of the tables need to be stat tested?
7. Will all of rows on the table need to be tested?
8. Are there any special labeling options needed?
9. Are there any special table printing options needed?
10. Is there a request for a specific type of variance?
11. Is there enough room in the cell to print all of the letters?
12. Trouble Shooting
13. Chart Summarizing Statistical Tests Available

Examples of Tests Available and Discussion of Issues

2A. Table Printing Phase Statistics (REGION tests)

A) One-Way Analysis of Variance

Syntax: EDIT={:TABLE_TESTS=[$R 1 TO 2 BY 1 TO LAST]TABLE_TESTS=[$R 3 TO 4 BY 1 TO LAST]TABLE_TESTS=[$R 5 TO 6 BY 1 TO LAST]}Syntax: LOCAL_EDIT={: COLUMN_STATISTICS_VALUES=VALUES(4,3,2,1)COLUMN_MEAN, COLUMN_STD, ANOVA}

B) Chi-Square

Syntax: EDIT={:TABLE_TESTS=[$R 1 TO 2 BY 1 TO LAST]TABLE_TESTS=[$R 3 TO 4 BY 1 TO LAST]TABLE_TESTS=[$R 5 TO 6 BY 1 TO LAST]}Syntax: LOCAL_EDIT={: CHI_SQUARE }Syntax: [-COLUMN_STATISTICS_VALUES] DON'T KNOW/NOT SURE

Syntax: LOCAL_EDIT={:COLUMN_STATISTICS_VALUES=VALUES(10,...,1)STATISTICS_TESTS=I=BCDDO_STATS=.90}Syntax: [PRINT_ROW=MEAN, DO_STATISTICS=ALL_PAIRS]  MEAN

Syntax: LOCAL_EDIT={:COLUMN_STATISTICS_VALUES=VALUES(10,...,1)STATISTICS_TESTS=I=BCDDO_STATS=.90}Syntax: [PRINT_ROW=MEAN, DO_STATISTICS=NK]  MEAN

C) Kruskal-Wallis

Syntax: LOCAL_EDIT={:COLUMN_STATISTICS_VALUES=VALUES(10,...,1)STATISTICS_TESTS=I=BCDDO_STATS=.90}Syntax: [PRINT_ROW=MEAN, DO_STATISTICS=KW] MEAN

2B2. Table Building Phase Statistics (COLUMN Tests)

Syntax: STATISTICS=: PRINTABLE_T I=BC,I=GHSyntax:  [STATS DO_T_TEST=11]  T[STATS DO_SIG_T=11]  SIGTSyntax: EDIT={: -STATS_FOOTNOTE }

B) Inclusive T-Tests

Syntax: STATISTICS=: T=ABCDEFGSyntax: EDIT={: DO_STATISTICS=.90 }

C) All Possible Pairs (Default)

Syntax: STATISTICS=: ABCDEFG Syntax: EDIT={: ALL_POSSIBLE_PAIRS_TESTDO_STATISTICS=.90 }

D) Newman-Keuls

Syntax: STATISTICS=: AB,CD,EFGSyntax: EDIT={: NEWMAN_KEULS_TESTDO_STATISTICS=.90 }

E) Anova Scan

The ANOVA-Scan and Fisher tests are similar in that both firstperform an ANOVA (analysis of variance) on the set of columns,and then only test the individual columns if the ANOVA showssignificance.

Syntax: STATISTICS=: I=AB,I=CD,I=EFGSyntax: EDIT={: ANOVA_SCANDO_STATISTICS=.90 }

The difference between the Anova Scan and the Fisher test, isthat the Fisher takes into account that multiple columns are being tested.

Syntax: STATISTICS=: I=AB,I=CD,I=EFGSyntax: EDIT={: FISHERDO_STATISTICS=.90 }

Syntax: STATISTICS=: RM=AB,RM=CD,RM=EFGH,RM=IJKL,RM=MNOPSyntax: EDIT={: DO_STATISTICS=.90 }

A direct comparison of two rows is similar to the test thatis performed on columns.

Syntax: STATISTICS=: D=1,2

B) Distributed Preference Tests

A distributed preference test allows a “No Preference” (orsimilar neutral third category) to be distributed betweenthe two original categories while ensuring the integrity ofthe underlying statistical test.

Syntax: ROW=: SELECT_VALUE([7^1/X],VALUES(1,.5)) WITH &SELECT_VALUE([7^2/X],VALUES(1,.5))Syntax: STATISTICS=: P=1,2

3. Will the tables be weighted?

Syntax: ~SET MULTIPLE_WEIGHT_STATISTICS

A) One base
1. Total Row (default)
2. Any Response Row

Syntax: [BASE_ROW PRINT_ROW=AR VPER=AR REPRINT_BASE] Base }

B) Base changes in the body of the table

Syntax: ROW=: $[BASE] [72^1.3] $[] &[5^0/1/2/3/4/5/6/7/8/9] BY [72^1.3] &$[MEAN,STD] [5] BY [72^1.3]Syntax: ROW=: $[FILTER_BASE] [72^1.3] $[] &[5^0/1/2/3/4/5/6/7/8/9] $[MEAN,STD] [5]Creates effective_n row that needs to be dealt with in the stub

Syntax: EDIT={: DO_STATISTICS=.90 }

C) .95(Default)

Syntax: EDIT={: DO_STATISTICS=.95 }

D) .99

Syntax: EDIT={: DO_STATISTICS=.99 }

E) approximately .nn (not usable with Newman-Keuls)

Syntax: EDIT={: DO_STATISTICS=APPROXIMATELY .85 }

F) bi-level

Syntax: EDIT={: DO_STATISTICS=.90+.95 }

Syntax: LOCAL_EDIT={: -DO_STATISTICS }Syntax: STATISTICS=;

Syntax: ~SET MEAN_STATISTICS_ONLY

C) Pick and choose on a table by table by table basis

Syntax: ROW=: [21^4,5] $[-DO_STATISTICS] [21^5/4/3] &$[DO_STATISTICS] [21^1,2] $[-DO_STATISTICS] [21^2/1]Syntax: ROW=: [21^(STATISTICS)4,5/5/4/3/(STATISTICS)1,2/2/1]

D) Rows that can’t be tested:

Syntax: [BASE_ROW] BASESyntax: [-BASE_ROW] TOTAL

B) Show footnote

———————————
(sig=.05) (all_pairs) columns tested BC, DEF, GHIJ

Syntax: EDIT={: STATS_FOOTNOTE }Syntax: EDIT={: -STATS_FOOTNOTE }

C) Change the default letters that are assigned to columns

Syntax: EDIT={: STAT_MARKING="ZYXW" }Syntax: EDIT={: -STAT_MARKING }

D) For T-Tests, print t-value or significance of any row whichhas stats tests

Syntax:  STATISTICS=: PRINTABLE_T I=AB,I=AC,I=AD,I=AE,I=AF,I=AGSyntax: [STATS, DO_T_TEST=*] T-VALUE[STATS, DO_SIG_T=*]  SIGNIFICANCESyntax: [STATS, DO_T_TEST=9] T-VALUE[STATS, DO_SIG_T=9]  SIGNIFICANCE OF TSyntax: [PRINT_ROW=MEAN] MEAN[PRINT_ROW=STD] STANDARD DEVIATION[PRINT_ROW=SE] STANDARD ERROR[DO_T_TEST=PRINT_MEAN] T-VALUE[DO_SIG_T=PRINT_MEAN] SIGNIFICANCE OF T

E) Mark_Chi_Square

Marks cells as significant based on chi-square testing. Foreach significant CHI_SQUARE test on the table, a formula isused to determine which cells are the most extreme.

Syntax: EDIT={: -CHI_SQUARE_ANOVA_FORMAT }

G) Chi-square or Anova showing only the significance under the table

Syntax: EDIT={: SHOW_SIGNIFICANCE_ONLY }

9. Are there any special table printing options needed?

A) Don’t print stats on some rows

Syntax: [-DO_STATISTICS]  VERY GOOD

B) Only print stats when sufficient base

Minimum baseSyntax: EDIT={: MINIMUM_BASE=100 }Flag_minimum_baseSyntax: EDIT={: FLAG_MINIMUM_BASE }

Syntax: EDIT={: SEPARATE_VARIANCE }

A. Error Messages in C-Mentor Manual 7.10.3 Error And Warning Messages
B. Statistical Formulas in C-Mentor Manual Appendix A
C. Statistics Dump

Syntax: ~SET STATISTICS_DUMP

Files Included: