Posts

I wrote this entry when I wanted to QC my WWC effect size calculation using R and SAS.

WWC standard doc

https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v2_1_standards_handbook.pdf

Page 37.

In SAS:

data test;

N_Yes=4300;
Mean_Yes=400;
StdDev_Yes=200;

N_No=4000;
Mean_NO=300;
StdDev_NO=200;

/*create statistics*/
mean_dif=(Mean_Yes-Mean_NO);
/*Standardized effects*/

g1=((N_Yes-1)*(StdDev_Yes*StdDev_Yes)) +((N_No-1)*(StdDev_No*StdDev_No));
g2=N_Yes + N_No -2;
g3=sqrt(g1/g2);
WWC_effect=mean_dif/g3;

run;

In R:

FGC_l_N_YES=4300
FGC_l_Mean_YES=400
FGC_l_StdDev_YES=200

REG_l_N_YES=4000
REG_l_Mean_YES=300
REG_l_StdDev_YES=200

simple_gap=FGC_l_Mean_YES-REG_l_Mean_YES

g1<- ((FGC_l_N_YES-1)*(FGC_l_StdDev_YES*FGC_l_StdDev_YES))+((REG_l_N_YES-1)*(REG_l_StdDev_YES*REG_l_StdDev_YES))
g2= FGC_l_N_YES + REG_l_N_YES -2
g3= sqrt(g1/g2)
simple_gap_std= simple_gap/g3
simple_gap_std

g1
g2
g3
simple_gap_std

My web calculator

https://www.estat.us/file/calc_t_test1b.php

Treatment N:4300
Treatment mean:400
Treatment SD:200

Comparison N:4000
Comparison mean:300
Comparison SD:200
The group mean difference:100

[RESULTS FOR CONTINUOUS OUTCOME]

Probability: Under Development (Still working on this)
T-score is: 22.761
Significant at alpha 0.05 (two tail test;I used a z-test and ignored degree freedom; threshold 1.96)

T-test (the same test as above but with three thresholds)T 1.96, 2.576, 3.291, each for p=0.05, 0.01, 0.001
Sig at p=.001***

Hedges d 0.5

<?php

function compute()
{

$roundunit=3;

$Tmean = $_POST['Tmean'];
$Cmean = $_POST['Cmean'];
$TSD = $_POST['TSD'];
$CSD = $_POST['CSD'];
$TN = $_POST['TN'];
$CN = $_POST['CN'];

$mean_dif=$Tmean-$Cmean;
$SE=sqrt(
(($TSD*$TSD) / $TN)+(($CSD*$CSD) / $CN)
);

$T=$mean_dif/$SE;
$DF=$TN+$CN-2;

$P="Under Development (Still working on this)";

/*I will get T for binary variable comparison*/
$N_SUCCESS_T=$TN*$Tmean;
$N_SUCCESS_C=$CN*$Cmean;
$P_=($N_SUCCESS_T+$N_SUCCESS_C)/($TN+$CN);
$Z_numerator=$Tmean-$Cmean-0;
$Z_denom=SQRT(($P_*(1-$P_))*((1/$TN)+(1/$CN)));
$Z_bin=abs($Z_numerator/$Z_denom);
$Z_bin_abs=abs($Z_numerator/$Z_denom);

/*$P=stats_dens_normal($T, 0,1);*/
/*$P=stats_dens_gamma(float $X, float $shape, float $scale);*/
/*$P= $T / 100 ;*/

/*Hedges g*/
/*g numerator*/
$g_numerator=($Tmean-$Cmean)*(1-3/((4*($TN+$CN))-9));
/*g demnominator*/
$g_denominator=SQRT(((($TN-1)* ($TSD**2) )+(($CN-1)* ($CSD**2) ))/($TN+$CN-2));
$hedges_d=$g_numerator/$g_denominator;
$hedges_d_abs=abs($hedges_d);

/*if binary variabels*/
$T_Odds=$Tmean/(1-$Tmean);
$C_Odds=$Cmean/(1-$Cmean);
$Odds_ratio=$T_Odds/$C_Odds;
$Tstep1=log($T_Odds);
$Cstep1=log($C_Odds);
$step2=$Tstep1-$Cstep1;
$WWC_binary_effect=$step2/1.65;

/*
if ($hedges_d >= 0.2) echo "Small Effect (Cohen)";
if ($hedges_d >= 0.5) echo "Medium Effect (Cohen)";
if ($hedges_d >= 0.8) echo "Large Effect (Cohen)";
*/

echo " ";
echo "WWC group comparison of continuous and binary variables";
echo " ";
echo " ";

echo "Treatment N:" .$TN;
echo " ";

echo "Treatment mean:" .$Tmean;
echo " ";
echo "Treatment SD:" .$TSD;

echo " ";
echo " ";

echo "Comparison N:" .$CN;
echo " ";
echo "Comparison mean:" .$Cmean;
echo " ";
echo "Comparison SD:" .$CSD;

echo " ";
echo "The group mean difference:".round($mean_dif,$roundunit);
echo " ";
echo " ";

echo "[RESULTS FOR CONTINUOUS OUTCOME]";
echo " ";
/*echo "Probability " .round($P,2);*/
echo "Probability: " .$P;
echo " ";

$abs_T=abs($T);

echo "T-score is: " .round($T,$roundunit);
echo " ";

if($abs_T < 1.96 ) {
echo "Not significant at alpha 0.05 (two tail test;I used a z-test and ignored degree of freedom; threshold 1.96)";
}elseif($abs_T >=1.96){
echo "Significant at alpha 0.05 (two tail test;I used a z-test and ignored degree freedom; threshold 1.96)";
}
echo " ";
echo " ";
echo "T-test (the same test as above but with three thresholds)";
echo "T 1.96, 2.576, 3.291, each for p=0.05, 0.01, 0.001";
echo " ";
if($abs_T < 1.96) {
echo "Not sig. at alpha 0.05";
}elseif($abs_T>=1.960 and $abs_T < 2.576 ){
echo "Sig at p=.05*";

}elseif($abs_T>=2.576 and $abs_T < 3.291 ){
echo "Sig at p=.01**";

}elseif($abs_T>=3.291 ){
echo "Sig at p=.001***";
}else {
echo "N/A";
}

echo " ";
echo " ";

echo "Hedges d " .round($hedges_d,$roundunit);

echo " ";
/*cohen's rule of thumb*/
echo "Cohen's rule of thumb for effect size interpretation";
echo " ";
if($hedges_d_abs < 0.2) {
echo "Close to zero and Not even small Effect (Cohen)";
}elseif($hedges_d_abs>=0.2 and $hedges_d_abs < 0.5){
echo "Small effect (Cohen)";
}elseif($hedges_d_abs>=0.5 and $hedges_d_abs < 0.8){
echo "Medium effect (Cohen)";
}elseif($hedges_d_abs>=0.8){
echo "Large effect (Cohen)";
}else {
echo "others";
}
echo " ";

echo "Baseline equivalence test";
echo " ";
if($hedges_d_abs <= 0.05) {
echo "Satisfies the baseline equivalence requirement";
}elseif($hedges_d_abs>0.05 and $hedges_d_abs <= 0.25){
echo "Requires statistical adjustment to satisfy the baseline equivalence requirement";
}elseif($hedges_d_abs>0.25){
echo "Does not satisfy the baseline equivalence requirement";
}elseif($hedges_d_abs>=10){
echo "Something Strange happened";
}else {
echo "N/A";
}

echo " ";
echo " ";

echo "[RESULTS FOR BINARY OUTCOME]";

echo " ";
echo "If outcomes were binary variables (range 0 to 1), the WWC effect size would be ";
echo "" .round($WWC_binary_effect,$roundunit);

echo " ";
echo "T-score for the binary outcome is: " .round($Z_bin,$roundunit);
echo " ";

echo " ";
echo "T-test for binary outcomes";
echo " ";
if($Z_bin_abs < 1.96) {
echo "Not sig. at alpha 0.05";
}elseif($Z_bin_abs>=1.960 and $Z_bin_abs < 2.576 ){
echo "Sig at p=.05*";

}elseif($Z_bin_abs>=2.576 and $Z_bin_abs < 3.291 ){
echo "Sig at p=.01**";

}elseif($Z_bin_abs>=3.291 ){
echo "Sig at p=.001***";
}else {
echo "N/A";
}

echo " ";
echo " ";

}

/*echo "The result is: " . compute();*/
compute();

REFERENCE

Cohen's rule of thumb about effect sizes:
 
<li>If greater than 02, Small Effect
 
<li>If greater than 0.5, Medium Effect
 
<li>If greater 0.8 then Large Effect
 
Cohen, J. Statistical power for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum (1988).
 
<a href="https://wmich.edu/sites/default/files/attachments/u58/2015/Effect_Size_Substantive_Interpretation_Guidelines.pdf">
Effect Size Substantive Interpretation Guidelines: Issues in the Interpretation of Effect Sizes Jeff Valentine and Harris Cooper, Duke University(see page. 5)</a>
 
 
WWC related info:
 
<a href="https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_procedures_handbook_v4_draft.pdf">WWC procedures handbook (see page. 14)</a>
 
<a href="https://ies.ed.gov/ncee/wwc/Docs/OnlineTraining/wwc_training_m3.pdf">WWC standards slides (Definition of small sample size correction, slide 14)</a>
 
WWC considers the effect size greater than .25 substnatively important.
<a href="https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_handbook_v4.pdf">P.22 of WWC standards</a>

T-Table
 
<a href="https://www.sjsu.edu/faculty/gerstman/StatPrimer/t-table.pdf">T-Table</a>

<a href="calc_t_test1.php">Back to the calculcator </a>

<a href="https://www.estat.us">My website</a>

line graph that also shows the difference of two lines

T-test using R

QC Comparison of WWC effect size code using R and SAS

SAS t-test for proportions

How to generate data from a model (using R)

When the laptop's fan start making a loud noise

Power analysis and parameters

Survival Analysis in R (in under 10-minutes)

i Bond