QC Comparison of WWC effect size code using R and SAS

I wrote this entry when I wanted to QC my WWC effect size calculation using R and SAS.

 

WWC standard doc

https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v2_1_standards_handbook.pdf

Page 37.

 

In SAS:

data test;

N_Yes=4300;
Mean_Yes=400;
StdDev_Yes=200;

N_No=4000;
Mean_NO=300;
StdDev_NO=200;

/*create statistics*/
mean_dif=(Mean_Yes-Mean_NO);
/*Standardized effects*/

g1=((N_Yes-1)*(StdDev_Yes*StdDev_Yes)) +((N_No-1)*(StdDev_No*StdDev_No));
g2=N_Yes + N_No -2;
g3=sqrt(g1/g2);
WWC_effect=mean_dif/g3;

run;

 

In R:

FGC_l_N_YES=4300
FGC_l_Mean_YES=400
FGC_l_StdDev_YES=200

REG_l_N_YES=4000
REG_l_Mean_YES=300
REG_l_StdDev_YES=200

simple_gap=FGC_l_Mean_YES-REG_l_Mean_YES

g1<- ((FGC_l_N_YES-1)*(FGC_l_StdDev_YES*FGC_l_StdDev_YES))+((REG_l_N_YES-1)*(REG_l_StdDev_YES*REG_l_StdDev_YES))
g2= FGC_l_N_YES + REG_l_N_YES -2
g3= sqrt(g1/g2)
simple_gap_std= simple_gap/g3
simple_gap_std

g1
g2
g3
simple_gap_std

 

My web calculator

https://www.estat.us/file/calc_t_test1b.php

 

Treatment N:4300
Treatment mean:400
Treatment SD:200

Comparison N:4000
Comparison mean:300
Comparison SD:200
The group mean difference:100

[RESULTS FOR CONTINUOUS OUTCOME]

Probability: Under Development (Still working on this)
T-score is: 22.761
Significant at alpha 0.05 (two tail test;I used a z-test and ignored degree freedom; threshold 1.96)

T-test (the same test as above but with three thresholds)T 1.96, 2.576, 3.291, each for p=0.05, 0.01, 0.001
Sig at p=.001***

Hedges d 0.5

SAS t-test for proportions

data kaz;

N_yes=1162;
N_no=381;

mean_yes=0.129088;
mean_no=0.170604;

DEG_FD=N_Yes + N_No -2;
/*QC’ed
tValue=2.228;
DEG_FD=10;
*/

tValue=(Mean_NO-Mean_Yes)/(SQRT((Mean_NO*(1-Mean_NO) /
N_No )+Mean_Yes*(1-Mean_Yes)/ N_Yes));

/*2 tail test*/
P_value=(1-probt(abs(tValue),DEG_FD))*2;
run;

<?php

function compute()
{

$roundunit=3;

$Tmean = $_POST['Tmean'];
$Cmean = $_POST['Cmean'];
$TSD = $_POST['TSD'];
$CSD = $_POST['CSD'];
$TN = $_POST['TN'];
$CN = $_POST['CN'];

$mean_dif=$Tmean-$Cmean;
$SE=sqrt(
(($TSD*$TSD) / $TN)+(($CSD*$CSD) / $CN)
);

$T=$mean_dif/$SE;
$DF=$TN+$CN-2;

$P="Under Development (Still working on this)";

/*I will get T for binary variable comparison*/
$N_SUCCESS_T=$TN*$Tmean;
$N_SUCCESS_C=$CN*$Cmean;
$P_=($N_SUCCESS_T+$N_SUCCESS_C)/($TN+$CN);
$Z_numerator=$Tmean-$Cmean-0;
$Z_denom=SQRT(($P_*(1-$P_))*((1/$TN)+(1/$CN)));
$Z_bin=abs($Z_numerator/$Z_denom);
$Z_bin_abs=abs($Z_numerator/$Z_denom);

/*$P=stats_dens_normal($T, 0,1);*/
/*$P=stats_dens_gamma(float $X, float $shape, float $scale);*/
/*$P= $T / 100 ;*/

/*Hedges g*/
/*g numerator*/
$g_numerator=($Tmean-$Cmean)*(1-3/((4*($TN+$CN))-9));
/*g demnominator*/
$g_denominator=SQRT(((($TN-1)* ($TSD**2) )+(($CN-1)* ($CSD**2) ))/($TN+$CN-2));
$hedges_d=$g_numerator/$g_denominator;
$hedges_d_abs=abs($hedges_d);

/*if binary variabels*/
$T_Odds=$Tmean/(1-$Tmean);
$C_Odds=$Cmean/(1-$Cmean);
$Odds_ratio=$T_Odds/$C_Odds;
$Tstep1=log($T_Odds);
$Cstep1=log($C_Odds);
$step2=$Tstep1-$Cstep1;
$WWC_binary_effect=$step2/1.65;

/*
if ($hedges_d >= 0.2) echo "Small Effect (Cohen)";
if ($hedges_d >= 0.5) echo "Medium Effect (Cohen)";
if ($hedges_d >= 0.8) echo "Large Effect (Cohen)";
*/

echo "<br>";
echo "WWC group comparison of continuous and binary variables";
echo "<br>";
echo "<br>";

 

echo "Treatment N:" .$TN;
echo "<br>";

echo "Treatment mean:" .$Tmean;
echo "<br>";
echo "Treatment SD:" .$TSD;

echo "<br>";
echo "<br>";

echo "Comparison N:" .$CN;
echo "<br>";
echo "Comparison mean:" .$Cmean;
echo "<br>";
echo "Comparison SD:" .$CSD;

echo "<br>";
echo "The group mean difference:".round($mean_dif,$roundunit);
echo "<br>";
echo "<br>";

echo "[RESULTS FOR CONTINUOUS OUTCOME]";
echo "<br><br>";
/*echo "Probability " .round($P,2);*/
echo "Probability: " .$P;
echo "<br>";

$abs_T=abs($T);

echo "T-score is: " .round($T,$roundunit);
echo "<br>";

if($abs_T < 1.96 ) {
echo "Not significant at alpha 0.05 (two tail test;I used a z-test and ignored degree of freedom; threshold 1.96)";
}elseif($abs_T >=1.96){
echo "Significant at alpha 0.05 (two tail test;I used a z-test and ignored degree freedom; threshold 1.96)";
}
echo "<br>";
echo "<br>";
echo "T-test (the same test as above but with three thresholds)";
echo "T 1.96, 2.576, 3.291, each for p=0.05, 0.01, 0.001";
echo "<br>";
if($abs_T < 1.96) {
echo "Not sig. at alpha 0.05";
}elseif($abs_T>=1.960 and $abs_T < 2.576 ){
echo "Sig at p=.05*";

}elseif($abs_T>=2.576 and $abs_T < 3.291 ){
echo "Sig at p=.01**";

}elseif($abs_T>=3.291 ){
echo "Sig at p=.001***";
}else {
echo "N/A";
}

 

 

 

 

echo "<br>";
echo "<br>";

echo "Hedges d " .round($hedges_d,$roundunit);

echo "<br>";
/*cohen's rule of thumb*/
echo "Cohen's rule of thumb for effect size interpretation";
echo "<br>";
if($hedges_d_abs < 0.2) {
echo "Close to zero and Not even small Effect (Cohen)";
}elseif($hedges_d_abs>=0.2 and $hedges_d_abs < 0.5){
echo "Small effect (Cohen)";
}elseif($hedges_d_abs>=0.5 and $hedges_d_abs < 0.8){
echo "Medium effect (Cohen)";
}elseif($hedges_d_abs>=0.8){
echo "Large effect (Cohen)";
}else {
echo "others";
}
echo "<br>";

echo "Baseline equivalence test";
echo "<br>";
if($hedges_d_abs <= 0.05) {
echo "Satisfies the baseline equivalence requirement";
}elseif($hedges_d_abs>0.05 and $hedges_d_abs <= 0.25){
echo "Requires statistical adjustment to satisfy the baseline equivalence requirement";
}elseif($hedges_d_abs>0.25){
echo "Does not satisfy the baseline equivalence requirement";
}elseif($hedges_d_abs>=10){
echo "Something Strange happened";
}else {
echo "N/A";
}

echo "<br>";
echo "<br>";

echo "[RESULTS FOR BINARY OUTCOME]";

echo "<br>";
echo "If outcomes were binary variables (range 0 to 1), the WWC effect size would be ";
echo "" .round($WWC_binary_effect,$roundunit);

echo "<br>";
echo "T-score for the binary outcome is: " .round($Z_bin,$roundunit);
echo "<br>";

echo "<br>";
echo "T-test for binary outcomes";
echo "<br>";
if($Z_bin_abs < 1.96) {
echo "Not sig. at alpha 0.05";
}elseif($Z_bin_abs>=1.960 and $Z_bin_abs < 2.576 ){
echo "Sig at p=.05*";

}elseif($Z_bin_abs>=2.576 and $Z_bin_abs < 3.291 ){
echo "Sig at p=.01**";

}elseif($Z_bin_abs>=3.291 ){
echo "Sig at p=.001***";
}else {
echo "N/A";
}

echo "<br>";
echo "<br>";

}

 

/*echo "The result is: " . compute();*/
compute();

?>

 

<br>

REFERENCE

<br>
Cohen's rule of thumb about effect sizes:
<br>
<li>If greater than 02, Small Effect
<br>
<li>If greater than 0.5, Medium Effect
<br>
<li>If greater 0.8 then Large Effect
<br>
Cohen, J. Statistical power for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum (1988).
<br>
<a href="https://wmich.edu/sites/default/files/attachments/u58/2015/Effect_Size_Substantive_Interpretation_Guidelines.pdf">
Effect Size Substantive Interpretation Guidelines: Issues in the Interpretation of Effect Sizes Jeff Valentine and Harris Cooper, Duke University(see page. 5)</a>
<br>
<br>
WWC related info:
<br>
<a href="https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_procedures_handbook_v4_draft.pdf">WWC procedures handbook (see page. 14)</a>
<br>
<a href="https://ies.ed.gov/ncee/wwc/Docs/OnlineTraining/wwc_training_m3.pdf">WWC standards slides (Definition of small sample size correction, slide 14)</a>
<br>
WWC considers the effect size greater than .25 substnatively important.
<a href="https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_handbook_v4.pdf">P.22 of WWC standards</a>

<br>
T-Table
<br>
<a href="https://www.sjsu.edu/faculty/gerstman/StatPrimer/t-table.pdf">T-Table</a>

<br>
<a href="calc_t_test1.php">Back to the calculcator </a>
<br>

<a href="https://www.estat.us">My website</a>
<br>
<br>

i Bond

 

https://www.treasurydirect.gov/indiv/products/prod_ibonds_glance.htm