Compute single common metric mean, difference between groups, and/or change over time.

metric_mean computes the mean common metric score at a single point in time. metric_growth computes the mean change in the common metric score between two points in time. Both functions can disaggregate results based on a group characteristic used for equity comparisons. They can also account for metrics where multiple data points come from the same classroom, like those based on student surveys or assignments.

Usage

metric_mean(
  data,
  metric,
  use_binary = F,
  equity_group = NULL,
  by_class = F,
  scaleusewarning = T
)

metric_growth(
  data1,
  data2,
  metric,
  use_binary = T,
  equity_group = NULL,
  by_class = F,
  scaleusewarning = T
)

Arguments

data: Data from a single timepoint. Used in metric_mean.
metric: Quoted name of the common metric. Options are "engagement", "belonging", "relevance", "assignments", "tntpcore", or "ipg".
use_binary: A logical (T/F) option to use the binary version of the metric. The default is FALSE so that the mean and growth calculations are based on the overall metric value. If you want these calculations done on the binary version (e.g., looking at the percent of teachers with 'high expectations' rather than the average expectations score) then set this option to TRUE. Note that the metric tntpcore has no binary version.
equity_group: Optional quoted name of the categorical column/variable in data that contains the equity group designation. For example, if data has an indicator variable called class_frl that is either "Under 50 FRL", then analyst could set equity_group = "class_frl" to get differences in metric between these two groups. Default is no equity comparison.
by_class: A logical (T/F) option indicating if multiple rows of data come from the same class. When by_class = T, analysis will automatically account for different sample sizes between classes and adjust the standard errors to account for the lack of independence between data deriving from the same class. If set to FALSE, data must have a variable titled class_id. Default if FALSE.
scaleusewarning: A logical (T/F) indicating whether function should generate a warning when not all values of a scale are used. For example, student survey data that only contains values of 1s and 2s could mean that data is on a 1-4 scale, when it should be on a 0-3 scale. When scaleusewarning = T, the function will warn you of this. This warning does not mean your data is wrong. For example, the Academic Ownership domain from TNTP CORE has 5 potential values: 1, 2, 3, 4, or 5. It's not uncommon to have data where teachers were never rated above a 4 on this domain. In this case, the printed warning can be ignored. Default if TRUE. If you are confident your data is on the right scale, you can suppress the warning by setting to TRUE.
data1: Data from the initial timepoint. Used in metric_growth.
data2: Data from the final timepoint. Used in metric_growth.

Value

A list of results including the overall mean or mean by equity group (for metric_mean), the mean change over time or mean change for each group (for metric_growth). Means are accompanied by standard errors and 95 intervals. Also included are list elements for number of data points used in analysis.

Data and Variable Format

metric_mean and metric_growth should be used with the raw metric data. Each row of data should represent a single rated outcome. For example, each row of data will be a single completed survey, a single rated assignment, a single classroom observation, etc. The data should not have the metric already calculated but instead have the components needed to make this calculation. For example, data on student engagement should not have a column or variable titled engagement, but should have variables corresponding the four survey questions used to calculate engagement. Leave all items in their raw form - the functions automatically account for items that need to be reverse coded. The only requirement is that the data contains the needed variables and that the variables are numeric (i.e., data values should be 0s and 1s, not 'No' and 'Yes'. This ensures that the common metrics are calculated correctly and consistently across projects. Each metric has its own set of needed variables that must be spelled exactly as shown below. They are:

engagement:: eng_like, eng_losttrack, eng_interest, eng_moreabout
belonging:: tch_problem, bel_ideas, bel_fitin, tch_interestedideas
relevance:: rel_asmuch, rel_future, rel_outside, rel_rightnow
expectations:: exp_fairtomaster, exp_oneyearenough, exp_allstudents, exp_appropriate
expectations_old:: exp_allstudents, exp_toochallenging, exp_oneyear, exp_different, exp_overburden, exp_began
tntpcore:: ec, ao, dl, cl
ipg:: form, grade_level, ca1_a, ca1_b, ca1_c, ca2_overall, ca3_overall, col. K-5 Literacy observations must also have rfs_overall. Science observations must also have ca1_d, ca1_e, ca1_f, and science_filter
assignments:: content, relevance, practice

Note that these are the NAMES of the variables needed in your data. It can be okay if some of these variables have NA values for specific rows. For example, K-5 Literacy observations on the IPG require either all of the Core Actions (ca1_a, ca1_b, ca1_c, ca2_overall, ca3_overall) and/or rfs_overall. If an observation has all the core actions it still needs a variable called rfs_overall, but the value can just be NA. See the vignette("analyzing_metrics") for more details. Note on Expectations. The items used to measure expectations shifted from a collection of six, mostly reverse-coded worded items to four positively worded items. Both expectations metrics are available, with the current 4-item expectations metric known as "expectations" and the older 6-item expectations metric known as "expectations_old". See the vignette("analyzing_metrics") for more details.

Examples

# Compute the mean engagement score for an entire project at a single time point. Setting
# by_class = TRUE because multiple surveys come from the same class.
metric_mean(ss_data_final, metric = "engagement", by_class = TRUE)
#> [1] "2 Row(s) in data were NOT used because missing at least one value needed to create common measure."
#> $`Overall mean`
#>  1       emmean    SE df lower.CL upper.CL
#>  overall   5.99 0.458 25     5.05     6.93
#> 
#> Degrees-of-freedom method: satterthwaite 
#> Confidence level used: 0.95 
#> 
#> $`Number of data points`
#> [1] 998
#> 
#> $`Number of included classes`
#> [1] 26
#> 

# Do the same, but now compare results by a class's FRL population
metric_mean(ss_data_final, metric = "engagement", equity_group = "class_frl_cat", by_class = TRUE)
#> [1] "2 Row(s) in data were NOT used because missing at least one value needed to create common measure."
#> $`Group means`
#>  equity_group     emmean    SE df lower.CL upper.CL
#>  At least 50% FRL   4.02 0.335 24     3.33     4.71
#>  Under 50% FRL      7.96 0.335 24     7.27     8.65
#> 
#> Degrees-of-freedom method: satterthwaite 
#> Confidence level used: 0.95 
#> 
#> $`Difference(s) between groups`
#>  contrast                         estimate    SE df t.ratio p.value
#>  At least 50% FRL - Under 50% FRL    -3.95 0.473 24  -8.333  <.0001
#> 
#> Degrees-of-freedom method: satterthwaite 
#> 
#> $`Number of data points`
#> [1] 998
#> 
#> $`Number of included classes`
#> [1] 26
#> 

# Look at change in engagement over time, then look at how differences in engagement between a
# class's FRL population change over time
metric_growth(
  ss_data_initial,
  ss_data_final,
  metric = "engagement",
  by_class = TRUE
 )
#> [1] "6 Row(s) in data were NOT used because missing at least one value needed to create common measure."
#> [1] "2 Row(s) in data were NOT used because missing at least one value needed to create common measure."
#> $`Means at each timepoint`
#>  time    emmean     SE   df lower.CL upper.CL
#>  Final    0.326 0.0743 25.3   0.1731    0.479
#>  Initial  0.234 0.0743 25.3   0.0809    0.387
#> 
#> Degrees-of-freedom method: satterthwaite 
#> Confidence level used: 0.95 
#> 
#> $`Differences between timepoints`
#>  contrast        estimate     SE   df t.ratio p.value
#>  Final - Initial   0.0922 0.0115 1965   8.035  <.0001
#> 
#> Degrees-of-freedom method: satterthwaite 
#> 
#> $`Number of data points`
#> [1] 1992
#> 
#> $`Number of included classes`
#> [1] 26
#> 
 metric_growth(
  ss_data_initial,
  ss_data_final,
  metric = "engagement",
  equity_group = "class_frl_cat",
  by_class = TRUE
 )
#> [1] "6 Row(s) in data were NOT used because missing at least one value needed to create common measure."
#> [1] "2 Row(s) in data were NOT used because missing at least one value needed to create common measure."
#> $`Group means at each timepoint`
#> time = Initial:
#>  equity_group      emmean     SE   df lower.CL upper.CL
#>  At least 50% FRL 0.00365 0.0714 24.6   -0.143    0.151
#>  Under 50% FRL    0.46356 0.0714 24.6    0.316    0.611
#> 
#> time = Final:
#>  equity_group      emmean     SE   df lower.CL upper.CL
#>  At least 50% FRL 0.00645 0.0714 24.6   -0.141    0.154
#>  Under 50% FRL    0.64546 0.0714 24.6    0.498    0.793
#> 
#> Degrees-of-freedom method: satterthwaite 
#> Confidence level used: 0.95 
#> 
#> $`Differences between groups at each timepoint`
#> time = Initial:
#>  contrast                         estimate    SE   df t.ratio p.value
#>  At least 50% FRL - Under 50% FRL   -0.460 0.101 24.6  -4.557  0.0001
#> 
#> time = Final:
#>  contrast                         estimate    SE   df t.ratio p.value
#>  At least 50% FRL - Under 50% FRL   -0.639 0.101 24.6  -6.332  <.0001
#> 
#> Degrees-of-freedom method: satterthwaite 
#> 
#> $`Change in differences between groups over time`
#> contrast = At least 50% FRL - Under 50% FRL:
#>  contrast1       estimate     SE   df t.ratio p.value
#>  Final - Initial   -0.179 0.0226 1965  -7.923  <.0001
#> 
#> Degrees-of-freedom method: satterthwaite 
#> 
#> $`Number of data points`
#> [1] 1992
#> 
#> $`Number of included classes`
#> [1] 26
#>