Power Analysis in a SMART Design: Sample Size Estimation for Determining the Best Embedded Dynamic Treatment Regime (EDTR)

Sample size calculator for sequential, multiple assignment, randomized trials (SMART).

Guidelines for choosing the covariance matrix

These guidelines are based off Section 7 of Artman and others (2018).

The diagonal elements of the covariance matrix are the variances (square of the standard deviations) of the EDTRs. The off‐diagonals are the covariances (product of the correlation and the standard deviations of each of the pairs of EDTRs corresponding to the row and column, respectively). The lower triangle (covariances below diagonal) and upper triangle (covariances above diagonal) are equal as covariance matrices are symmetric. The covariance matrix is the covariance for \(n=1\) subjects and it must be positive-definite to be valid.
The power tends to be a monotone decreasing function of variances and a monotone increasing function of the correlations (or equivalently covariances). The variances should be chosen based off prior knowledge or for a conservative estimate of sample size, the largest plausible variance for each of the EDTRs should be chosen. Similarly, the smallest plausible covariances should be chosen. In the absence of information about the correlation between EDTR outcomes, it is reasonable to assume all correlations are equal. The correlation between the best and non-best EDTRs has a greater impact on power than the correlation between two non-best EDTRs. Therefore, the common correlation should correspond to that between the best and second best EDTRs. If it is implausible for the correlation between two EDTRs to be negative, a conservative estimate of the covariance would be obtained by setting the covariances equal to zero. For an exchangeable covariance matrix (one with equal variances and equal covariances), the minimum correlation such that the matrix will be positive-definite (permissible) is bounded by \(-1/(N-1)\) where N is the number of EDTRs. A conservative choice for the covariance would then be the variance times \(-1/(N-1)\) plus a small positive number. The effect on sample size (or power) of a specific covariance matrix may be explored by inputting different values for the variances and covariances.

As an alternative to sizing SMARTs based off a conservative covariance matrix, we propose conducting a pilot SMART to estimate the correlations in the covariance matrix. For more details, see Artman and others (2018).

Notes

The effect sizes are not Cohen's d since Cohen's d is not applicable to comparing correlated EDTR outcomes. They are the unstandardized differences between each mean EDTR outcome and the best mean EDTR outcome. The Type I error rate \(\alpha\) is the probability of excluding the true best EDTR as inferior. The minimum detectable difference \(\Delta_{\mathrm{min}}\) is the number such that with power \(1-\beta\), all EDTRs with effect size greater than or equal to \(\Delta_{\mathrm{min}}\) are excluded as inferior. The power and sample size are calculated using Monte Carlo simulation.

Advantages of our method

Our method is built on multiple comparisons with the best (MCB) (Hsu, 1981, 1984, 1996; Ertefaie, 2015). MCB entails comparing each EDTR with the best EDTR by constructing \(N-1\) simultaneous \(100(1-\alpha)\%\) one-sided confidence intervals (CI) for the difference between each EDTR and the best EDTR. If the CI excludes 0, then the corresponding EDTR is considered statistically significantly inferior to the best EDTR. Our method sizes a SMART so that EDTRs which are clinically meaningfully inferior to the best EDTR (determined by \(\Delta_{\mathrm{min}}\)) are excluded from the set of best EDTRs with power (probability) \(1-\beta\). Regardless of sample size, using MCB includes the best EDTR with probability \(1-\alpha\). Our method is applicable to an arbitrary SMART design.
Alternative methods size SMARTs so that the best EDTR has the largest sample estimate (Oetting and others, 2011). This approach based on estimation alone and not inference fails to account for the fact that some EDTRs may be statistically indistinguishable from the true best EDTR for the given data in which case they should not necessarily be excluded as suboptimal. Our approach goes one step further by providing a means to size SMARTs in order to construct narrow confidence intervals which not only tell which is the best EDTR, but also provide the ability to screen out inferior EDTRs. Other approaches size SMARTs for all pairwise comparisons using the Bonferroni correction (Ogbagaber and others, 2016) and for testing the hypothesis about only two particular EDTRs (Oetting and others, 2011). MCB involves only \(N-1\) comparisons while all pairwise comparisons would be \(\binom{N}{2}\), so MCB yields greater power. Furthermore, methods that size a SMART for testing the difference between only two EDTRs fail to fully address the multiple testing problem introduced by searching for optimal EDTR entailing greater than two comparisons.
This Shiny app is built on the smartsizer R package: https://CRAN.R-project.org/package=smartsizer.
If you use this tool, please cite Artman and others (2018).
For more details about our method, see Artman and others (2018).

Artman, W. J., Nahum-Shani, I., Wu, T., Mckay, J. R., & Ertefaie, A. (2018). Power analysis in a SMART design: sample size estimation for determining the best embedded dynamic treatment regime. Biostatistics.

Ertefaie, A., Wu, T., Lynch, K. G. and Nahum-Shani, I. (2015). Identifying a set that contains the best dynamic treatment regimes. Biostatistics 17, 135‐148.

Hsu, J. C. (1981). Simultaneous confidence intervals for all distances from the ''best''. The Annals of Statistics 9, 1026‐1034.

Hsu, J. C. (1984). Constrained simultaneous confidence intervals for multiple comparisons with the best. Annals of Statistics 12, 1136‐1144.

Hsu, J. C. (1996). Multiple Comparisons: Theory and Methods. London: CRC Press.

Oetting, A., Levy, J., Weiss, R., & Murphy, S. (2011). Statistical methodology for a smart design in the development of adaptive treatment strategies. Causality and psychopathology: Finding the determinants of disorders and their cures (pp. 179‐205). Arlington, VA: Oxford University Press.

Ogbagaber, S. B., Karp, J. and Wahed, A. S., (2016). Design of sequentially randomized trials for testing adaptive treatment strategies. Statistics in medicine, 35, pp.840‐858.

Power Analysis in a SMART Design: Sample Size Estimation for Determining the Best Embedded Dynamic Treatment Regime (EDTR)

Power calculator for sequential, multiple assignment, randomized trials (SMART).

Guidelines for choosing the covariance matrix

These guidelines are based off Section 7 of Artman and others (2018).

The diagonal elements of the covariance matrix are the variances (square of the standard deviations) of the EDTRs. The off‐diagonals are the covariances (product of the correlation and the standard deviations of each of the pairs of EDTRs corresponding to the row and column, respectively). The lower triangle (covariances below diagonal) and upper triangle (covariances above diagonal) are equal as covariance matrices are symmetric. The covariance matrix is the covariance for \(n=1\) subjects and it must be positive-definite to be valid.
The power tends to be a monotone decreasing function of variances and a monotone increasing function of the correlations (or equivalently covariances). The variances should be chosen based off prior knowledge or for a conservative estimate of sample size, the largest plausible variance for each of the EDTRs should be chosen. Similarly, the smallest plausible covariances should be chosen. In the absence of information about the correlation between EDTR outcomes, it is reasonable to assume all correlations are equal. The correlation between the best and non-best EDTRs has a greater impact on power than the correlation between two non-best EDTRs. Therefore, the common correlation should correspond to that between the best and second best EDTRs. If it is implausible for the correlation between two EDTRs to be negative, a conservative estimate of the covariance would be obtained by setting the covariances equal to zero. For an exchangeable covariance matrix (one with equal variances and equal covariances), the minimum correlation such that the matrix will be positive-definite (permissible) is bounded by \(-1/(N-1)\) where N is the number of EDTRs. A conservative choice for the covariance would then be the variance times \(-1/(N-1)\) plus a small positive number. The effect on sample size (or power) of a specific covariance matrix may be explored by inputting different values for the variances and covariances.

As an alternative to sizing SMARTs based off a conservative covariance matrix, we propose conducting a pilot SMART to estimate the correlations in the covariance matrix. For more details, see Artman and others (2018).

Notes

The effect sizes are not Cohen's d since Cohen's d is not applicable to comparing correlated EDTR outcomes. They are the unstandardized differences between each mean EDTR outcome and the best mean EDTR outcome. The Type I error rate \(\alpha\) is the probability of excluding the true best EDTR as inferior. The minimum detectable difference \(\Delta_{\mathrm{min}}\) is the number such that with power \(1-\beta\), all EDTRs with effect size greater than or equal to \(\Delta_{\mathrm{min}}\) are excluded as inferior. The power and sample size are calculated using Monte Carlo simulation.

Advantages of our method

Our method is built on multiple comparisons with the best (MCB) (Hsu, 1981, 1984, 1996; Ertefaie, 2015). MCB entails comparing each EDTR with the best EDTR by constructing \(N-1\) simultaneous \(100(1-\alpha)\%\) one-sided confidence intervals (CI) for the difference between each EDTR and the best EDTR. If the CI excludes 0, then the corresponding EDTR is considered statistically significantly inferior to the best EDTR. Our method sizes a SMART so that EDTRs which are clinically meaningfully inferior to the best EDTR (determined by \(\Delta_{\mathrm{min}}\)) are excluded from the set of best EDTRs with power (probability) \(1-\beta\). Regardless of sample size, using MCB includes the best EDTR with probability \(1-\alpha\). Our method is applicable to an arbitrary SMART design.
Alternative methods size SMARTs so that the best EDTR has the largest sample estimate (Oetting and others, 2011). This approach based on estimation alone and not inference fails to account for the fact that some EDTRs may be statistically indistinguishable from the true best EDTR for the given data in which case they should not necessarily be excluded as suboptimal. Our approach goes one step further by providing a means to size SMARTs in order to construct narrow confidence intervals which not only tell which is the best EDTR, but also provide the ability to screen out inferior EDTRs. Other approaches size SMARTs for all pairwise comparisons using the Bonferroni correction (Ogbagaber and others, 2016) and for testing the hypothesis about only two particular EDTRs (Oetting and others, 2011). MCB involves only \(N-1\) comparisons while all pairwise comparisons would be \(\binom{N}{2}\), so MCB yields greater power. Furthermore, methods that size a SMART for testing the difference between only two EDTRs fail to fully address the multiple testing problem introduced by searching for optimal EDTR entailing greater than two comparisons.
This Shiny app is built on the smartsizer R package: https://CRAN.R-project.org/package=smartsizer.
If you use this tool, please cite Artman and others (2018).
For more details about our method, see Artman and others (2018).

Artman, W. J., Nahum-Shani, I., Wu, T., Mckay, J. R., & Ertefaie, A. (2018). Power analysis in a SMART design: sample size estimation for determining the best embedded dynamic treatment regime. Biostatistics.

Ertefaie, A., Wu, T., Lynch, K. G. and Nahum-Shani, I. (2015). Identifying a set that contains the best dynamic treatment regimes. Biostatistics 17, 135‐148.

Hsu, J. C. (1981). Simultaneous confidence intervals for all distances from the ''best''. The Annals of Statistics 9, 1026‐1034.

Hsu, J. C. (1984). Constrained simultaneous confidence intervals for multiple comparisons with the best. Annals of Statistics 12, 1136‐1144.

Hsu, J. C. (1996). Multiple Comparisons: Theory and Methods. London: CRC Press.

Oetting, A., Levy, J., Weiss, R., & Murphy, S. (2011). Statistical methodology for a smart design in the development of adaptive treatment strategies. Causality and psychopathology: Finding the determinants of disorders and their cures (pp. 179‐205). Arlington, VA: Oxford University Press.

Ogbagaber, S. B., Karp, J. and Wahed, A. S., (2016). Design of sequentially randomized trials for testing adaptive treatment strategies. Statistics in medicine, 35, pp.840‐858.