A reinforcement learning model to inform optimal decision paths for HIV elimination

Seyedeh N. Khatami; Chaitra Gopalappa; Seyedeh N. Khatami; Chaitra Gopalappa

doi:10.3934/mbe.2021380

Mathematical Biosciences and Engineering

2021, Volume 18, Issue 6: 7666-7684. doi: 10.3934/mbe.2021380

Previous Article Next Article

Research article Special Issues

A reinforcement learning model to inform optimal decision paths for HIV elimination

Seyedeh N. Khatami ,
Chaitra Gopalappa ^,

Mechanical and Industrial Engineering Department, University of Massachusetts Amherst, Amherst, MA 01003, USA

Academic Editor: Dobromir Dimitrov

Received: 12 July 2021 Accepted: 25 August 2021 Published: 06 September 2021

The 'Ending the HIV Epidemic (EHE)' national plan aims to reduce annual HIV incidence in the United States from 38,000 in 2015 to 9300 by 2025 and 3300 by 2030. Diagnosis and treatment are two most effective interventions, and thus, identifying corresponding optimal combinations of testing and retention-in-care rates would help inform implementation of relevant programs. Considering the dynamic and stochastic complexity of the disease and the time dynamics of decision-making, solving for optimal combinations using commonly used methods of parametric optimization or exhaustive evaluation of pre-selected options are infeasible. Reinforcement learning (RL), an artificial intelligence method, is ideal; however, training RL algorithms and ensuring convergence to optimality are computationally challenging for large-scale stochastic problems. We evaluate its feasibility in the context of the EHE goal. We trained an RL algorithm to identify a 'sequence' of combinations of HIV-testing and retention-in-care rates at 5-year intervals over 2015-2070 that optimally leads towards HIV elimination. We defined optimality as a sequence that maximizes quality-adjusted-life-years lived and minimizes HIV-testing and care-and-treatment costs. We show that solving for testing and retention-in-care rates through appropriate reformulation using proxy decision-metrics overcomes the computational challenges of RL. We used a stochastic agent-based simulation to train the RL algorithm. As there is variability in support-programs needed to address barriers to care-access, we evaluated the sensitivity of optimal decisions to three cost-functions. The model suggests to scale-up retention-in-care programs to achieve and maintain high annual retention-rates while initiating with a high testing-frequency but relaxing it over a 10-year period as incidence decreases. Results were mainly robust to the uncertainty in costs. However, testing and retention-in-care alone did not achieve the 2030 EHE targets, suggesting the need for additional interventions. The results from the model demonstrated convergence. RL is suitable for evaluating phased public health decisions for infectious disease control.
Citation: Seyedeh N. Khatami, Chaitra Gopalappa. A reinforcement learning model to inform optimal decision paths for HIV elimination[J]. Mathematical Biosciences and Engineering, 2021, 18(6): 7666-7684. doi: 10.3934/mbe.2021380

Related Papers:

Abstract

The 'Ending the HIV Epidemic (EHE)' national plan aims to reduce annual HIV incidence in the United States from 38,000 in 2015 to 9300 by 2025 and 3300 by 2030. Diagnosis and treatment are two most effective interventions, and thus, identifying corresponding optimal combinations of testing and retention-in-care rates would help inform implementation of relevant programs. Considering the dynamic and stochastic complexity of the disease and the time dynamics of decision-making, solving for optimal combinations using commonly used methods of parametric optimization or exhaustive evaluation of pre-selected options are infeasible. Reinforcement learning (RL), an artificial intelligence method, is ideal; however, training RL algorithms and ensuring convergence to optimality are computationally challenging for large-scale stochastic problems. We evaluate its feasibility in the context of the EHE goal. We trained an RL algorithm to identify a 'sequence' of combinations of HIV-testing and retention-in-care rates at 5-year intervals over 2015-2070 that optimally leads towards HIV elimination. We defined optimality as a sequence that maximizes quality-adjusted-life-years lived and minimizes HIV-testing and care-and-treatment costs. We show that solving for testing and retention-in-care rates through appropriate reformulation using proxy decision-metrics overcomes the computational challenges of RL. We used a stochastic agent-based simulation to train the RL algorithm. As there is variability in support-programs needed to address barriers to care-access, we evaluated the sensitivity of optimal decisions to three cost-functions. The model suggests to scale-up retention-in-care programs to achieve and maintain high annual retention-rates while initiating with a high testing-frequency but relaxing it over a 10-year period as incidence decreases. Results were mainly robust to the uncertainty in costs. However, testing and retention-in-care alone did not achieve the 2030 EHE targets, suggesting the need for additional interventions. The results from the model demonstrated convergence. RL is suitable for evaluating phased public health decisions for infectious disease control.

References

[1]	Centers for Disease Control and Prevention, Estimated HIV incidence and prevalence in the United States, 2010-2015, 2020. Available from: https://www.cdc.gov/hiv/pdf/library/reports/surveillance/cdc-hiv-surveillance-supplemental-report-vol-23-1.pdf.
[2]	HIV. gov., What is 'Ending the HIV Epidemic: A Plan for America'?, 2019. Available from: https://www.hiv.gov/federal-response/ending-the-hiv-epidemic/overview.
[3]	America's HIV Epidemic Analysis Dashboard, 2020. Available from: https://ahead.hiv.gov/indicators/incidence/. Published 2020. Accessed December 2020.
[4]	Centers for Disease Control and Prevention, Evidence of HIV Treatment and Viral Suppression in Preventing the Sexual Transmission of HIV, 2020. Available from: https://www.cdc.gov/hiv/pdf/risk/art/cdc-hiv-art-viral-suppression.pdf.
[5]	B. M. Branson, H. H. Handsfield, M. A. Lampe, R. S. Janssen, A. W. Taylor, S. B. Lyss, et al., Revised recommendations for HIV testing of adults, adolescents, and pregnant women in health-care settings, Morb. Mortal Wkly. Rep., 55 (2006), 1-CE.
[6]	A. F. Dailey, B. E. Hoots, H. I. Hall, R. Song, H. Demorah. Vital signs: human immunodeficiency virus testing and diagnosis delays-United States, Morb. Mortal Wkly. Rep., 66 (2017), 1300. doi: 10.15585/mmwr.mm6647e1
[7]	Centers for Disease Control and Prevention, HIV Surveillance Report, 2015. Available from: http://www.cdc.gov/hiv/library/reports/hiv-surveillance.html.
[8]	U.S. Department of Health & Human Services, 2017 National HIV/AIDS Strategy (NHAS) Progress Report Released, 2018. Available from: https://www.hiv.gov/blog/2017-national-hivaids-strategy-nhas-progress-report-released.
[9]	R. K. Shrestha, H. A. Clark, S. L. Sansom, B. Song, H. Buckendahl, C. B. Calhoun, et al., Cost-effectiveness of finding new HIV diagnoses using rapid HIV testing in community-based organizations, Public Health Rep., 123 (2008), 94-100. doi: 10.1177/00333549081230S312
[10]	R. K. Shrestha, L. Gardner, G. Marks, J. Craw, M. Faye, T. P. Giordano, et al., Estimating the cost of increasing retention in care for HIV-infected patients: results of the CDC/HRSA retention in care trial, J. Acquir. Immune Defic. Syndr., 68 (2015), 345. doi: 10.1097/QAI.0000000000000462
[11]	M. L. G. Buot, J. P. Docena, B. K. Ratemo, M. J. Bittner, J. T. Burlew, A. R. Nuritdinov, et al., Beyond race and place: distal sociological determinants of HIV disparities, PloS One, 9 (2014), e91711. doi: 10.1371/journal.pone.0091711
[12]	J. McMahon, C. Wanke, N. Terrin, S. Skinner, T. Knox, Poverty, hunger, education, and residential status impact survival in HIV, AIDS Behav., 15 (2011), 1503-1511. doi: 10.1007/s10461-010-9759-z
[13]	HIV AND AIDS SOCIAL ISSUES, 2016. Available from: https://www.avert.org/professionals/hiv-social-issues.
[14]	M. L. Brandeau, S. Z. Gregory, Optimal investment in HIV prevention programs: more is not always better, Health Care Manag. Sci., 12 (2009), 27. doi: 10.1007/s10729-008-9074-7
[15]	L. Guinness, L. Kumaranayake, K. Hanson, A cost function for HIV prevention services: is there a'u'--shape?, Cost Eff. Resour. Allocation, 5 (2007), 1-12. doi: 10.1186/1478-7547-5-1
[16]	J. A. Pellowski, S. C. Kalichman, K. A. Matthews and N. Adler. A pandemic of the poor: Social disadvantage and the US HIV epidemic, Am. Psychol., 68 (2013), 197. doi: 10.1037/a0032694
[17]	C. Gopalappa, P. G. Farnham, A. B. Hutchinson, S. L. Sansom, Cost effectiveness of the National HIV/AIDS Strategy goal of increasing linkage to care for HIV-infected persons, J. Acquir. Immune Defic. Syndr., 61 (2012), 99-105. doi: 10.1097/QAI.0b013e31825bd862
[18]	F. Lin, P. G. Farnham, R. K. Shrestha, J. Mermin, S. L. Sansom, Cost effectiveness of HIV prevention interventions in the US, Am. J. Prev. Med., 50 (2016), 699-708. doi: 10.1016/j.amepre.2016.01.011
[19]	R. A. Bonacci, D. R. Holtgrave, US HIV incidence and transmission goals, 2020 and 2025, Am. J. Prev. Med., 53 (2017), 275-281. doi: 10.1016/j.amepre.2017.03.012
[20]	A. L. Avancena, D. W. Hutton, Optimization models for HIV/AIDS resource allocation: A systematic review, Value Health, 2020.
[21]	S. Kok, A. R. Rutherford, R. Gustafson, R. Barrios, J. S. Montaner, K. Vasarhelyi, Optimizing an HIV testing program using a system dynamics model of the continuum of care, Health Care Manag. Sci., 18 (2015), 334-362. doi: 10.1007/s10729-014-9312-0
[22]	K. O. Okosun, O. Makinde, I. Takaidza, Impact of optimal control on the treatment of HIV/AIDS and screening of unaware infectives, Appl. Math. Model., 37 (2013), 3802-3820. doi: 10.1016/j.apm.2012.08.004
[23]	L. N. Steimle, D. L. Kaufman, B. T. Denton, Multi-model Markov decision processes: A new method for mitigating parameter ambiguity, Optim. Online, 2018.
[24]	S. M. Shechter, M. D. Bailey, A. J. Schaefer, M. S. Roberts, The optimal time to initiate HIV therapy under ordered health states, Oper. Res., 56 (2008), 20-33. doi: 10.1287/opre.1070.0480
[25]	J. E. Mason, B. T. Denton, N. D. Shah, S. A. Smith, Optimizing the simultaneous management of blood pressure and cholesterol for type 2 diabetes patients, Eur. J. Oper. Res., 233 (2014), 727-738. doi: 10.1016/j.ejor.2013.09.018
[26]	C. Gopalappa, P. G. Farnham, Y. H. Chen, S. L. Sansom, Progression and transmission of HIV/AIDS (PATH 2.0) a new, agent-based model to estimate HIV transmissions in the United States, Med. Decis. Making, 37 (2017), 224-233. doi: 10.1177/0272989X16668509
[27]	C. Yu, Y. Dong, G. Ren, Incorporating causal factors into reinforcement learning for dynamic treatment regimes in HIV, BMC Med. Inf. Decis. Making, 19 (2019), 19-29. doi: 10.1186/s12911-019-0737-8
[28]	S. Parbhoo, A reinforcement learning design for HIV clinical trials, 2014.
[29]	V. Kompella, R. Capobianco, S. Jong, J. Browne, S. Fox, L. Meyers, et al., Reinforcement learning for optimization of COVID-19 mitigation policies, preprint, arXiv: 2010.10560.
[30]	R. Padmanabhan, N. Meskin, T. Khattab, M. Shraim, M. Al-Hitmi, Reinforcement learning-based decision support system for COVID-19, Biomed. Signal Process. Control, (2021), 102676.
[31]	M. I. Uddin, S. A. Ali Shah, M. A. Al-Khasawneh, A. A. Alarood, E. Alsolami, Optimal policy learning for COVID-19 prevention using reinforcement learning, J. Inf. Sci., 2020.
[32]	H. Khadilkar, T. Ganu, D. P. Seetharam, Optimising lockdown policies for epidemic control using reinforcement learning, Trans. Indian Natl. Acad. Eng., 5 (2020), 129-132. doi: 10.1007/s41403-020-00129-3
[33]	R. Wan, X. Zhang, R. Song, Multi-objective reinforcement learning for infectious disease control with application to COVID-19 spread, preprint, arXiv: 2009.04607.
[34]	M. Arango, L. Pelov, Covid-19 pandemic cyclic lockdown optimization using reinforcement learning, preprint, arXiv: 2009.04647.
[35]	T. Smieszek, L. Fiebig, R. W. Scholz, Models of epidemics: when contact repetition and clustering should be included, Theor. Biol. Med. Model., 6 (2009), 1-15. doi: 10.1186/1742-4682-6-1
[36]	A. Gosavi, Simulation-based optimization: Parametric optimization techniques and reinforcement learning, Interfaces, 35 (2005), 535.
[37]	R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, 2018.
[38]	Z. Li, D. W. Purcell, S. L. Sansom, D. Hayes, I. Hall, Vital signs: HIV transmission along the continuum of care-United States, 2016, Morb. Mortal. Wkly. Rep., 68 (2019), 267-272. doi: 10.15585/mmwr.mm6811e1
[39]	E. U. Jacobson, K. A. Hicks, E. L. Tucker, P. G. Farnham, S. L. Sansom, Effects of reaching national goals on HIV incidence, by race and ethnicity, in the United States, J. Public Health Manag. Pract., 24 (2018), E1-E8.
[40]	U.S. Department of Health & Human Services, 2017 National HIV/AIDS Strategy (NHAS) Progress Report Released, 2018. Available from: https://www.hiv.gov/blog/2017-national-hivaids-strategy-nhas-progress-report-released.
[41]	Centers for Disease Control and Prevention, HIV Prevention Progress Report, 2019. Available from: https://www.cdc.gov/hiv/pdf/policies/progressreports/cdc-hiv-preventionprogressreport.pdf.
[42]	UNAIDS, 90-90-90 An ambitious treatment target to help end the AIDS epidemic, 2014. Available from: https://www.unaids.org/sites/default/files/media_asset/90-90-90_en.pdf.
[43]	O. Gottesman, F. Johansson, J. Meier, J. Dent, D. Lee, S. Srinivasan, et al., Evaluating reinforcement learning algorithms in observational health settings, preprint, arXiv: 1805.12298.
[44]	C. Kreatsoulas, S. Subramanian, Machine learning in social epidemiology: Learning from experience, SSM- Popul. Health, 4 (2018), 347. doi: 10.1016/j.ssmph.2018.03.007
[45]	E. M. Gardner, M. P. McLees, J. F. Steiner, C. del Rio, W. J. Burman, The spectrum of engagement in HIV care and its relevance to test-and-treat strategies for prevention of HIV infection, Clin. Infect. Dis., 52 (2011), 793-800. doi: 10.1093/cid/ciq243
[46]	L. Kumaranayake, The economics of scaling up: cost estimation for HIV/AIDS interventions, Aids, 22 (2008), S23-S33. doi: 10.1097/01.aids.0000327620.47103.1d
[47]	A. Lansky, J. Christopher, O. Emeka, S. Catlainn, M. P. Joyce, E. DiNenno, et al., Estimating the number of heterosexual persons in the United States to calculate national rates of HIV infection, PloS One, 10 (2015), e0133543. doi: 10.1371/journal.pone.0133543
[48]	A. Chandra, V. G. Billioux, C. Copen, C. Sionean, HIV risk-related behaviors in the United States household population aged 15-44 years: data from the National Survey of Family Growth, 2002 and 2006-2010, Natl. Health Stat. Rep., 46 (2012), 1-19.
[49]	N. Khurana, E. Yaylali, P. G. Farnham, K. A. Hicks, B. T. Allaire, E. Jacobson, et al., Impact of improved HIV care and treatment on PrEP effectiveness in the United States, 2016-2020, J. Acquir. Immune Defic. Syndr., 78 (2018), 399-405. doi: 10.1097/QAI.0000000000001707
[50]	U. Wilensky, NetLogo, Evanston, IL: Center for Connected Learning and Computer-Based Modeling, Northwestern University, 1999.
[51]	Centers for Disease Control and Prevention, Recommendations for HIV screening of gay, bisexual, and other men who have sex with men-United States, 2017, MMWR Morb. Mortal Wkly. Rep., 66 (2017), 830.

mbe-18-06-380-supplementary.pdf

Reader Comments

Your name:*

Email:*
© 2021 the Author(s), licensee AIMS Press. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0)