Gauri Jain

research

1. PRICAI 2024

   IRL Making a Difference "IRL": Inverse Reinforcement Learning for Restless Multi-Armed Bandits with Applications in Maternal and Child Health

   Gauri Jain, Pradeep Varakantham, Haifeng Xu, Aparna Taneja, Prashant Doshi, and Milind Tambe

   PRICAI Nov 2024

   Abs

   Public health practitioners often have the goal of monitoring patients and maximizing patients’ time spent in “favorable" or healthy states while being constrained to using limited resources. Restless multi-armed bandits (RMAB) are an effective model to solve this problem as they are helpful to allocate limited resources among many agents under resource constraints, where patients behave differently depending on whether they are intervened on or not. However, RMABs assume the reward function is known. This is unrealistic in many public health settings because patients face unique challenges and it is impossible for a human to know who is most deserving of any intervention at such a large scale. To address this shortcoming, this paper is the first to present the use of inverse reinforcement learning (IRL) to learn desired rewards for RMABs, and we demonstrate improved outcomes in a maternal and child health telehealth program. First we allow public health experts to specify their goals at an aggregate or population level and propose an algorithm to design expert trajectories at scale based on those goals. Second, our algorithm WHIRL uses gradient updates to optimize the objective, allowing for efficient and accurate learning of RMAB rewards. Third, we compare with existing baselines and outperform those in terms of run-time and accuracy. Finally, we evaluate and show the usefulness of WHIRL on thousands on beneficiaries from a real-world maternal and child health setting in India. We publicly release our code here: https://github.com/Gjain234/WHIRL.

2. IJCAI 2023 Workshop

   Inverse Reinforcement Learning for Restless Multi-Armed Bandits with Application to Maternal and Child Health

   Gauri Jain, Pradeep Varakantham, Haifeng Xu, Aparna Taneja, and Milind Tambe

   IJCAI Bridge AI Workshop Aug 2023

   Abs URL

   We study restless multi-armed bandits (RMABs) in the context of public health, where there is a need to optimize resource allocation decisions. Until now, RMABs typically solve for the optimal planning policy by assuming the reward function in the problem is fully known. However, in this work, we aim to study whether we can learn the most optimal rewards for an RMAB problem given some demonstrated, ideal behavior. To achieve this, we turn to inverse reinforcement learning (IRL) which is a field of study motivated by the desire to understand and learn the underlying reward structure of an agent’s observed behavior. Existing IRL approaches predominantly focus on single agent systems, presenting limitations in dealing with the expansive state spaces characteristic of public health scenarios, where tens of thousands of arms are active simultaneously. We propose a new IRL algorithm specifically for RMAB settings that uses techniques from decision focused learning (DFL) to directly optimize the objective function, allowing for efficient and accurate updates to the learned rewards. We compare our algorithm with the max entropy IRL baseline on runtime and accuracy and find that our algorithm performs better on both metrics. We also propose a framework for how to apply this algorithm in the public health domain where expert trajectories come from domain experts.

3. AAAI 2023 Workshop

   BiomeAzuero2022: A Fine-Grained Dataset and Baselines for Tree Species Classification with Ground Images

   Ziwei Gu*, Gauri Jain*, Hongwen Song*, Isak Diaz Diaz, Margaux Masson-Forsythe, and Jorge Valdes

   AAAI AI for Social Good Workshop Feb 2023

   URL
4. Informs 2022

   Sequential Fair Allocation: Achieving the Optimal Envy-Efficiency Tradeoff Curve

   Sean Sinclair, Gauri Jain, Siddhartha Banerjee, and Christina Lee Yu

   Operations Research Nov 2022

   Abs URL

   We consider the problem of dividing limited resources to individuals arriving over T rounds. Each round has a random number of individuals arrive, and individuals can be characterized by their type (i.e. preferences over the different resources). A standard notion of ’fairness’ in this setting is that an allocation simultaneously satisfy envy-freeness and efficiency. The former is an individual guarantee, requiring that each agent prefers their own allocation over the allocation of any other; in contrast, efficiency is a global property, requiring that the allocations clear the available resources. For divisible resources, when the number of individuals of each type are known upfront, the above desiderata are simultaneously achievable for a large class of utility functions. However, in an online setting when the number of individuals of each type are only revealed round by round, no policy can guarantee these desiderata simultaneously, and hence the best one can do is to try and allocate so as to approximately satisfy the two properties. We show that in the online setting, the two desired properties (envy-freeness and efficiency) are in direct contention, in that any algorithm achieving additive counterfactual envy-freeness up to a factor of LT necessarily suffers a efficiency loss of at least 1/LT. We complement this uncertainty principle with a simple algorithm, HopeGuardrail, which allocates resources based on an adaptive threshold policy and is able to achieve any fairness-efficiency point on this frontier. In simulation results, our algorithm provides allocations close to the optimal fair solution in hindsight, motivating its use in practical applications as the algorithm is able to adapt to any desired fairness efficiency trade-off.

5. NeurIPS 2020

   Adaptive Discretization for Model-Based Reinforcement Learning

   Sean Sinclair, Tianyu Wang, Gauri Jain, Siddhartha Banerjee, and Christina Lee Yu

   Advances in Neural Information Processing Systems 33 May 2020

   Abs URL

   We introduce the technique of adaptive discretization to design an efficient model-based episodic reinforcement learning algorithm in large (potentially continuous) state-action spaces. Our algorithm is based on optimistic one-step value iteration extended to maintain an adaptive discretization of the space. From a theoretical perspective we provide worst-case regret bounds for our algorithm which are competitive compared to the state-of-the-art model-based algorithms. Moreover, our bounds are obtained via a modular proof technique which can potentially extend to incorporate additional structure on the problem. From an implementation standpoint, our algorithm has much lower storage and computational requirements due to maintaining a more efficient partition of the state and action spaces. We illustrate this via experiments on several canonical control problems, which shows that our algorithm empirically performs significantly better than fixed discretization in terms of both faster convergence and lower memory usage. Interestingly, we observe empirically that while fixed discretization model-based algorithms vastly outperform their model-free counterparts, the two achieve comparable performance with adaptive discretization.