Optimal experimental design is a classic topic in statistics, with many well-studied problems, applications, and solutions. The design problem we study is the placement of sensors to monitor spatiotemporal processes, explicitly accounting for the temporal dimension in our modeling and optimization. We observe that recent advancements in computational sciences often yield large datasets based on physics-based simulations, which are rarely leveraged in experimental design. We introduce a novel model-based sensor placement criterion, along with a highly-efficient optimization algorithm, which integrates physics-based simulations and Bayesian experimental design principles to identify sensor networks that “minimize information loss” from simulated data. Our technique relies on sparse variational inference and Gauss-Markov priors, and thus may adapt many techniques from Bayesian experimental design. We validate our method through a case study monitoring air temperature in Phoenix, Arizona, using state-of-the-art physics-based simulations. Our results show our framework to be superior to random or quasi-random sampling, particularly with a limited number of sensors. We conclude by discussing practical considerations and implications of our framework, including more complex modeling tools and real-world deployments.
@article{waxman2025mil,author={Waxman, Daniel and Llorente, Fernando and Lamer, Katia and Djurić, Petar {M.}},title={Designing an Optimal Sensor Network via Minimizing Information Loss},note={Submitted.},year={2025}}
Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes
We revisit the classical problem of Bayesian ensembles and address the challenge of learning optimal combinations of Bayesian models in an online, continual learning setting. To this end, we reinterpret existing approaches such as Bayesian model averaging (BMA) and Bayesian stacking through a novel empirical Bayes lens, shedding new light on the limitations and pathologies of BMA. Further motivated by insights from online optimization, we propose Online Bayesian Stacking (OBS), a method that optimizes the log-score over predictive distributions to adaptively combine Bayesian models. A key contribution of our work is establishing a novel connection between OBS and portfolio selection, bridging Bayesian ensemble learning with a rich, well-studied theoretical framework that offers efficient algorithms and extensive regret analysis. We further clarify the relationship between OBS and online BMA, showing that they optimize related but distinct cost functions. Through theoretical analysis and empirical evaluation, we identify scenarios where OBS outperforms online BMA and provide principled guidance on when practitioners should prefer one approach over the other.
@article{waxman2025obs,author={Waxman, Daniel and Llorente, Fernando and Djurić, Petar {M.}},title={Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes},note={Submitted.},year={2025},}
ICASSP ’25
Decentralized Online Ensembles of Gaussian Processes for Multi-Agent Systems
@inproceedings{llorente2025decentralized,booktitle={2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},author={Llorente*, Fernando and Waxman*, Daniel and Djurić, Petar {M.}},title={Decentralized Online Ensembles of Gaussian Processes for Multi-Agent Systems},year={2025},}
2024
NeurIPS ’24
Tangent Space Causal Inference: Leveraging Vector Fields for Causal Discovery in Dynamical Systems
Causal discovery with time series data remains a challenging yet increasingly important task across many scientific domains. Convergent cross mapping (CCM) and related methods have been proposed to study time series that are generated by dynamical systems, where traditional approaches like Granger causality are unreliable. However, CCM often yields inaccurate results depending upon the quality of the data. We propose the Tangent Space Causal Inference (TSCI) method for detecting causalities in dynamic systems. TSCI works by considering vector fields as explicit representations of the systems’ dynamics and checks for the degree of synchronization between the learned vector fields. The TSCI approach is model-agnostic and can be used as a drop-in replacement for CCM and its generalizations. We present both a basic TSCI algorithm, which is lightweight and more effective than the basic CCM algorithm, as well as augmented versions of TSCI that leverage the expressive power of latent variable models and deep learning. We validate our theory on standard systems, and we demonstrate improved causal inference performance across a number of benchmarks.
@article{butler2024tangent,author={Butler*, Kurt and Waxman*, Daniel and Djurić, Petar {M.}},title={Tangent Space Causal Inference: Leveraging Vector Fields for Causal Discovery in Dynamical Systems},note={Advances in Neural Information Processing Systems (NeurIPS) 2024},year={2024}}
FUSION ’24
A Gaussian Process-based Streaming Algorithm for Prediction of Time Series With Regimes and Outliers
Online prediction of time series under regime switching is a widely studied problem in the literature, with many celebrated approaches. Using the non-parametric flexibility of Gaussian processes, the recently proposed INTEL algorithm provides a product of experts approach to online prediction of time series under possible regime switching, including the special case of outliers. This is achieved by adaptively combining several candidate models, each reporting their predictive distribution at time t. However, the INTEL algorithm uses a finite context window approximation to the predictive distribution, the computation of which scales cubically with the maximum lag, or otherwise scales quartically with exact predictive distributions. We introduce LINTEL, which uses the exact filtering distribution at time t with constant-time updates, making the time complexity of the streaming algorithm optimal. We additionally note that the weighting mechanism of INTEL is better suited to a mixture of experts approach, and propose a fusion policy based on arithmetic averaging for LINTEL. We show experimentally that our proposed approach is over five times faster than INTEL under reasonable settings with better quality predictions.
@inproceedings{waxman2024online,author={Waxman, Daniel and Djurić, Petar {M.}},booktitle={2024 27th International Conference on Information Fusion (FUSION)},title={A Gaussian Process-based Streaming Algorithm for Prediction of Time Series With Regimes and Outliers},year={2024},}
Practical Bayesian learning often requires (1) online inference, (2) dynamic models, and (3) ensembling over multiple different models. Recent advances have shown how to use random feature approximations to achieve scalable, online ensembling of Gaussian processes with desirable theoretical properties and fruitful applications. One key to these methods’ success is the inclusion of a random walk on the model parameters, which makes models dynamic. We show that these methods can be generalized easily to any basis expansion model and that using alternative basis expansions, such as Hilbert space Gaussian processes, often results in better performance. To simplify the process of choosing a specific basis expansion, our method’s generality also allows the ensembling of several entirely different models, for example, a Gaussian process and polynomial regression. Finally, we propose a novel method to ensemble static and dynamic models together.
@article{waxman2024doebe,author={Waxman, Daniel and Djurić, Petar {M.}},title={Dynamic Online Ensembles of Basis Expansions},journal={Transactions on Machine Learning Research (TMLR)},year={2024},}
We introduce Dagma-DCE, an interpretable and model-agnostic scheme for differentiable causal discovery. Current non- or over-parametric methods in differentiable causal discovery use opaque proxies of “independence” to justify the inclusion or exclusion of a causal relationship. We show theoretically and empirically that these proxies may be arbitrarily different than the actual causal strength. Juxtaposed with existing differentiable causal discovery algorithms, Dagma-DCE uses an interpretable measure of causal strength to define weighted adjacency matrices. In a number of simulated datasets, we show our method achieves state-of-the-art level performance. We additionally show that Dagma-DCE allows for principled thresholding and sparsity penalties by domain-experts. The code for our method is available open-source at https://github.com/DanWaxman/DAGMA-DCE, and can easily be adapted to arbitrary differentiable models.
@article{waxman2024dagmadce,author={Waxman, Daniel and Butler, Kurt and Djurić, Petar {M.}},title={DAGMA-DCE: Interpretable, Non-Parametric Differentiable Causal Discovery},year={2024},volume={5},journal={IEEE Open Journal of Signal Processing},pages={393-401},doi={10.1109/OJSP.2024.3351593},}
2023
ACSSC ’23
Fusion of Gaussian Process Predictions With Monte Carlo
Marzieh Ajirak, Daniel Waxman, Fernando Llorente, and Petar M. Djurić
In 2023 57th Asilomar Conference on Signals, Systems, and Computers, 2023
In science and engineering, we often work with models designed for accurate prediction of variables of interest. Recognizing that these models are approximations of reality, it becomes desirable to apply multiple models to the same data and integrate their outcomes. In this paper, we operate within the Bayesian paradigm, relying on Gaussian processes as our models. These models generate predictive probability density functions (pdfs), and the objective is to integrate them systematically, employing both linear and log-linear pooling. We introduce novel approaches for log-linear pooling, determining input-dependent weights for the predictive pdfs of the Gaussian processes. The aggregation of the pdfs is realized through Monte Carlo sampling, drawing samples of weights from their posterior. The performance of these methods, as well as those based on linear pooling, is demonstrated using a synthetic dataset.
@inproceedings{ajirak2023fusion,author={Ajirak, Marzieh and Waxman, Daniel and Llorente, Fernando and Djurić, Petar {M.}},booktitle={2023 57th Asilomar Conference on Signals, Systems, and Computers},title={Fusion of Gaussian Process Predictions With Monte Carlo},year={2023},volume={},number={},pages={1367--1371},doi={},}
EUSIPCO ’23
Detecting Confounders in Multivariate Time Series Using Strength of Causation
One of the most important problems in science is understanding causation. This is particularly challenging when one has access to observational data only and is further compounded in the presence of latent confounders. In this paper, we propose a method for detecting confounders in multivariate time series using a recently introduced concept referred to as differential causal effect (DCE). The solution is based on feature-based Gaussian processes that are used for estimating both, the DCE of the observed time series and the latent confounders. We demonstrate the performance of the proposed method with several examples. They show that the proposed approach can detect confounders and can accurately estimate causal strengths.
@inproceedings{liu2023detecting,author={Liu, Yuhao and Cui, Chen and Waxman, Daniel and Butler, Kurt and Djurić, Petar {M.}},booktitle={2023 31st European Signal Processing Conference (EUSIPCO)},title={Detecting Confounders in Multivariate Time Series Using Strength of Causation},year={2023},volume={},number={},pages={1400-1404},doi={10.23919/EUSIPCO58844.2023.10289850},video={https://youtu.be/I5K6Jc4LJfM}}
talks
“Causal Inference via Quantifying Influences” at the Acoustics Research Institute of the Austrian Academy of Sciences (Institut für Schallforschung der Österreichische Akademie der Wissenschaften) [abstract link] [slides]
“Bayesian Combination” at the 2023 Bellairs Workshop on Machine Learning and Statistical Signal Processing for Data on Graphs