Probabilistic Forecast Reconciliation in Global Health Supply Chain Operations

.sticker-float[![logo](resources/carbs_logo1.jpg)]

<br><br>
.center[.title[Probabilistic Forecast Reconciliation in Global Health Supply Chain Operations]]

.bottom[
Bahman Rostami-Tabar, Associate Professor 
<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M64 112c-8.8 0-16 7.2-16 16v22.1L220.5 291.7c20.7 17 50.4 17 71.1 0L464 150.1V128c0-8.8-7.2-16-16-16H64zM48 212.2V384c0 8.8 7.2 16 16 16H448c8.8 0 16-7.2 16-16V212.2L322 328.8c-38.4 31.5-93.7 31.5-132 0L48 212.2zM0 128C0 92.7 28.7 64 64 64H448c35.3 0 64 28.7 64 64V384c0 35.3-28.7 64-64 64H64c-35.3 0-64-28.7-64-64V128z"/></svg>[rostami-tabarb@cardiff.ac.uk](rostami-tabarb@cardiff.ac.uk)
<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:currentColor;overflow:visible;position:relative;"><path d="M0 256C0 209.4 12.47 165.6 34.27 127.1L144.1 318.3C166 357.5 207.9 384 256 384C270.3 384 283.1 381.7 296.8 377.4L220.5 509.6C95.9 492.3 0 385.3 0 256zM365.1 321.6C377.4 302.4 384 279.1 384 256C384 217.8 367.2 183.5 340.7 160H493.4C505.4 189.6 512 222.1 512 256C512 397.4 397.4 511.1 256 512L365.1 321.6zM477.8 128H256C193.1 128 142.3 172.1 130.5 230.7L54.19 98.47C101 38.53 174 0 256 0C350.8 0 433.5 51.48 477.8 128V128zM168 256C168 207.4 207.4 168 256 168C304.6 168 344 207.4 344 256C344 304.6 304.6 344 256 344C207.4 344 168 304.6 168 256z"/></svg>[www.bahmanrt.com](https://www.bahmanrt.com/)
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- Forecast reconciliation in global health suply chains

- Immunization supply chains

- Forecasting problem

- Forecasting experiment setup

- Forecast accuracy evaluation

- Conclusions & next steps
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .remember[Forecast reconciliation in global health supply chains]

- .gray[Immunization supply chains]

- .gray[Forecasting problem]

- .gray[Forecasting experiment setup]

- .gray[Forecast accuracy evaluation]

- .gray[Conclusions & next steps]

]

---
## Hierarchical structures are evrywhere

.pull-left[
- Tourism forecasting by region, purpose of travel
- Unemployment by region
- Calls received in a call center by type, region
- Ambulance demand by region, station, priority
- Sales by product family, type of customer, brand
- and many more
]

.pull-right[
<img src="resources/hierarchy-left.jpeg" height="450px">
]

---
## Forecasting medical item
.pull-left[
<img src="figure/hierarchy-icrc-1.png" width="100%" style="display: block; margin: auto;" />
]

.pull-right[
- International Committee of the Red Cross (ICRC), Geneva
- 47 products
- 2,674 time series
- 19 regional hub, 6 country warehouses
- Forecasting monthly for 1-12 months ahead
- Intermittent time series
- Presence of obsolete items
]

---
## Forecasting pharmacitcal products

.pull-left[
<img src="img/map_ethipia.png" height="450px">
]

.pull-right[
- Ethiopian Pharmaceuticals Supply Service - EPSS
- 19 regional hub
- 45 zones
- 48 products
- 7,674 time series
- Forecasting monthly for 1-12 months ahead
- Data quality issues
]

---
## Forecasting vaccine doses

.pull-left[
.center[
<img src="img/kenya.png" height="450px">
]
]

.pull-right[
- John Snow Inc. - JSI
- 9 regions
- 47 county
- 306 sub-county
- 4 types of vaccine
- 1,452 time series
- Forecasting monthly for 1-12 months ahead
- Data quality issues
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Forecast reconciliation in global health supply chains]

- .remember[Immunization supply chains]

- .gray[Forecasting problem]

- .gray[Forecasting experiment setup]

- .gray[Forecast accuracy evaluation]

- .gray[Conclusions & next steps]

]

---
.pull-left[
<img src="img/f4sg2022_vaccine/children_immunization.png" height="350px">
]

.left[
.pull-right[
- Approximately .remember[1 in 5] African children .remember[do not receive] all basic vaccines.
- More than .remember[30 million children] under five still. .remember[suffer from vaccine-preventable diseases (VPDs)] every year in Africa.
- Over .remember[half a million children die] from VPDs annually
- Supply chain issues contribute to this pressing issue*.

]

.footnote[Source: [**WHO**](https://www.afro.who.int/health-topics/immunization]), [*Boeck et al., 2023](https://www.emerald.com/insight/content/doi/10.1108/JHLSCM-07-2022-0078/full/html)]

---
## What do we want to achieve?

.pull-left[
### High coverage
<img src="img/f4sg2022_vaccine/coverage.jpg" height="500px">
]

.pull-right[

✅  Reduce stock outs

✅  Reduce missed opportunities

✅  Lower waste

✅ Lower inventory costs

✅  and better coordination
]

.footnote[source:[Allan etal.(2021)](https://www.medrxiv.org/content/10.1101/2021.02.14.21251721v1.full)
]

---
class:inverse, center, middle

.title[ Forecasting accurately the needs for vaccines is one of the key elements in achieving these goals]

---
## The immunization supply chain

.footnote[Source: Effective Vaccine Management Assessment (EVM), 2021]

---

## Vaccines

.three-column[
### Vial
<img src="img/f4sg2022_vaccine/vial.jpg" height="600px">
]

.three-column[
### Dose
<img src="img/f4sg2022_vaccine/dose.jpg" height="600px">
]

.three-column[
### Administrated
<iframe src="https://giphy.com/embed/xT5LMGfQrJPpmXKUEM" height="600px" allowFullScreen></iframe><p><a href="https://giphy.com/gifs/season-16-the-simpsons-16x11-xT5LMGfQrJPpmXKUEM">via GIPHY</a></p>
]
---
.center[
.pull-left[
### Open wastage
<img src="img/f4sg2022_vaccine/open_waste.jpeg" height="600px" >
]

.pull-right[
### Close wastage
<img src="img/f4sg2022_vaccine/vaccine_wastage.jpeg" height="600px">
]
]

---
class: inverse, center, middle

.title[Doses used (consumption/needs) <br> = <br>doses administrated + wastage]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Forecast reconciliation in global health supply chains]

- .gray[Immunization supply chains]

- .remember[Forecasting problem]

- .gray[Forecasting experiment setup]

- .gray[Forecast accuracy evaluation]

- .gray[Conclusions & next steps]

]

---
## Map of Kenya

.center[
<img src="img/kenya.png" height="450px">
]

---
## Structure of vaccine data

.center[
<img src="img/f4sg2022_vaccine/hierarchy.png" height="380px">
]

---
## Hierarchical structure

.pull-left[
<img src="figure/hierarchy-icrchs-1.png" width="100%" style="display: block; margin: auto;" />
]

.pull-right[A .remember[hierarchical time series] is a collection of several time series that are linked together in a hierarchical structure (attributes naturally disaggregate in a unique hierarchical manner).
]

---
## Grouped structure

.pull-left[
<img src="figure/hierarchy3-definition-1.png" width="100%" style="display: block; margin: auto;" />
]

.pull-right[
A .remember[grouped time series] is a collection of time series that can be grouped together in a number of non-uniquely hierarchical ways (attributes do not naturally disaggregate in a unique hierarchical manner).
]

---
## Classical approaches to forecast vaccine needs

- Demographic methods (developed by WHO, one-size-fits-all model)
      - expected target population
      - coverage: is the expected coverage rate
      - doses/target: is the number of doses per target, as per the national vaccination schedule
      - wastage factor

.alert-note[.remember[**Forecasts are annual and at the national level**]]

---
## Limitations of current forecasting method

❌  Forecasts are yearly and at the national level(not useful to inform operational / tactical decisions.)

❌  Forecasts are  often produced based on unrealistic assumptions.

❌  Forecasts do not acknowledge uncertainty.

❌  Do not capture the information available at multiple hierarchical levels.

❌  Forecasts ignore the hierarchical nature of the problem, not coherent.

❌  Lead to conflicting decisions & lack of coordination.

---
## How to forecast hierarchical time series?

.pull-left[
- Base forecast (generated using any forecasting model)
- Bottom-Up
- Top Down
- Middle-Out
- .remember[Forecast reconciliation]
]

.pull-right[
<img src="img/f4sg2022_vaccine/pyramid.jpg" width="90%" style="display: block; margin: auto;" />
]

---
 
## Forecast reconciliation
 
- This approach involves first generating independent base forecast for each series in the hierarchy (i.e. Base)

- As these base forecasts are independently generated they will not be “aggregate consistent” (i.e., they will not add up according to the hierarchical structure). They are .remember[not coherent].

- The reconciliation approaches combine the independent base forecasts and generates a set of revised forecasts that are as close as possible to the univariate forecasts but also aggregate consistently with the hierarchical structure.

- Unlike any other existing method, this approach uses all the information available within a hierarchy.

---
## Reconciled forecasts-notations

- Let `$\large {b}_t$` be a vector of `$\large n_b$` _bottom-level_ time series at time `$\large t$`, and let `$\large {a}_t$` be a corresponding vector of `$\large n_a = n-n_b$` aggregated time series, where `$\large a_t = {A}{b}_t,$`

- `$\large A$` is the `$\large n_a\times n_b$` "aggregation" matrix specifying how the bottom-level series `${b}_t$` are to be aggregated to form `$\large {a}_t$`.

- The full vector of time series is given by `$\large y_t = \begin{bmatrix}{a}_t \\{b}_t\end{bmatrix}.$`

- This leads to the `$\large n\times n_b$` "summing" or "structural" matrix given by
`$\large S = \begin{bmatrix}{A} \\ {I}_{n_b}\end{bmatrix}$`, such that `$\large {y}_t = {S}{b}_t$`.

- Similar approach can be used to create reconciled forecasts, `$\large \tilde{{y}}_t = {SG}{\hat{y}}_t$`.

---
class: middle, center

## Coherency and reconciliation

.pull-left[
<img src="figure/simple-hierarchy1-1.png" width="120%" style="display: block; margin: auto;" />
]

.pull-right[
<img src="figure/wast_hiararchy/hierarchy_matrix.png" height="400px">
]

---
## Hierarchical, and linearly constrained time series
.pull-left[
<img src="figure/hierarchy1-linear-1.png" width="100%" style="display: block; margin: auto;" />
]

.pull-right[
- Total = Central & West + North South & East
- Central & West = HD + SB + PO
- North = BC
- South & East = CV + CT + AB
]

.remember[Coherent forecast] satisfies these aggregation constraints.

---
##  Linear reconciliation methods, Bottom-up and others

Forecast reconciliation approaches combine and reconcile all the base forecasts in order to produce coherent forecasts.

Linear reconciliation methods (Wickramasuriya, Athanasopoulos, and Hyndman 2019) can be written as

`$$\large \tilde{{y}}_h = {S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h ={S}{G}\hat{{y}}_h = {M}\hat{{y}}_h,$$`

where `${W}$` is an `$n \times n$` positive definite matrix, and `$\hat{{y}}_h$` contains the `$h$`-step forecasts of `${y}_{T+h}$` given data to time `$T$`. `${W}^{-1}$` is the variance-covariance matrix of h-step ahead errors.

- W is the only unknown to get the reconciled forecast.

---
## Producing probabilistic forecasts

- We use bootstrapping to generate probabilistic forecasts:

- Suppose that `$(\hat{{y}}_h^{[1]},\dots,\hat{{y}}_h^{[B]})$` are a set of `$B$` simulated sample paths, generated independently from the models used to produce the base forecasts. 
    - Then `$({S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h^{[1]},\dots,{S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h^{[B]})$` provides a set of reconciled sample paths, from which percentiles can be calculated.

---
class: center
## Probabilistic forecast

.center[
<img src="figure/wast_hiararchy/daily_probabilistic_forecast.png" height="550px">
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Forecast reconciliation in global health supply chains]

- .gray[Immunization supply chains]

- .gray[Forecasting problem]

- .remember[Forecasting experiment setup]

- .gray[Forecast accuracy evaluation]

- .gray[Conclusions & next steps]

]

---
## Data

- Monthly vaccine consumption in Kenya from January 2013 until December 2021

- Four type of vaccines
    - Measles
    - Bacillus Calmette–Guérin (BCG) - for tuberculosis
    - DPT, a class of combination vaccines against three infectious diseases in humans: diphtheria-tetanus-pertussis
    - OPV, Oral poliovirus vaccines, used in the fight to eradicate polio
- 306 sub-county, 47 county & 9 regions
- Total of 1452 time series
- Useful predictors including child population, conflicts, strike, extreme weather conditions.

---
## Total number of series for each type of vaccine

<br><br>
<table class="table table-striped" style="font-size: 20px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> Level </th>
   <th style="text-align:right;"> Number of series </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> All country (Total) </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Regions </td>
   <td style="text-align:right;"> 9 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> County </td>
   <td style="text-align:right;"> 47 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Sub-county </td>
   <td style="text-align:right;"> 306 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 363 </td>
  </tr>
</tbody>
</table>

---
## Doses used (raw data): total and regions

---
## Doses used: National & regions

.center[
<img src="img/f4sg2022_vaccine/p1.png" height="600px">
]

---
## Doses used: trend and seasonality features

.center[
<img src="img/f4sg2022_vaccine/admin_hts_feature_cat_plot.png" height="600px">
]

---
## Forecasting setup

- Forecast horizon: 12 months
- Point and probabilistic forecasts are generated and evaluated for the entire hierarchy
- Forecast evaluation

- Time series cross-validation with re-estimation
    - Used 36 months as test set

---
## Forecasting models

- **Naive:** Assuming that the future days will be similar to past days. We use the empirical distribution of the past daily demand
- **Exponential Smoothing State Space model (ETS):** ETS models can combine trend, seasonality, and error components in a time series
- **Poisson Regression using tscount (TSGLM):** Poisson regression model that takes into account serial dependence and predictors
- **Ensemble method:** Finally, we use an ensemble method that combines the forecasts generated from the Naive, ETS, and TSGLM models to form a mixture distribution

---
## Forecasting performance metrics- point forecast

.pull-left[
### Mean Absolute Scaled Error
`$$\large \text{MASE} = \text{mean}(|q_{j}|),$$`

where

`$$q_{j} = \frac{ e_{j}}
 {\displaystyle\frac{1}{T-m}\sum_{t=m+1}^T |y_{t}-y_{t-m}|},$$`
]

.pull-right[
### Mean Squared Scaled Error

`$$\large \text{MSSE} = \text{mean}(q_{j}^2),$$`

where,

`$$q^2_{j} = \frac{ e^2_{j}}
 {\displaystyle\frac{1}{T-m}\sum_{t=m+1}^T (y_{t}-y_{t-m})^2},$$`
]

---
## Forecasting performance metrics- probabilistic forecast

.pull-left[

`$$\large \text{CRPS} = \text{mean}(p_j),$$`

where

`$$p_j = \int_{-\infty}^{\infty} \left(G_j(x) - F_j(x)\right)^2dx,$$`

]

.pull-right[
<img src="figs/crps.jpg" height="450px">
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Forecast reconciliation in global health supply chains]

- .gray[Immunization supply chains]

- .gray[Forecasting problem]

- .gray[Forecasting experiment setup]

- .remember[Forecast accuracy evaluation]

- .gray[Conclusions & next steps]

]

---
## Overall forecast accuracy

.small[
<table class=" lightable-classic" style="font-family: Cambria; width: auto !important; margin-left: auto; margin-right: auto;">
<caption>MSSE</caption>
 <thead>
  <tr>
   <th style="text-align:left;"> Method </th>
   <th style="text-align:right;"> Total </th>
   <th style="text-align:right;"> Region </th>
   <th style="text-align:right;"> County </th>
   <th style="text-align:right;"> Sub County </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Base </td>
   <td style="text-align:right;"> 0.526 </td>
   <td style="text-align:right;"> 0.594 </td>
   <td style="text-align:right;"> 0.656 </td>
   <td style="text-align:right;"> 0.651 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Bottom Up </td>
   <td style="text-align:right;"> 0.509 </td>
   <td style="text-align:right;"> 0.594 </td>
   <td style="text-align:right;"> 0.645 </td>
   <td style="text-align:right;"> 0.649 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Top Down </td>
   <td style="text-align:right;"> 0.526 </td>
   <td style="text-align:right;"> 0.807 </td>
   <td style="text-align:right;"> 0.739 </td>
   <td style="text-align:right;"> 0.717 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Reconciliation </td>
   <td style="text-align:right;"> 0.428 </td>
   <td style="text-align:right;"> 0.510 </td>
   <td style="text-align:right;"> 0.600 </td>
   <td style="text-align:right;"> 0.609 </td>
  </tr>
</tbody>
</table>

<table class=" lightable-classic" style="font-family: Cambria; width: auto !important; margin-left: auto; margin-right: auto;">
<caption>CRPS</caption>
 <thead>
  <tr>
   <th style="text-align:left;"> Method </th>
   <th style="text-align:right;"> Total </th>
   <th style="text-align:right;"> Region </th>
   <th style="text-align:right;"> County </th>
   <th style="text-align:right;"> Sub County </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Base </td>
   <td style="text-align:right;"> 349.345 </td>
   <td style="text-align:right;"> 100.880 </td>
   <td style="text-align:right;"> 55.236 </td>
   <td style="text-align:right;"> 30.256 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Bottom Up </td>
   <td style="text-align:right;"> 320.343 </td>
   <td style="text-align:right;"> 89.345 </td>
   <td style="text-align:right;"> 45.223 </td>
   <td style="text-align:right;"> 29.341 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Top Down </td>
   <td style="text-align:right;"> 374.431 </td>
   <td style="text-align:right;"> 105.123 </td>
   <td style="text-align:right;"> 66.345 </td>
   <td style="text-align:right;"> 51.340 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Reconciliation </td>
   <td style="text-align:right;"> 256.345 </td>
   <td style="text-align:right;"> 74.345 </td>
   <td style="text-align:right;"> 36.234 </td>
   <td style="text-align:right;"> 23.654 </td>
  </tr>
</tbody>
</table>
]

.footnote[Forecast accuracy improvement against the current approach in immunization programs is calculated but not included in the presentation]

---
## Accuracy over horizon
.pull-left[
### CRPS
<img src="figure/accuracy-h-1.png" width="80%" style="display: block; margin: auto;" />
]

.pull-right[
### MSSE
<img src="figure/accuracy-hmsse-1.png" width="80%" style="display: block; margin: auto;" />
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Forecast reconciliation in global health supply chains]

- .gray[Immunization supply chains]

- .gray[Forecasting problem]

- .gray[Forecasting experiment setup]

- .gray[Forecast accuracy evaluation]

- .remember[Conclusions & next steps]

]

---
## Conclusions
### Benefit of hierarchical forecasting

✅  Plans at any level are based on .remember[coherent forecasts] and therefore can be aligned.

✅  Hierarchical forecasting framework can be used as a tool to .remember[improve coordination] between teams across the supply chain at the national, sub-national, regional and local levels.

✅  Result can be .remember[more accurate than the independent (base) forecasts].

✅  Hierarchical forecasting framework an be used to create coherent forecast, regardless of how base forecasts are created, even with judgmental forecasts.

---
## Next steps

- Developing an impact case study with JSI

- Developing hybrid methods that integrates current forecasting method in immunization programs

- Evaluate the implication of forecast accuracy on utilities such as cost, service level, coordination, etc by linking to replenishment policies.

---

## Acknowledgement

- John and Snow Inc. (JSI) team

- The representative of the National Vaccine Immunization Program in Kenya

- [Developers of Tidyverts](https://tidyverts.org/) and [Tidyverse](https://www.tidyverse.org/) packages

---
## References

- Effective Vaccine Management (EVM 2.0) Assessment Report –  2021, WHO

- Vaccine Logistics, WHO training manual

- [Bahman Rostami-Tabar, Rob J Hyndman (2023), Probabilistic Forecast Reconciliation For Emergency Services Demand]([https://robjhyndman.com/seminars/fem.html)

- [Hyndman and George Athanasopoulos, Forecasting: Principles and Practice,3rd ed](https://otexts.com/fpp3/hierarchical.html)

---
## My new book
### [https://dfep.netlify.app/](https://dfep.netlify.app/)

.center[
<img src="img/book_cover.png" height="550px">
]

---
.pull-left[
## About me
Bahman Rostami-Tabar <br>
Associate Professor of Data-Driven Decision Science
Cardiff University, UK
<br><br>

Slides @ [www.bahmanrt.com](www.bahmanrt.com)
<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#1da1f2;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg>[@Bahman_R_T](https://twitter.com/Bahman_R_T)
<svg aria-hidden="true" role="img" viewBox="0 0 448 512" style="height:1em;width:0.88em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:blue;overflow:visible;position:relative;"><path d="M100.28 448H7.4V148.9h92.88zM53.79 108.1C24.09 108.1 0 83.5 0 53.8a53.79 53.79 0 0 1 107.58 0c0 29.7-24.1 54.3-53.79 54.3zM447.9 448h-92.68V302.4c0-34.7-.7-79.2-48.29-79.2-48.29 0-55.69 37.7-55.69 76.7V448h-92.78V148.9h89.08v40.8h1.3c12.4-23.5 42.69-48.3 87.88-48.3 94 0 111.28 61.9 111.28 142.3V448z"/></svg> Connect: [Bahman ROSTAMI-TABAR](https://www.linkedin.com/in/bahman-rostami-tabar-1046171a/)
]

.pull-right[
## Outline

- Forecast reconciliation in global health supply chains

- Forecasting problem

- Forecasting experiment setup

- Forecast accuracy evaluation

- Conclusions & next steps
]