Hierarchical Time Series Forecasting of ambulance demand

.sticker-left[![isf](resources/carbs_logo1.jpg)]

.center[.title[Probabilistic Forecast Reconciliation For Health Services]]
<br>
.bottom[
**_Presenter:_** Dr. Bahman Rostami-Tabar (<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#1da1f2;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg>[@Bahman_R_T](https://twitter.com/Bahman_R_T)), Cardiff University, UK. <br>
**_Co-author:_** Prof. Rob J Hyndman (<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#1da1f2;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg>[@robjhyndman](https://twitter.com/robjhyndman)), Monash University, Australia.

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- Hierarchical and grouped time series structures

- Hierarchical forecasting approaches

- Forecasting experiment setup

- Forecasting performance evaluation

- Conclusion
 
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .remember[Hierarchical and grouped time series structures]

- .gray[Hierarchical forecasting approaches]

- .gray[Forecasting experiment setup]

- .gray[Forecasting performance evaluation]

- .gray[Conclusion]

]

---
## Hierarchical data structures are everywhere

- Ambulance demand
- Accident and Emergency admissions
- Calls received in a call center
- Length of Stay
- Waiting time
- pharmaceutical product
- and many more

---
## Verified incidents in Wales

.center[
<img src="figure/wast_hiararchy/wales_roll_out.png" height="550px">
]

---
## Hierarchical structure

---
## Hierarchical structure

A .remember[hierarchical time series] is a collection of several time series that are linked together in a hierarchical structure (attributes naturally disaggregate in a unique hierarchical manner).

---
## Grouped structure

---
## Grouped structure

---
## Grouped structure

A .remember[grouped time series] is a collection of time series that can be grouped together in a number of non-uniquely hierarchical ways (attributes do not naturally disaggregate in a unique hierarchical manner).

.pull-left[
<img src="figure/hierarchy3-definition-1.png" width="100%" style="display: block; margin: auto;" />
]

.pull-right[
<img src="figure/hierarchy2-definition-1.png" width="100%" style="display: block; margin: auto;" />
]

---
class: centre
## Hierarchical and grouped structure

.pull-left[
<img src="figure/hierarchy12-1.png" width="120%" style="display: block; margin: auto;" />
]

.pull-right[
<img src="figure/wast_hiararchy/group.png" width="90%" style="display: block; margin: auto;" />
]

---
## Incident data

.remark-code[

```
#> # A tsibble: 967,400 x 6 [1D]
#> # Key:       control, lhb_code, priority, nature [691]
#>   date       control lhb_code priority nature                   incidents
#>   <date>     <chr>   <fct>    <fct>    <fct>                        <dbl>
#> 1 2017-12-26 N       BC       AMBER    BREATHING PROBLEMS              62
#> 2 2017-01-01 N       BC       AMBER    FALLS                           58
#> 3 2016-03-28 N       BC       AMBER    CHEST PAIN                      55
#> 4 2016-04-14 N       BC       GREEN    HEALTH CARE PROFESSIONAL        53
#> 5 2016-07-22 N       BC       GREEN    HEALTH CARE PROFESSIONAL        53
#> 6 2016-07-29 N       BC       GREEN    HEALTH CARE PROFESSIONAL        53
#> 7 2017-01-03 N       BC       AMBER    BREATHING PROBLEMS              52
#> 8 2018-01-14 N       BC       AMBER    BREATHING PROBLEMS              52
#> # … with 967,392 more rows
```
]

---
## Hierarchically aggregated incident data

```
#> # A tsibble: 2,142,000 x 6 [1D]
#> # Key:       control, priority, nature, lhb [1,530]
#>   date       control      lhb          priority     nature       incident
#>   <date>     <chr*>       <chr*>       <chr*>       <chr*>          <dbl>
#> 1 2015-10-01 <aggregated> <aggregated> <aggregated> <aggregated>     1020
#> 2 2015-10-02 <aggregated> <aggregated> <aggregated> <aggregated>     1021
#> 3 2015-10-03 <aggregated> <aggregated> <aggregated> <aggregated>     1025
#> 4 2015-10-04 <aggregated> <aggregated> <aggregated> <aggregated>     1043
#> 5 2015-10-05 <aggregated> <aggregated> <aggregated> <aggregated>     1067
#> 6 2015-10-06 <aggregated> <aggregated> <aggregated> <aggregated>     1063
#> 7 2015-10-07 <aggregated> <aggregated> <aggregated> <aggregated>      973
#> 8 2015-10-08 <aggregated> <aggregated> <aggregated> <aggregated>     1057
#> # … with 2,141,992 more rows
```

---
## Total number of series

<table class="table table-striped" style="font-size: 20px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> Level </th>
   <th style="text-align:right;"> Number of series </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> All country (Total) </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Control </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Health board </td>
   <td style="text-align:right;"> 7 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Priority </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Priority * Control </td>
   <td style="text-align:right;"> 9 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Priority * Health board </td>
   <td style="text-align:right;"> 21 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Nature of incident </td>
   <td style="text-align:right;"> 35 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Nature of incident * Control </td>
   <td style="text-align:right;"> 105 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Nature of incident * Health board </td>
   <td style="text-align:right;"> 245 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Priority * Nature of incident </td>
   <td style="text-align:right;"> 104 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Control * Priority * Nature of incident </td>
   <td style="text-align:right;"> 306 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Control * Health board * Priority * Nature of incident (Bottom level) </td>
   <td style="text-align:right;"> 691 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 1530 </td>
  </tr>
</tbody>
</table>

---
## Forecasts are often required at all levels

How to produce a coherent forecast for a large collections of related time series

.pull-left[
### Hierarchical levels
- National or area level (strategic and long-term) such as workforce resource planning and budgeting; 
- Health board level (tactical and medium-term) such as temporary capacity expansions, resource sharing, and staffing;
- hospital or station level (operational and short-term) such as planning rosters.]

.pull-right[
### Grouped levels

Forecasts might also be required at different levels for a specific area of interest:

- nature of incidents 
  - priority levels
  - gender
  - age group
  - etc
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Hierarchical and grouped time series structures]

- .remember[Hierarchical forecasting approaches]

- .gray[Forecasting experiment setup]

- .gray[Forecasting performance evaluation]

- .gray[Conclusion]

]

---

.pull-left[
.smalll[
<table class="table table-striped" style="font-size: 20px; width: auto !important; margin-left: auto; margin-right: auto;">
 <thead>
  <tr>
   <th style="text-align:left;"> Level </th>
   <th style="text-align:right;"> Number of series </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> All country </td>
   <td style="text-align:right;"> 1 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Control </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Health board </td>
   <td style="text-align:right;"> 7 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Priority </td>
   <td style="text-align:right;"> 3 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Priority * Control </td>
   <td style="text-align:right;"> 9 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Priority * Health board </td>
   <td style="text-align:right;"> 21 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Nature of incident </td>
   <td style="text-align:right;"> 35 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Nature of incident * Control </td>
   <td style="text-align:right;"> 105 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Nature of incident * Health board </td>
   <td style="text-align:right;"> 245 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Priority * Nature of incident </td>
   <td style="text-align:right;"> 104 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Control * Priority * Nature of incident </td>
   <td style="text-align:right;"> 306 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Control * Health board * Priority * Nature of incident (Bottom level) </td>
   <td style="text-align:right;"> 691 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 1530 </td>
  </tr>
</tbody>
</table>
]
]

.pull-right[
---
## Base forecast

- Producing independent forecasts, also refereed to as base forecasts, typically by different teams as the need for such forecasts arise.

- Same or different forecasting model could be used in each level.

- Forecasts are not coherent, the would not add up as time series does.
]

---
class: middle
.pull-left[
### Hierarchical series
<img src="figure/wast_hiararchy/fig-dataviz2-1.png" width="75%" style="display: block; margin: auto;" />
]

.pull-right[
### Hierarchical forecasting approaches
<img src="figure/wast_hiararchy/pyramid.png" width="90%" style="display: block; margin: auto;" />
]

---
## Reconciled forecasts

- This approach involves first generating independently  base forecast for each series in the hierarchy (they will not add up according to the hierarchical structure). 
- Reconciliation is a post-forecasting process aimed at improving the accuracy of the base forecasts (however obtained) 
- The Reconciliation approach combines the base forecasts and generates a set of revised forecasts that aggregate consistently with the hierarchical structure (they are coherent).

---
class: center
## Probabilistic forecast

.center[
<img src="figure/wast_hiararchy/daily_probabilistic_forecast.png" height="550px">
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Hierarchical and grouped time series structures]

- .gray[Hierarchical forecasting approaches]

- .remember[Forecasting experiment setup]

- .gray[Forecasting performance evaluation]

- .gray[Conclusion]

]

---
## Time series of ambulance demand

**An inherent hierarchical and grouped structure**

- Historical daily demand for over 5 years
- Demand for ambulance services at the country level can be disaggregated into:

- a geographical hierarchy into control, health boards
    - groups (attributes) such as the nature of incidents and priority.

---
## Strength of trend and seasonality

.center[
<img src="figure/wast_hiararchy/fig-feature-1.png" height="550px">
]

---
## Time plots: few examples

.center[
<img src="figure/wast_hiararchy/fig-dataviz2-1.png" height="600px">
]

---
## Forecasting setup

- Forecasting to inform a planning horizon of 42 days
- The forecast horizon is 84 days ahead
- Point and probabilistic forecasts are generated and evaluated for the entire hierarchy
- Forecasts are evaluated using time series cross validation tested on the last year of data

.center[
<img src="figure/wast_hiararchy/planning_horizon.png" height="400px">
]

---
## Forecasting models

- **Naive:** Assuming that the future days will be similar to past days. We use the empirical distribution of the past daily demand
- **Exponential Smoothing State Space model (ETS):** ETS models can combine trend, seasonality, and error components in a time series
- **Generalized Linear Model (GLM):** GLMs are a family of models developed to extend the concept of linear regression models to non-Gaussian distributions 
- **Poisson Regression using tscount (TSGLM):** We also consider another Poisson regression model that takes into account serial dependence
- **Ensemble method:** Finally, we use an ensemble method that combines the forecasts generated from the Naive, ETS, GLM and TSGLM models to form a mixture distribution

---
## Forecasting performance metrics- point forecast

.pull-left[
`$$\large \text{MASE} = \text{mean}(|q_{j}|),$$`

where

`$$q_{j} = \frac{ e_{j}}
 {\displaystyle\frac{1}{T-m}\sum_{t=m+1}^T |y_{t}-y_{t-m}|},$$`
]

.pull-right[
`$$\large \text{MSSE} = \text{mean}(q_{j}^2),$$`

where,

`$$q^2_{j} = \frac{ e^2_{j}}
 {\displaystyle\frac{1}{T-m}\sum_{t=m+1}^T (y_{t}-y_{t-m})^2},$$`
]

---
## Forecasting performance metrics- probabilistic forecast

.pull-left[
`$$\large \text{CRPS} = \text{mean}(p_j),$$`

where

`$$p_j = \int_{-\infty}^{\infty} \left(G_j(x) - F_j(x)\right)^2dx,$$`
]

.pull-right[
<img src="img/BI_DW_NHS_wales/crps.jpg" height="450px">
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Hierarchical and grouped time series structures]

- .gray[Hierarchical forecasting approaches]

- .gray[Forecasting experiment setup]

- .remember[Forecasting performance evaluation]

- .gray[Conclusion]

]

---
## Forecast accuracy result

.center[
<img src="figure/wast_hiararchy/result.png" height="550px">
]

---
## Forecast accuracy result- optimal reconciliation

.center[
<img src="figure/wast_hiararchy/fig-accuracy-1.png" height="550px">
]

---
background-image: url("resources/hierarchy-left.jpeg")
background-size: contain
background-position: left
class: middle

.pull-right2[
## Outline

- .gray[Hierarchical and group time series structure]

- .gray[Time series of ambulance demand & forecasting setup]

- .gray[Forecasting models]

- .gray[Hierarchical and grouped time series forecasting approaches]

- .gray[Forecasting performance]

- .remember[Conclusion]

]

---
## What if hierarchies are ignored

❌  Forecasts are not consistent  no  coherency.

❌  Base forecasts can result in lack of consistency and coordination, leading to less effective planning and decision making.

❌  Teams operating in isolation leading to conflicts, duplication work, rework, or work that runs counter to the overall goal to improve the quality of delivery service.

❌ Not using all information available at the other levels of hierarchy to improve your forecast.

---
## Benefit of hierarchical/grouped time series forecasting

✅  Plans at any level are based on .remember[coherent forecasts] and therefore can be aligned.

✅  Hierarchical forecasting framework can be used as a tool to .remember[improve coordination] between teams across the care services at the national, sub-national, regional and local levels.

✅  Result in .remember[more accurate than the independent (base) forecasts].

✅  Hierarchical forecasting can be used to create coherent forecast, regardless of how base forecasts are created, even with judgmental forecasts.

---
## Temporal and cross-temporal hierarchies

- Here we discussed hierarchical and grouped time series, which deals with these structure at a one time granularity.

- The same issues arise when you need to forecast across different temporal granularities. You have an hourly time series, you want to forecast future daily, weekly, monthly, quarterly demand

- It is also possible to combine hierarchical and temporal hierarchies together

---
## Potential topics for research

- High dimensional hierarchical forecasting
- Forecasting with high frequency data
- Probabilistic hierarchical forecasting for discrete distributions
- Machine learning and AI approaches to hierarchical forecasting
- Hierarchical forecasting with explanatory variables
- Applications of hierarchical forecasting to other services in healthcare

---
## My new book
### [https://dfep.netlify.app/](https://dfep.netlify.app/)

.center[
<img src="img/book_cover.png" height="550px">
]

---

.pull-left[
Presenter: Dr. Bahman Rostami-Tabar (<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#1da1f2;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg>[@Bahman_R_T](https://twitter.com/Bahman_R_T)), Cardiff University, UK. <br>
Co-author: Prof. Rob J Hyndman (<svg aria-hidden="true" role="img" viewBox="0 0 512 512" style="height:1em;width:1em;vertical-align:-0.125em;margin-left:auto;margin-right:auto;font-size:inherit;fill:#1da1f2;overflow:visible;position:relative;"><path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"/></svg>[@robjhyndman](https://twitter.com/robjhyndman)), Monash University, Australia.

]

.pull-right[
## Outline

- Hierarchical and grouped time series structures

- Hierarchical forecasting approaches

- Forecasting experiment setup

- Forecasting performance evaluation

- Conclusion

]

---
## Reconciled forecasts-notations

- Let `$\large {b}_t$` be a vector of `$\large n_b$` _bottom-level_ time series at time `$\large t$`, and let `$\large {a}_t$` be a corresponding vector of `$\large n_a = n-n_b$` aggregated time series, where `$\large a_t = {A}{b}_t,$`

- `$\large A$` is the `$\large n_a\times n_b$` "aggregation" matrix specifying how the bottom-level series `${b}_t$` are to be aggregated to form `$\large {a}_t$`.

- The full vector of time series is given by `$\large y_t = \begin{bmatrix}{a}_t \\{b}_t\end{bmatrix}.$`

- This leads to the `$\large n\times n_b$` "summing" or "structural" matrix given by
`$\large S = \begin{bmatrix}{A} \\ {I}_{n_b}\end{bmatrix}$`, such that `$\large {y}_t = {S}{b}_t$`.

---
class: middle, center

.pull-left[
<img src="figure/simple-hierarchy1-1.png" width="120%" style="display: block; margin: auto;" />
]

.pull-right[
<img src="figure/wast_hiararchy/hierarchy_matrix.png" height="400px">
]

---
## Hierarchical, and linearly constrained time series
.pull-left[
<img src="figure/hierarchy1-linear-1.png" width="100%" style="display: block; margin: auto;" />
]

.pull-right[
- Total = Central & West + North South & East
- Central & West = HD + SB + PO
- North = BC
- South & East = CV + CT + AB
]

.remember[Coherent forecast] satisfies these aggregation constraints.

---
##  Linear reconciliation methods, Bottom-up and others

Forecast reconciliation approaches combine and reconcile all the base forecasts in order to produce coherent forecasts.

Linear reconciliation methods (Wickramasuriya, Athanasopoulos, and Hyndman 2019) can be written as

`$$\large \tilde{{y}}_h = {S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h ={S}{G}\hat{{y}}_h = {M}\hat{{y}}_h,$$`

where `${W}$` is an `$n \times n$` positive definite matrix, and `$\hat{{y}}_h$` contains the `$h$`-step forecasts of `${y}_{T+h}$` given data to time `$T$`.

.small[
- Different choices for `${W}$` lead to different solutions such as Ordinary Least Squares (OLS), Weighted Least Squares (WLS) and Minimum Trace (MinT). 
- We use the implementation of these methods in the `hts` package in R in the experiment.
]

---
## Producing probabilistic forecasts

- We use bootstrapping to generate probabilistic forecasts:

- Suppose that `$(\hat{{y}}_h^{[1]},\dots,\hat{{y}}_h^{[B]})$` are a set of `$B$` simulated sample paths, generated independently from the models used to produce the base forecasts. 
    - Then `$({S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h^{[1]},\dots,{S}({S}'{W}^{-1}{S})^{-1}{W}^{-1}\hat{{y}}_h^{[B]})$` provides a set of reconciled sample paths, from which percentiles can be calculated.