#---
#title: "Lab 2A"
#subtitle: "Feature space expansions"
#date: "`r format(Sys.time(), '%Y')`"
#output: 
#  html_document:
#    toc: true
#    toc_depth: 2
#    toc_float: true
#    df_print: paged
#    number_sections: false
#    theme: paper
#    highlight: tango
#---


```{r message = FALSE}
knitr::opts_chunk$set(message=FALSE)
library(ggplot2)
library(gamlss)
library(splines)
library(rpart)
library(caret)
```


---

In this lab you visualize data, and fit polynomials, splines and regression trees. You need the packages `ggplot2`, `gamlss`, `splines`,  `rpart` and `caret`. To get reproducible results, set the seed to 1 before you begin.

```{r}

```

# BMI


The data set for this exercise is `gamlss::dbbmi` (also see lecture slides). Check the help page of this data set for a description of the data.

a. Display a scatterplot with `age` on the x-axis and `bmi` on y-axis.

```{r}

```



b. Display the previous plot with the regression line of a polynomial model. Experiment with the `degree` of the polynomial until you a find a satisfactory compromise between fit and model complexity (the optimal bias-variance trade-off).

```{r}

```


c. Now display the previous plot with the regression lines of both a cubic and a natural cubic spline. Experiment with the `df` and compare the regression lines for equal `df`. What is the difference?

```{r}

```

d. Even for large values of `df` it is difficult to get good predictions for the ages close to 0. You may get better results by experimenting with the position of the knots. Try to get a better result by fitting a cubic spline with a single, well placed knot.

```{r}

```

# Fortune500 data 

The data set `Fortune500` for this exercise are on the 500 richest companies in the USA. 

a. Load the data in workspace, either indirectly by first saving it on your machine or directly with the command `load(url("https://maartencruyff.github.io/S31/2A-Feature Space Expansion/data/Fortune500.Rdata"))`.

```{r}

```

b. Visualize the relationship  in a scatterplot with `Assets` on the y-axis and `Profits` on the x-axis. Based on this plot, what seems to be the best choice for the polynomial degree of `Assets`?

```{r}

```


The aim is to select the best model for predicting `Assets` from `Profits`. The candidates are polynomial models (linear, quadratic, cubic and quartic), a regression tree, and _k_-nearest neighbors. To simplify matters, we do not create a training and test, but we perform a 10-fold cross-validation for each model using the entire sample, and select the model with the lowest cross-validated RMSE.



## Polynomial models

We start with testing the performances of the linear, quadratic, cubic and quartic models in the traditional way, i.e. by testing significance of the  R^2-change when increasing the polynomial degree. So we do not cross-validate, but we just test the R^2 change of the four models with the function `anova()`.   

a. Fit each polynomial model in ascending order of the polynomial degree within the `anova()` function. A significant result means that the higher-order polynomial fits better than the lower-order one. Which polynomial model comes out as the best?

```{r}

```

b. Reproduce the previous scatter plot, and add the regression line of the polynomial model (in the color blue) that came out as best from the ANOVA test. How do would you interpret this model in terms of bias and variance?

```{r}

```




c. The method we just used is highly data-driven, and thus prone to overfitting. We will now use cross-validation to select the best model. Fit the four models separately with the function  `train()` using 10-fold cross-validation, and compare the cross-validated RMSEs. Which models comes out as best now?

```{r}

```


d. Reproduce the previous scatter plot, and add the regression line of the polynomial model (in red) that came out as best in the cross-validation. How do would you interpret this model in terms of bias and variance?

```{r}

```

## Regression trees

A regression tree partitions the variable `Profits` in on-overlapping regions, and assigns mean of `Assets` within each region as predicted value, 

a. Use the function `rpart()` (check its help page), and fit a tree to the data using the "anova" method. Save the fitted object as `tree`.

```{r}

```



b. Plot the `tree` object, and interpret the plot.

```{r}

```

c. Trees tend to overfit, so we will now perform cross-validation on the `tree` object to find the optimal value for the hyperparameter `cp`, which controls the complexity of the tree. You can do this with the `train()` function with `method = rpart`. Save the object as `tree_cv`.


```{r}

```


d. Obtain the predictions of `tree_cv` witht he function `predict()`, and save them as `pred_tree_cv`.

```{r}

```


e. Reproduce the previous scatter plot, and add the regression line of of the tree (in brown).


```{r}

```


## KNN

KNN assigns the mean of the _k_ nearest neigbors as prediction, where _k_ is a hyperparameter to be determined with cross-validation.

b. Perform a 10-fold cross-validation on the data with the `train()` function to determine the optimal value fo _k_. Save the object as `knn_cv`. 

```{r}

```



b. Obtain the fitted values for the training data with `predict()`, as save them as `pred_knn_cv`.

```{r}

```


f. Reproduce the previous scatter plot, and add the KNN regression line of the tree (in green).


```{r}

```

g. Which of the fitted models do you prefer, and why?


---

END OF LAB
