Monte Carlo Testing

Introduction

Monte Carlo testing integrates cross-validation, sensitivity testing, and uncertainty evaluation into a single function. This allows you to select which types of testing you want to do and even mix and match them. We recommend you start with running a Monte Carlo scenario without any changes to the settings. This will run the model just as if you were running it without any randomness involved.

Cross-Validation

To perform cross-validation, change the "Percent Training" to a value like 70%. Your occurrences will be broken into two sets, one for testing with 70% of the data and one for training with 30% of the data. Increase the iterations to see the effect this has. If your model changes wildly between each iteration, it is not robust and you are probably in need of additional data.

Occurrence Uncertainty Evaluation

All data includes some uncertainty including any set of occurrences and covariates. For occurrences, this includes the locations of the coordinates. The occurrence coordinates have some error in the x and y locations. We can inject noise into the occurrence coordinates to simulate collecting data over and over again and then create habitat models that show the areas were have have high confidence there is habitat and areas where there is low confidence.

Set the "Percent Training" back to 100 and go to the "Occurrence Noise Injection" tab. To add some noise into your occurrences, or "jiggle" them, set the "Distribution" to "Normal" and change the value of standard deviation. This value will depend on your data and is in "map" or "reference" units such as meters or feet. If you are using a handheld GPS, these values will probably be around 10 meters. You can test this by taking repeated samples over the course of a few weeks and at different times of day at the same location. Then, just estimate the distribution of the data and set the standard of deviation using this value. When you run Monte Carlo, it will inject noise into the occurrence coordinates as if you revisited the locations over and over again. Note that if you have data from an unknown source you may have much wider standard of deviation for your data.

Covariate/Predictor Uncertainty Evaluation

All of our covariates have some level of uncertainty. In most cases, this is not documented and we must estimate it. The uncertainty will have some type of distribution, a normal curve for most remotely sensed data. You can also specify a uniform distribution or provide your own files with noise that was generated from a distribution specifically designed for each covariate (for the BioClim models we will be providing noise files in the near future).

On the "Covariate Noise Injection" tab, you can add noise to each of your covariate layers with settings similar to the occurrences or as a raster. In the first case, random values will be added to each pixel in the covariate as if we were repeatedly sampling the same area and obtaining different pixel values based on variances by day, sun angle, etc. If your data was down sampled to a lower resolution, you can use the distribution of each pixel in the higher resolution (i.e. the standard deviation of the pixels that made up each pixel in the covariate raster) as the uncertainty raster. In other cases, you may be able to find a "surrogate" raster to estimate the uncertainty for each pixel. One example would be to use a higher resolution elevation raster to simulate part of the uncertainty that occurs in climate rasters as the climate will vary based on changes in elevation.

After running these three Monte Carlo methods, you'll want to combine them and evaluate your model based on occurrence uncertainty, covariate uncertainty, and cross-validation. Increase the values and see if your model is truly stable. The number of iterations is important and 100 to 1000 may be required to have a stable output. To determine an appropriate number, run your model with 10 and then 20 iterations and check the "Cumulative" graphs at the bottom of the output. These graphs will show when the model performance measures stabilize. You may need to have a lot of disc space available for the intermediate files and may want to run this for several days.

Sensitivity Testing

Sensitivity Testing is the final stage in Monte Carlo testing. This is where we take an existing model and inject noise into it's parameters (for HEMI, this is the location of the control points on the model curve) and then see how well the model fits the data. A model that is not sensitive, or is "robust", will still fit the data well if a small amount of noise is added (i.e. the controls are moved a bit). A sensitive model will "fall apart" when the parameters are changed and this is an indication that the model may not hold up under different scenarios, especially moving the model to predict habitat in another area.

After you have a model you are confident in, use the Sensitivity Testing tab to inject some noise into the parameters of the model. This will shift the control points randomly to see if the model is robust to small changes. If you've done the steps above, this should show that the model is very robust.

Results

Settings

This is simply a table of the overall settings that you used to run Monte Carlo. After each run, you may want to copy the "Temp" file and give it a good name so you can come back to it later.

Input Data

This is a table of the covariates and the noise you injected into each of them.

Overall Results

The Monte Carlo features allow you to create an assortment of statistics and maps for your models. These appear in the HTML output and are stored in a folder location that is specified in the settings dialog. The results include:

• Mean habitat: This is the average raster for all the habitat models created during a run and will be your "best" model
• Uncertainty Map: This shows the areas were there is uncertainty about habitat, typically along the edges of a species habitat.
• Minimum habitat: This shows the minimum habitat area predicted and can be used as the area with the highest confidence of habitat. This map can be used for purposes like setting up a refuge for a species.
• Maximum habitat: Shows the maximum predicted habitat and should be used for activities such as monitoring for invasive species where we don't want to miss any habitat.

Overall ROC - the mean, 95% confidence interval, minimum, and maximum values for the Receiver Operator Curve (ROC) graph.

Statistical Distributions - histograms showing the distribution of the AUC, AIC, and Loglikelyhoods for all the model runs. These graphs should show a normal distribution if normal noise injection was used. With a uniform distribution, the statistical distributions may be more uniform. If enough iterations are used, the graphs should be similar from one run to another.

Model Graphs (HEMI 2 Only)

Graphs for each of the models for each covariate with the mean, 95% confidence interval, minimum, and maximum values for the model for each covariate value.

Jackknife Results (HEMI 2 Only)

Jackknifing refers to trying every possible combination of covariates in different models. Because of how HEMI 2 computes performance measures, these can be computed very quickly and are provided in this table. This approach can help determine which combinations of covariates should be used for final models.

Note: It is not advised to simply use the combination of covariates that has the lowest AIC or highest AUC. The recommended approach is to use a set of covariates that make sense based on the life cycle of the species and that do not correlate highly with each other.

The first table shows the performance of the model with each combination of covariates. The second table shows how the model performed with each covariate independently and how the model performed with each covariate removed from the set.

Statistics for Each Iteration

The table in the last section shows how the model performed on each iteration. You can click on the elements in the first column to examine the model for each iteration in detail.

Below the table are graphs that show the cumulative results for each of the performance measures. These graphs should be stable to the right when enough iterations have been run.

Below is information each of the settings organized by the tabs they are in.

Overall Settings

Number of Iterations: This is the number of repetitions that will be executed. The more iterations you execute, the more accurate your results will be but the longer it will take. It is recommended to start small (e.g. 5 to 10) and then increase the number and execute the analysis for several days if needed (1000 iterations is more than enough).

Percent Training: The percent of your occurrences that will be used for model fitting. The remaining occurrences will be used to test the model and develop the fit statistics.

Fit Model: Typically, you'll want to fit the model on each iteration. If you are close to having a final model, turn off the fitting and run sensitivity testing.

Send Uncertainty Map to Scene: This will send the final uncertainty map to the scene where you can continue the evaluation.

Output HTML: Outputs a set of HTML pages with details on the Monte Carlo run and the results of each iteration.

Open Browser with Results: When checked, your default web browser will open showing the HTML pages with your results.

Temp Folder: A temporary folder for the HTML Pages and other results. This may need to be on a hard disk with a lot of empty space if you are running a large number of iterations.

Occurrence Noise Injection Settings

This panel controls the amount of noise injected into the coordinate values for the occurrences. This effectively "shifts" the coordinates to represent the error introduced by GPS handhelds and other means of collecting coordinates. By shifting the occurrence coordinates, and then fitting the model, we can create multiple models representing going out into the field repeatedly to collect data.

Distribution: "None" will not inject any noise, "Normal" injects noise based on a normal distribution, "Uniform" injects noise based on a uniform distribution while "File-Based" allows you to specify a file with noise values to inject.

The following settings will appear based on the distribution selection.

Standard Deviation: The standard deviation of the normal curve used for normal distributions. This value should be estimated based on knowledge of the data collection. For GSP Handhelds it should be between 1 meter and 20 meters. For data created from products such as Geomancer, it should be much higher and may be several miles.

Mean: A mean of the normal distribution. This value is typically 0.

Minimum: The minimum value for the uniform distribution option.

Maximum: The minimum value for the uniform distribution option.

Note that the amount of noise injected should be set based on the expected error from the method used to capture the occurrence coordinates and in the units of the current occurrence spatial reference (i.e. in meters for UTM data and feet for State Plane data with linear units of feet). Note that geographic data should not be used because degrees are not uniform in direction as we move away from the equator.

Covariate Noise Injection Settings

This panel shows the options for injecting noise into the pixel values of the covariate rasters. The table contains rows for each covariates and columns for the settings. The controls in the left three columns control noise injection based on settings for all the pixels in each covariate. The three column on the right allow you to injection different amounts of noise for each pixel in a covariate.

For some covariates, you will have the same amount of uncertainty for each pixel. This includes remotely sensed data from airplanes and satellites that is at its original resolution. Without additional information, you can only estimate the amount of uncertainty, or "pixel mixing" that exists within each pixel. For these types of data, we recommend just setting estimated values, typically using a normal distribution, for the parameters.

For other covariates, you may have information that allows you to create rasters with the amount of uncertainty present in each pixel. If you down sample remotely sensed data to a lower resolution, you can find the standard deviation of the data (or min/max) in each pixel by performing analysis on the collection of pixels that overlap with each pixel in the down sampled raster. For data that was created with interpolation methods such as Kriging, you can use the uncertainty raster generated as the inputs for noise injection.

Note: It is rare that you will have a value for the "mean" that is non-zero. If you do not specify a value for the mean, scalar or raster, 0 will be used. Also, the noise will be in the units of the pixel values for each covariate and should be adjusted accordingly.

Sensitivity Testing Settings (HEMI 2 Only)

Sensitivity testing will check how sensitive your model is if the model parameters (for HEMI 2, the control point locations) are moved slightly. This is typically done after a model has been evaluated with all other means and you feel you have a robust model.

Because the left and right control points in the models in HEMI 2 are limited to moving along the sides and bottom of the chart, their possible range of values is a little unique.

• Left most control point
  • -1 at the top-left corner of the chart (x=0, y=1)
  • 0 at the bottom-left corner of the chart (x=0, y=0)
  • 1 at the bottom-right corner of the chart (x=1,y=0)
• Right most control point
  • 0 at the bottom-left corner of the chart (x=0, y=0)
  • 1 at the bottom-right corner of the chart (x=1,y=0)
  • 2 at the top-right corner of the chart (x=1, y=1)

Since all of the points have a maximum range of 2, the maximum values for the sensitivity testing should be within2 of each other. Typically you'll select a mean of 0 and a standard of deviation of less then 1 for normally distributed noise and a minimum of greater than -1 and a maximum of less than 1 for uniformly distribution noise. The noise will be generated for each control point on each model for each iteration. If the noise places the points outside their valid range, the point will be moved to is maximum allowed value in that direction.

Validation Settings

Percent Training for Cross-Validation - This is the percent of the data that will be retained for training (70% is common). The remaining data is used for testing.

Validation Data Set - Using a completely separate data set is preferred to cross-validation. If a separate data set is specified, it will be used for the "test" data set.