Insufficient Data

The challenge of insufficient data occurs when not enough data was collected to answer the question of interest (Table 2).

In the example the result is no longer significant (p = 0.1; Figure 2). Reasons for insufficient data include a lack of appreciation of the variation in the data.

Figure 2. Example insufficient data analysis.

Missing Data

Although it can also result in a failure to answer the question of interest, missing data is different from insufficient data in the sense that it should have been collected but wasn’t (Table 3; Figure 3). The presence of missing data often requires special analytic techniques. Reasons for missing data include equipment failure and crew inattention.

Figure 3. Example missing data analysis.

Biased Data

Although biased data also results from non-collection of data it is different from missing data in that the sense that the gaps are systematic (Table 4).

As well as presenting the same challenges as missing data, biased data can also result in incorrect estimates even though the data points are themselves correct. Thus the regression model now estimates the slope to be 1.68 (Figure 4).

Biased data can result from non-random subsampling decisions.

Figure 4. Example biased data analysis.

Erroneous Data

Like biased data, erroneous data can result in incorrect estimates although the problem is due to invalid data points as opposed to systematic gaps (Table 5). Equipment malfunction and crew inexperience are common causes of erroneous data. In the current example the slope is now estimated to be -2.19 (Figure 5)!

Figure 5. Example erroneous data analysis.

Messy Data

Wickham (2014) formally defined tidy data to be that for which:

1. Each variable forms a column.
2. Each observation forms a row.
3. Each type of observational unit forms a table.

Messy data is by definition any other arrangement of the data (Table 6). Tidy data is easy to manipulate, model and visualize. Messy data must first be tidied.

Undocumented Data

Undocumented data is that for which no metadata exists (Table 7). The absence of metadata can prevent any sort of analysis or worse yet result in an inappropriate analysis being performed (Figure 6).

Figure 6. Example undocumented data analysis.

Lost Data

Lost data can be defined as data that was collected but no longer exists in paper or electronic form (Table 8).

Data Solutions

Fortunately most of these data-related challenges can be overcome through recognition of the potential problems and a study design that includes where necessary a pilot study to identify all the data challenges; power analyses to ensure sufficient data are collected; equipment redundancy and specialized data collection and data entry forms so that the data are complete; field protocols that ensure unbiased subsampling; crew training and equipment redundancy so the data points are valid; a relational database so that the information is documented and tidy; and long-term data curation budgets so the information is archived for future use.

Analytic Challenges

There at least six types of analytic challenge

1. vague questions
2. hidden assumptions
3. derived indices
4. pseudo-replication
5. over-reliance on significance testing
6. researcher degrees of freedom

Vague Questions

An example of a vague question is

Does discharge affect fish abundance?

The question is obviously vague because it does not specify the species, life-stage or location (population). However nor does it specify the aspect of the hydrograph of interest (Olden and Poff 2003) identified 171 discharge metrics!) or the period of concern. It is also vague because it doesn’t specify how much of an effect is to be considered important.

Hidden Assumptions

Hidden assumptions are those that reseachers are not aware they are making. An example would be to collect fish abundance data at index sites and then use it to draw river-wide conclusions. The hidden assumption is that changes in fish density at the index sites are representative of changes at the other sites. Often this is not the case.

Once recognised hidden assumptions sometimes become heroic assumptions: assumptions that are unlikely to be true but which are necessary in order to draw any meaningful conclusions.

Derived Indices

Derived indices are values which are calculated from other values without consideration of the underlying sampling distributions. For example the catch-per-unit-effort (CPUE) is

$$CPUE = \frac{Catch}{Effort}$$

However, analysing $CPUE$ (as opposed to $Catch$ and $Effort$) precludes

1. use of a readily interpretable distribution, i.e., $Poisson$ for $Catch$
2. accounting for reduced uncertainty when lower effort
3. testing for catch depletion

Pseudo-replication

Pseudo-replication occurs when replicates are not statistically independent, which is almost all the time in ecological studies! When determining the effect of discharge on fish abundance non-independence can occur due to repeated measures, individual longevity, recruitment, aggregations, invasive species, downstream effects and climatic effects on fish abundance and discharge.

Over-Reliance on Significance Testing

Often an effect is judged solely on its significance. However significance alone is a poor criterion because significant and insignificant effects can be large or small (Figure 7). A large insignificant effect is potentially much more important than a small significant one.

Figure 7. Various effect sizes.

Researcher Degrees of Freedom

Researcher degrees of freedom is an umbrella term for

all the data-processing and analytical choices researchers make after seeing the data.

A classic example is when researchers are tasked with identifying the environmental predictor variable(s) responsible for changes in the response. In the case of discharge and fish abundance the number of predictors can stretch into the tens or even hundreds. If 20 time series are considered the expectation is that one will predict the response by chance.

If researchers decisions are undocumented then the extent to which the probability of getting a false positive has been inflated cannot be assessed.

Analytic Solutions

Most of the analysis-related challenges can be overcome through hierarchical (account for non-independence), Bayesian scientific (explicitly describe observational and biological processes) models that allow the estimation of secondary parameters (answers to specific questions) with credible intervals (effect sizes) and the release of version-controlled code (researcher decisions).