class: center, middle, title-slide # Significance, Thresholds<br/>and Decision-Making ## Joe Thorley ### 2019-05-09 .pull-right[  ] --- ## Uncertainty Our understanding of nature is incomplete. --  .footnote[ &copy; NASA ] -- And always will be. --- ## Decisions Yet we must act. -- background-image: url(19-significance_files/blindfolded-158204_640.png) --- <div id="htmlwidget-e14065940f174fa01c5c" style="width:500px;height:500px;" class="grViz html-widget"></div> <script type="application/json" data-for="htmlwidget-e14065940f174fa01c5c">{"x":{"diagram":"digraph {\n\ngraph [layout = \"neato\",\n outputorder = \"edgesfirst\",\n bgcolor = \"white\"]\n\nnode [fontname = \"Helvetica\",\n fontsize = \"10\",\n shape = \"circle\",\n fixedsize = \"true\",\n width = \"0.5\",\n style = \"filled\",\n fillcolor = \"aliceblue\",\n color = \"gray70\",\n fontcolor = \"gray50\"]\n\nedge [fontname = \"Helvetica\",\n fontsize = \"8\",\n len = \"1.5\",\n color = \"gray80\",\n arrowsize = \"0.5\"]\n\n \"1\" [label = \"Data\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1,1!\"] \n \"2\" [label = \"Action\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,-1!\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script> --- <div id="htmlwidget-2d106e2a961135e20e97" style="width:500px;height:500px;" class="grViz html-widget"></div> <script type="application/json" data-for="htmlwidget-2d106e2a961135e20e97">{"x":{"diagram":"digraph {\n\ngraph [layout = \"neato\",\n outputorder = \"edgesfirst\",\n bgcolor = \"white\"]\n\nnode [fontname = \"Helvetica\",\n fontsize = \"10\",\n shape = \"circle\",\n fixedsize = \"true\",\n width = \"0.5\",\n style = \"filled\",\n fillcolor = \"aliceblue\",\n color = \"gray70\",\n fontcolor = \"gray50\"]\n\nedge [fontname = \"Helvetica\",\n fontsize = \"8\",\n len = \"1.5\",\n color = \"gray80\",\n arrowsize = \"0.5\"]\n\n \"1\" [label = \"Data\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1,1!\"] \n \"2\" [label = \"Model\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,1!\"] \n \"3\" [label = \"Likelihood\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1.5,0!\"] \n \"4\" [label = \"Prior\", shape = \"rectangle\", color = \"grey\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2.5,0!\"] \n \"5\" [label = \"Posterior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,-1!\"] \n \"6\" [label = \"Action\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,-1!\"] \n\"1\"->\"3\" [color = \"black\"] \n\"2\"->\"3\" [color = \"black\"] \n\"3\"->\"5\" [color = \"black\"] \n\"4\"->\"5\" [color = \"grey\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script> --- ## Posterior Probability Distribution <img src="19-significance_files/figure-html/unnamed-chunk-3-1.svg" style="display: block; margin: auto;" /> -- The uncertainty is fully described by the posterior probability distribution --- <div id="htmlwidget-b5bec35f848cc07990e6" style="width:500px;height:500px;" class="grViz html-widget"></div> <script type="application/json" data-for="htmlwidget-b5bec35f848cc07990e6">{"x":{"diagram":"digraph {\n\ngraph [layout = \"neato\",\n outputorder = \"edgesfirst\",\n bgcolor = \"white\"]\n\nnode [fontname = \"Helvetica\",\n fontsize = \"10\",\n shape = \"circle\",\n fixedsize = \"true\",\n width = \"0.5\",\n style = \"filled\",\n fillcolor = \"aliceblue\",\n color = \"gray70\",\n fontcolor = \"gray50\"]\n\nedge [fontname = \"Helvetica\",\n fontsize = \"8\",\n len = \"1.5\",\n color = \"gray80\",\n arrowsize = \"0.5\"]\n\n \"1\" [label = \"Data\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1,1!\"] \n \"2\" [label = \"Model\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,1!\"] \n \"3\" [label = \"Likelihood\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1.5,0!\"] \n \"4\" [label = \"Prior\", shape = \"rectangle\", color = \"grey\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2.5,0!\"] \n \"5\" [label = \"Posterior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,-1!\"] \n \"6\" [label = \"Action\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,-1!\"] \n\"1\"->\"3\" [color = \"black\", color = \"black\"] \n\"2\"->\"3\" [color = \"black\", color = \"black\"] \n\"3\"->\"5\" [color = \"black\", color = \"black\"] \n\"4\"->\"5\" [color = \"grey\", color = \"grey\"] \n\"5\"->\"6\" [color = \"black\", style = \"dotted\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script> --- ## Posterior Probability Distribution <img src="19-significance_files/figure-html/unnamed-chunk-5-1.svg" style="display: block; margin: auto;" /> The posterior is typically summarized by a 95% confidence/credible interval (CI) -- which is typically summarized in terms of its significance (is/isn't) -- in this case it **isn't**. --- ## Significance <img src="19-significance_files/figure-html/unnamed-chunk-6-1.svg" style="display: block; margin: auto;" /> -- Sample size determines significance. -- Impact assessment equates non-significance with no-effect. -- Violates precautionary principle. -- Data collection is disincentivized. --- <div id="htmlwidget-33934d8bdcff3a003625" style="width:500px;height:500px;" class="grViz html-widget"></div> <script type="application/json" data-for="htmlwidget-33934d8bdcff3a003625">{"x":{"diagram":"digraph {\n\ngraph [layout = \"neato\",\n outputorder = \"edgesfirst\",\n bgcolor = \"white\"]\n\nnode [fontname = \"Helvetica\",\n fontsize = \"10\",\n shape = \"circle\",\n fixedsize = \"true\",\n width = \"0.5\",\n style = \"filled\",\n fillcolor = \"aliceblue\",\n color = \"gray70\",\n fontcolor = \"gray50\"]\n\nedge [fontname = \"Helvetica\",\n fontsize = \"8\",\n len = \"1.5\",\n color = \"gray80\",\n arrowsize = \"0.5\"]\n\n \"1\" [label = \"Model\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,1!\"] \n \"2\" [label = \"Data\", shape = \"rectangle\", color = \"grey\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1,1!\"] \n \"3\" [label = \"Likelihood\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1.5,0!\"] \n \"4\" [label = \"Prior\", shape = \"rectangle\", color = \"grey\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2.5,0!\"] \n \"5\" [label = \"Posterior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,-1!\"] \n \"6\" [label = \"Confidence\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"3,-1!\"] \n \"7\" [label = \"Significance\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"4,-1!\"] \n \"8\" [label = \"Action\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,-1!\"] \n\"1\"->\"3\" [color = \"black\"] \n\"2\"->\"3\" [color = \"grey\"] \n\"3\"->\"5\" [color = \"black\"] \n\"4\"->\"5\" [color = \"grey\"] \n\"5\"->\"6\" [color = \"grey\"] \n\"6\"->\"7\" [color = \"grey\"] \n\"7\"->\"8\" [color = \"grey\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script> -- > 'uncertainty laundering' Gelman (2016) --- ## Power Analysis Calculate the expected uncertainty in the estimated effect. <img src="19-significance_files/figure-html/unnamed-chunk-8-1.svg" style="display: block; margin: auto;" /> -- Set a minimum sample size. -- Implicit threshold. --- <div id="htmlwidget-75a612c2dc6b6d0aceaf" style="width:500px;height:500px;" class="grViz html-widget"></div> <script type="application/json" data-for="htmlwidget-75a612c2dc6b6d0aceaf">{"x":{"diagram":"digraph {\n\ngraph [layout = \"neato\",\n outputorder = \"edgesfirst\",\n bgcolor = \"white\"]\n\nnode [fontname = \"Helvetica\",\n fontsize = \"10\",\n shape = \"circle\",\n fixedsize = \"true\",\n width = \"0.5\",\n style = \"filled\",\n fillcolor = \"aliceblue\",\n color = \"gray70\",\n fontcolor = \"gray50\"]\n\nedge [fontname = \"Helvetica\",\n fontsize = \"8\",\n len = \"1.5\",\n color = \"gray80\",\n arrowsize = \"0.5\"]\n\n \"1\" [label = \"Model\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,1!\"] \n \"2\" [label = \"Data\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1,1!\"] \n \"3\" [label = \"Likelihood\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1.5,0!\"] \n \"4\" [label = \"Prior\", shape = \"rectangle\", color = \"grey\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2.5,0!\"] \n \"5\" [label = \"Posterior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,-1!\"] \n \"6\" [label = \"Confidence\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"3,-1!\"] \n \"7\" [label = \"Significance\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"4,-1!\"] \n \"8\" [label = \"Action\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,-1!\"] \n\"1\"->\"3\" [color = \"black\"] \n\"2\"->\"3\" [color = \"grey\"] \n\"3\"->\"5\" [color = \"black\"] \n\"4\"->\"5\" [color = \"grey\"] \n\"5\"->\"6\" [color = \"grey\"] \n\"6\"->\"7\" [color = \"grey\"] \n\"7\"->\"8\" [color = \"grey\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script> --- ## Prior Information Incorporate existing knowledge that impact likely to be negative. <img src="19-significance_files/figure-html/unnamed-chunk-10-1.svg" style="display: block; margin: auto;" /> --- ## Prior Information Incorporate existing knowledge that impact likely to be negative. <img src="19-significance_files/figure-html/unnamed-chunk-11-1.svg" style="display: block; margin: auto;" /> -- Incentivizes informative data collection. --- <div id="htmlwidget-80f704ca613253ed770b" style="width:500px;height:500px;" class="grViz html-widget"></div> <script type="application/json" data-for="htmlwidget-80f704ca613253ed770b">{"x":{"diagram":"digraph {\n\ngraph [layout = \"neato\",\n outputorder = \"edgesfirst\",\n bgcolor = \"white\"]\n\nnode [fontname = \"Helvetica\",\n fontsize = \"10\",\n shape = \"circle\",\n fixedsize = \"true\",\n width = \"0.5\",\n style = \"filled\",\n fillcolor = \"aliceblue\",\n color = \"gray70\",\n fontcolor = \"gray50\"]\n\nedge [fontname = \"Helvetica\",\n fontsize = \"8\",\n len = \"1.5\",\n color = \"gray80\",\n arrowsize = \"0.5\"]\n\n \"1\" [label = \"Model\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,1!\"] \n \"2\" [label = \"Data\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1,1!\"] \n \"3\" [label = \"Likelihood\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1.5,0!\"] \n \"4\" [label = \"Prior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2.5,0!\"] \n \"5\" [label = \"Posterior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,-1!\"] \n \"6\" [label = \"Confidence\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"3,-1!\"] \n \"7\" [label = \"Significance\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"4,-1!\"] \n \"8\" [label = \"Action\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,-1!\"] \n\"1\"->\"3\" [color = \"black\"] \n\"2\"->\"3\" [color = \"black\"] \n\"3\"->\"5\" [color = \"black\"] \n\"4\"->\"5\" [color = \"black\"] \n\"5\"->\"6\" [color = \"grey\"] \n\"6\"->\"7\" [color = \"grey\"] \n\"7\"->\"8\" [color = \"grey\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script> --- ## Precautionary Principle Test whether under a threshold. -- <img src="19-significance_files/figure-html/unnamed-chunk-13-1.svg" style="display: block; margin: auto;" /> --- <div id="htmlwidget-31f47f388e32b395d03e" style="width:500px;height:500px;" class="grViz html-widget"></div> <script type="application/json" data-for="htmlwidget-31f47f388e32b395d03e">{"x":{"diagram":"digraph {\n\ngraph [layout = \"neato\",\n outputorder = \"edgesfirst\",\n bgcolor = \"white\"]\n\nnode [fontname = \"Helvetica\",\n fontsize = \"10\",\n shape = \"circle\",\n fixedsize = \"true\",\n width = \"0.5\",\n style = \"filled\",\n fillcolor = \"aliceblue\",\n color = \"gray70\",\n fontcolor = \"gray50\"]\n\nedge [fontname = \"Helvetica\",\n fontsize = \"8\",\n len = \"1.5\",\n color = \"gray80\",\n arrowsize = \"0.5\"]\n\n \"1\" [label = \"Model\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,1!\"] \n \"2\" [label = \"Data\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1,1!\"] \n \"3\" [label = \"Likelihood\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1.5,0!\"] \n \"4\" [label = \"Prior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2.5,0!\"] \n \"5\" [label = \"Posterior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,-1!\"] \n \"6\" [label = \"Confidence\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"3,-1!\"] \n \"7\" [label = \"Significance\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"4,-1!\"] \n \"8\" [label = \"Action\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,-1!\"] \n\"1\"->\"3\" [color = \"black\"] \n\"2\"->\"3\" [color = \"black\"] \n\"3\"->\"5\" [color = \"black\"] \n\"4\"->\"5\" [color = \"black\"] \n\"5\"->\"6\" [color = \"grey\"] \n\"6\"->\"7\" [color = \"grey\"] \n\"7\"->\"8\" [color = \"black\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script> --- ## Statistical Decision Theory Choose option that maximizes expected net benefit given the uncertainty. -- Requires loss function (challenging to develop) -- but criteria are explicit. -- and decisions optimal. --- <img src="19-significance_files/figure-html/unnamed-chunk-15-1.svg" style="display: block; margin: auto;" /> Expected loss = 0.73 - 0.4 = 0.33 <img src="19-significance_files/figure-html/unnamed-chunk-16-1.svg" style="display: block; margin: auto;" /> Expected loss = 0.07 - 0.4 = -0.33 --- <div id="htmlwidget-bbfe5c68f32951c9ac09" style="width:500px;height:500px;" class="grViz html-widget"></div> <script type="application/json" data-for="htmlwidget-bbfe5c68f32951c9ac09">{"x":{"diagram":"digraph {\n\ngraph [layout = \"neato\",\n outputorder = \"edgesfirst\",\n bgcolor = \"white\"]\n\nnode [fontname = \"Helvetica\",\n fontsize = \"10\",\n shape = \"circle\",\n fixedsize = \"true\",\n width = \"0.5\",\n style = \"filled\",\n fillcolor = \"aliceblue\",\n color = \"gray70\",\n fontcolor = \"gray50\"]\n\nedge [fontname = \"Helvetica\",\n fontsize = \"8\",\n len = \"1.5\",\n color = \"gray80\",\n arrowsize = \"0.5\"]\n\n \"1\" [label = \"Data\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1,1!\"] \n \"2\" [label = \"Model\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,1!\"] \n \"3\" [label = \"Likelihood\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"1.5,0!\"] \n \"4\" [label = \"Prior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2.5,0!\"] \n \"5\" [label = \"Posterior\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"2,-1!\"] \n \"6\" [label = \"Action\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,-1!\"] \n \"7\" [label = \"Options\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,1!\"] \n \"8\" [label = \"Loss\", shape = \"rectangle\", color = \"black\", width = \"0.75\", fillcolor = \"#F0F8FF\", fontcolor = \"#000000\", pos = \"5,0!\"] \n\"1\"->\"3\" [color = \"black\"] \n\"2\"->\"3\" [color = \"black\"] \n\"3\"->\"5\" [color = \"black\"] \n\"4\"->\"5\" [color = \"black\"] \n\"5\"->\"6\" [color = \"black\"] \n\"7\"->\"8\" [color = \"black\"] \n\"8\"->\"6\" [color = \"black\"] \n}","config":{"engine":"dot","options":null}},"evals":[],"jsHooks":[]}</script> ---  --- ## References Amrhein, V., Greenland, S., and McShane, B. 2019. Scientists rise up against statistical significance. Nature 567(7748): 305–307. doi:10.1038/d41586-019-00857-9. Diaz, S. 2019. Summary for policymakers of the global assessment report on biodiversity and ecosystem services of the Intergovernmental Science-Policy Platform on Biodiversity and Ecosystem Services. Gelman, A. 2016. The problems with p-values are not just with p-values. The American Statistician 70: 10. Williams, P.J., and Hooten, M.B. 2016. Combining statistical inference and decisions in ecology. Ecological Applications 26(6): 1930–1942. doi:10.1890/15-1593.1. ---- .footnote[ Thorley 2019. Significance, Thresholds and Decision-Making. A presentation at Regulated Rivers II: Science, Restoration, and Management of Altered Riverine Environments. Columbia Mountains Institute. Nelson, BC. <https://www.joethorley.io/slides/19-significance#1>. ]