2024 SDP Conference

2024 SDP Annual Conference, In-Person, April 16-18

Presentations

Day 1

Welcome & Keynote Address

April 16, 2024 8:30am - April 16, 2024 9:30am

"Presentation not available"

Keynote Speaker: Dave Snowden

Abstract:

How do you make effective decisions when prior analysis is not possible and in fact, may even be dangerous? Dave Snowden will share cutting-edge complexity-appropriate approaches based on his decades of experience in weak signal detection, foresight and decision support. Explore cases at the intersection of complex decision-making and community engagement, as well as new methods for building resilient organizations, communities and countries. Rather than traditional anticipatory methods, Dave will share strategies to build responsive decision processes with the capability to respond quickly in a crisis, including anticipatory alerts to draw decision-makers attention to points when small interventions could make a massive difference.

Government Track

Data in Government Decision Making

The Build American Center -UMD MTI |   A collaboration with US Department of Transportation for collection of data on DoT projects, research, AI, etc.

April 16, 2024 10am - April 16, 2024 10:45am

Click here to download PDF copy of presentation

Speaker: Professor Qingbin Cui (University of Maryland Project Management)

Abstract:

The presentation showcases diverse application cases within transportation agencies that harness data for informed decision-making, striving to attain overarching project and program objectives. This encompasses factors such as time and cost performance, public sentiments, considerations of social justice, and the impact on climate. A particular emphasis will be placed on elucidating the significance of unstructured project data in enhancing project risk management.

US Air Force and Space Force Contracting Information Systems (CON-IT)

April 16, 2024 10:45am - April 16, 2024 11:15am

Click here to download PDF copy of presentation

Speaker: William "Schatten" Douglas (United State Air Force)

Government Accountability Office: Best Practice Guides

April 16, 2024 11:15am - April 16, 2024 11:45am

Click here to download PDF copy of presentation

Speaker: Jennifer Leotta (GAO)

Abstract:

In 2005, the Government Accountability Office (GAO) began to develop and use best practice guides to assess project controls for Federal programs. We currently, have four guides: The Cost Estimating and Assessment Guide (GAO-20-195G), the GAO Schedule Assessment Guide (GAO-16-89G), the Technology Readiness Assessment Guide (GAO-20-48G), and the GAO Agile Assessment Guide (GAO-24-105506). This presentation will discuss the process we use to develop these guides, a brief overview of the best practices in each guide, and a discussion of how these guides help us obtain the necessary data to assess Federal programs.

New to DQ Track

What is a Decision Professional?

April 16, 2024 10am - April 16, 2024 10:30am

Click here to download PDF copy of presentation

Speaker: Ellen Coopersmith (Decision Frameworks)

Abstract:

What does it mean to be a decision professional?

The society is committed to creating a great decision every time, but how do we actually do that? Three professional themes tie us together:Decision Leadership: Serving the human decision maker as they make choices that shape the future – taking full responsibility for both finding good answers but also doing so in a way that responds to human foibles, fears and motivations.

  • Decision Quality: It is easy to second guess a choice in hindsight, but much harder to know you’ve made a good decision at the time you make it. We have (and improve) measurements and standards for decision quality.
  • Decision Engineering: We are practitioners who use a robust toolkit of proven methods to produce better decisions. We find the right tools and assemble them to meet the requirements of the situation, which spans supporting specific decisions to adjusting organizational processes to improve decision making.

The Fundamentals of DQ and How did we get here?

April 16, 2024 10:30am - April 16, 2024 11:30am

Click here to download PDF copy of presentation

Speaker: Carl Spetzler (Strategic Decisions Group)

Abstract:

Carl will present the body of knowledge and skills that Decision Professionals have to master. He will then show how this field evolved from the Decision Analysis paradigm as it was conceived by Ron Howard in 1964. This session will be followed by classic application examples that illustrate how DQ adds value to complex decision situations.

Q&A Session

April 16, 2024 11:30am - April 16, 2024 12pm

Moderator: Terry Karner

Panelist: Carl Spetzler & Ellen Coopersmith

Advancing the Profession Track

Applied Superforecasting

April 16, 2024 10am - April 16, 2024 10:40am

Click here to download PDF copy of presentation

Speaker: Warren Hatch (Good Judgement Inc.)

Abstract:

Every decision is a forecast: we take actions that we believe will improve the probability of a desired outcome. As a process, Superforecasting can measurably improve the accuracy of individuals and teams while helping to improve the flow of information within organizations. This presentation will review new research findings on the forecasting frontier and illustrate their application with use cases in the private and public sectors.

How can we use generative AI to assist in building decision models?

April 16, 2024 10:40am - April 16, 2024 11:15am

Click here to download PDF copy of presentation

Speaker: Max Henrion (Lumina)

Abstract:

How can we use generative AI to assist in building decision models?

Large Language Models are radically changing our ideas of what software can do -- from traditional tools that are reliable but need complete and specific quantitative specification to new tools that can work with ill-defined qualitative problems but are unreliable and prone to errors (hallucinations) -- somewhat like us. How can we use them effectively to help define and structure complex decision problems? I'll give some examples with surprising results, in terms of what they can do, with surprising benefits and challenges.

DECISION LEADERSHIP

April 16, 2024 11:15am - April 16, 2024 12pm

Click here to download PDF copy of presentation

Speaker: David Matheson (SmartOrg)

Abstract:

One definition of a leader is someone who creates a future that is not the momentum of the past. As decisions get further from “momentum”, they increasingly require attention to “heart” issues: how do you motivate strong action in a potentially frightening direction? Traditional DQ, which focuses on “head” issues – how to find a good answer – is only part of the story, and inadequate to guide leaders or mobilize real commitment among their followers in these situations. This powerful new lens on decision making will increase your connection as an ally to decision leaders and improve your ability to diagnose situations and guide decision processes. Traditional boundaries blur, such as the idea that there is a clear decision maker or that execution follows a decision, in a way that increases the insight and impact of the decision professional. While the society has developed a good foundation in this area, it is just the beginning: complex decision-intensive problems such as the energy transition or digitization of the organization require this leadership lens. We have considerable room to build diagnostic and prescriptive tools for dealing with these leadership issues.

Government Track

Panel Discussion

Case Study: Australian Defense Industry Case Study: Transparent and collaborative R&D funding prioritisation to support R&D investment decision making in the face of high uncertainty, poor data and multiple stakeholders

April 16, 2024 1pm - April 16, 2024 1:30pm

Click here to download PDF copy of presentation

Speaker: Paul Gordon (Catalyze)

Abstract:

The presentation showcases diverse application cases within transportation agencies that harness data for informed decision-making, striving to attain overarching project and program objectives. This encompasses factors such as time and cost performance, public sentiments, considerations of social justice, and the impact on climate. A particular emphasis will be placed on elucidating the significance of unstructured project data in enhancing project risk management.

State of Government Data Collection

April 16, 2024 1:30pm - April 16, 2024 1:50pm

Click here to download PDF copy of presentation

Speaker: John Driessnack (University of Maryland College of Performance Management)

New to DQ Track

Classic Application Cases

How SmithKline Beecham Makes Better Resource Allocation Decisions

April 16, 2024 1pm - April 16, 2024 1:30pm

Click here to download PDF copy of presentation

Speaker: Tom Keelin (Keelin Reeds Partners)

Abstract:

In one of our profession's most consequential DA/DQ projects, SmithKline Beecham (SB--prior to its acquisition by Glaxo) retooled its decision processes, reallocated resources across its 20 late-stage development projects in four therapeutic areas on two continents, and shifted significantly more resource into R&D. Shareholder value added of better decision-making, compared to the decisions that would have been made otherwise, was more than a billion dollars. Focusing on the soft issues (like building credibility and trust} and introducing new methodology (like considering project alternatives at the portfolio level) enabled SB to more effectively address the hard ones -- how much and where to invest. The subsequent Harvard Business Review (HBR) article became one of HBR's most widely distributed. Widespread adoption and adaptation of the approach in pharmaceutical, energy, and other industries ultimately led to downstream value creation in dozens of major companies and job creation for decision professionals.

Unleaded Amoco

April 16, 2024 1:30pm - April 16, 2024 2pm

Click here to download PDF copy of presentation

Speaker: Carl Spetzler (Strategic Decisions Group)

Abstract:

Lead additives to enhance the octane level of gasoline were still legal. However, Standard Oil of Indiana (later Amoco) had acquired an oil company that produced high octane unleaded “white” gas. Should we introduce such unleaded gas nationwide? This was a huge decision with highly uncertain consequences. It had already been analyzed a number of times and a new president wished to resolve the question once and for all. He commissioned a major study team that evaluated 11 scenarios in great depth. Then added a Decision Analysis in the last month. This provided a great comparison between traditional methods and DA.

Rigs to Reefs: From controversy to consensus on decommissioning California's offshore oil platforms.

April 16, 2024 2pm - April 16, 2024 2:30pm

Click here to download PDF copy of presentation

Speaker: Max Henrion (Lumina Decision Systems)

Abstract:

Since the 1969 Santa Barbara oil spill, anything about offshore oil is controversial in California, even what to do with defunct oil platforms. I'll describe how a multidisciplinary team developed an interactive decision analysis model using a multi-attribute value function that enabled conflicting stakeholders to understand the options and arrive at agreement about a solution. This project won the Decision Analysis Practice Award in 2014.

R&D Project Strategy

April 16, 2024 2:30pm - April 16, 2024 3pm

Click here to download PDF copy of presentation

Speaker: David Matheson (SmartOrg)

Abstract:

Whether or not you work in innovation, innovation decisions are a critical part of a decision professional’s repertory because they deal with issues that appear in many decision types: driving upside, option thinking, staged investments and tremendous levels of uncertainty and ambiguity. Though concrete examples, such as HP’s foray’s into LightScribe, we will illustrate how common methods, like strategy tables, range assessments, tornado diagrams and probability assessments, can be deployed and adapted to support these types of choices. The result usually isn’t merely a “better” choice, the very act of analyzing in this way gives teams a roadmap that helps them improve outcomes.

Advancing the Profession Track

Deep Adoption of DQ

April 16, 2024 1pm - April 16, 2024 1:30pm

Click here to download PDF copy of presentation

Speaker: Carl Spetzler (Strategic Decisions Group)

Abstract:

Applying DQ to strategic questions has proven extremely valuable – but it is too infrequent to change an organization’s culture. With deep adoption, DQ becomes part of the language and culture for all decisions: “That’s how we make decisions around here”. In this session we will explore how we can achieve deep adoption of DQ and what benefits accrue from this adoption.

AI & Data Science, Friend or Foe?

April 16, 2024 1:30pm - April 16, 2024 2pm

Click here to download PDF copy of presentation

Speaker: John Mark Agosta (Applied A.I Researcher)

Abstract:

Is it more valuable to make millions of 1 dollar decisions or one 1 million dollar decision? Automating decisions in software opens a new frontier in Decision Science. It is another proving ground for Decision Quality (DQ).

Don’t get confused by what “AI” was and what has it now become. AI’s origins have common roots with DQ in the pursuit of rationality. The siren song of current generative AI language models and their appearance of “common sense” do not replicate rational decision making. This talk explains how the value modelling familiar to Decision Analysts can be integrated to complement Data Science’s predictive modelling, to solve the so-called “alignment” problem. Then speculatively, I broach how this new world of automation raises an interesting moral question in light of Newcomb’s paradox.

Achieving Decision Quality More Efficiently-- Historical Perspective and Emerging Trends

April 16, 2024 2pm - April 16, 2024 2:30pm

Click here to download PDF copy of presentation

Speaker: Tom Keelin (Keelin Reeds Partners)

Abstract:

Achieving a great decision every time is the gold standard of our profession, while how to best do so remains the perennial challenge. Different decisions require different processes, analytical approaches, and tools-- all of which have steadily improved since the emergence of decision analysis as a profession 60 years ago. This talk will address each link in the decision quality chain from the standpoint of how substantive improvements have evolved to form the current state-of-the-art and how other substantive improvements are emerging to enable practitioners to achieve decision quality faster and better.

Day 2

Energy Track

Factoring probabilistic evaluations in traditionally qualitative multi-attribute decisions, such as plant turnarounds

April 17, 2024 8:30am - April 17, 2024 9:15am

Click here to download PDF copy of presentation

Co-Presenting: Luis Mendoza (Decision Frameworks), Ellen Coopersmith & Ashley Corso (Decision Frameworks)

Abstract:

What is the true cost of a plant turnaround? How can we better inform our decision when selecting a plant turnaround strategy?

Plant turnarounds are scheduled plant shutdowns in which preventive maintenance and equipment upgrades are conducted in order to ensure safe and compliant operation, improve competitiveness, etc. Teams are often tasked to think strategically about which improvements, beyond the required maintenance, will better position the company to gain competitive advantage and thrive in changing market conditions. However, this strategic thinking often focuses on qualitative multi-attribute analysis to reach consensus and ignores probabilistic quantitative evaluations to better illustrate the trade-offs between competing strategies.

The cost of the turnaround involves the operating cost of executing the maintenance itself and unplanned (surprise) fixes that only become apparent when the plant/factory is shutdown, as well as the capital cost of equipment replacements and upgrades. Furthermore, a significant cost is in the disruption of operations and loss of profits during the shutdown as well as the degradation in operating efficiency over time between turnarounds.

Multi-attribute analysis is an important tool in framing compelling alternatives and assessing them qualitatively. This paper/presentation focuses on the value to be gained by enhancing traditionally qualitative decisions with quantitative analysis using a simple model with well-defined uncertainties and decisions and iterating the permutations using decision analysis software to gain insight into the quantifiable trade-offs.

Structuring Portfolio Analysis and Management to Achieve High Decision Quality

April 17, 2024 9:15am - April 17, 2024 10am

Click here to download PDF Copy of Presentation

Speaker: Tony Kenck (Practical Portfolio Management)

Abstract:

Implementing portfolio management in an organization, whether as a one-off study or an ongoing practice, is challenging. One daunting aspect involves selecting the best choice of portfolio analysis method.

Multiple definitions of portfolio analysis exist—whether standalone analysis, ranking by a single variable, multi-axis graphs and bubble charts, simulation, or optimization all the way to stochastic optimization. I will describe and explain a framework tied to decision quality to help drive productive discussions and expectations.

Twenty-plus years ago, many portfolio practitioners believed that only three factors were needed to enable portfolio management: data, computing power, and visualization. My framework adds two more elements: frame and management commitment. The framework is similar to the SDG six-factor DQ framework, but alternatives smear across data and the computing model (logically correct reasoning). Visualization is an integral part of values and trade-offs.

I discuss a novel decision tree to help determine the best tool to fulfill the needs of the frame of the question or problem to be addressed. Along the way, I talk about different types of analysis and attempt to put a reasonable frame around what portfolio analysis and management mean.

In addition, it is essential to distinguish between the needs of an individual effort and developing tools and a process for ongoing work. If you are tasked with “doing portfolio,” you need clarity on expectations.

Pharma Track

Looking beyond the 95%: A New Data-Driven Framework to Assess Regulatory Approval Risk

April 17, 2024 9am - April 17, 2024 9:30am

Click here to download PDF copy of presentation

Speakers: Lan Ding (GSK) and Co-presenting Eric Johnson (GSK)

Abstract:

Regulatory approval plays a pivotal role in drug development, yet reliance on aggregate industry approval rates can mask compound-specific risks. To guide more informed decision-making, GSK pioneered a robust framework that systematically captures regulatory risks and opportunities on a customized, per program basis.

Spearheaded by Lan, a cross-functional team spanning statistics, decision science and global regulatory leaders overcame organizational barriers to design the new evaluation framework. Implementation required persuading stakeholders to adopt more granular, data-driven risk evaluation despite fundamental changes to existing processes.

The session will detail learnings from the 12-month journey. How do you drive adoption of value-added but disruptive change? What objections were raised against augmenting models perceived as sufficient? How did emphasizing decision quality over motives shift mindsets? The team strived for enhancements offering better capital allocation and patient impact – come hear how.

Natural Resources Track

Organizational Alignment and Optimization of Resource Allocation to Conservation Goals

April 17, 2024 8:40am - April 17, 2024 8:55am

Click here to download PDF copy of presentation

Authors: Ellen Pero, Riley Andrade, Randy Wilson, Laura Brandt, Todd Jones-Farrand, Conor McGowan

Abstract:

Efficient use of assets in natural resource conservation is important given increasing demand (i.e., conservation need) and limited funding. Allocating organizational resources across large federal conservation agencies can be challenging due to large geographic scopes of responsibility, multiple and potentially competing agency objectives, mandated legal constraints, as well as uncertainty in outcomes relative to variable resource inputs and personnel performance. We adopted a values-based, structured decision-making approach and PrOACT framework with the Science Application and Migratory Bird Program (SAMB) within the US Fish & Wildlife Service’s (USFWS) Region 4 to develop an organization-wide objective hierarchy and programmatic metric set to align organization expenditures and actions toward conservation goals. We used expert elicitation to inform a stochastic, constrained-optimization model to provide decision-support for allocation of agency full-time equivalents (FTEs) to maximize SAMB conservation value in the southeast US. We highlight sensitivity of the decision-support tool to shifting agency priorities, and we demonstrate additional tool applications relative to evaluation of SAMB conservation value in the southeast under relaxed legal constraints, shifting FTE allowances, and alternative organizational priorities.

Prioritization of Species Status Assessments for Decision Support

April 17, 2024 8:55am - April 17, 2024 9:10am

Click here to download PDF copy of presentation

Authors: Ashley Goode, Erin Rivenbark, Jessica Gilbert, Conor McGowan

Abstract:

Species status assessments are used to inform U.S. Fish and Wildlife Service (USFWS) decision-making for Endangered Species Act (ESA) classification decisions, recovery planning, and more. The large number of species that require assessment and uncertainty in the data available impede the process of assigning and completing the assessments, which makes creating a multi-year work plan extremely difficult. An optimized triaging system that maximizes the use of the best available information, while managing the complex ESA workload and meeting deadlines, is necessary. We used a structured decision-making framework to approach the problem with the goal of creating a prioritization tool that would be effective at scheduling assessments, given the best information available and priorities of the Service. We collected data on the species awaiting assessment and developed a value function that incorporates existing deadlines, taxonomic uncertainty, and controversy of the species, and population and habitat data availability and quality. We used a constrained linear optimization algorithm to maximize the value function and ensure that workload capacity was not exceeded. Comparison of model scenarios indicates that imposed deadlines impact the model more than capacity constraints. Additionally, differential weighting of the metrics significantly affected the outcome of the model. In the future, elicitation of metric weights should be done routinely before the model is run for use in official planning to ensure 3 alignment with current USFWS priorities. Output from this optimization can be used to inform a five-year work plan, allocate resources, and discuss workforce decisions.

Optimizing Conservation Actions to Recover Sensitive Species Across Maui Nui

April 17, 2024 9:10am - April 17, 2024 9:25am

Click here to download PDF copy of presentation

Authors: Melissa R. Price, Kristen C. Harmon, Abbey Camaclang, Tara Martin, Scott Fretz

Abstract:

With hundreds of species at risk and on the brink of extinction, conservation practitioners must decide where to focus effort and which conservation actions to implement given limited funds. Decision processes that aim to maximize conservation benefit for a given cost should address complementarity of actions to ensure that species in low-diversity habitats are not excluded, and both cost and effectiveness of different actions across taxonomic groups. To address this need we modified a Priority Threat Management approach to guide resource allocation decisions for the conservation of biodiversity in Maui Nui (the islands of Maui, Moloka‘i, Lāna‘i, and Kahoʻolawe). Over a series of online meetings and in-person workshops, species experts and conservation managers contributed: (1) key threats to sensitive species; (2) management strategies to address key threats; and (3) expected cost, feasibility, and benefit of management strategies. Elicited data were analyzed to identify strategies that would provide 3 optimal gains in recovery across multiple taxonomic groups given costs and feasibility. Predator control and fencing were both identified as cost-efficient actions with the greatest gains in recovery across taxonomic groups, but those actions alone were not effective at recovering many plants and invertebrates. Participants emphasized the importance of investing in research and development of novel techniques to address persistent problems such as avian malaria, pests, and diseases. Further, despite the high initial cost of landscape-scale control of invasive species, participants highlighted the importance of long-term benefits. Findings from this study will improve both the efficient use of existing funds, and competitiveness for increased resources needed to achieve recovery.

Visualizing and communicating 'Decision Space' - the role of risk assessments in informing management decisions

April 17, 2024 9:25am - April 17, 2024 9:40am

Click here to download PDF copy of presentation

Author: Jonathan Cummings

Abstract:

In the absence of information about risk and without any consideration of risk tolerance, when making a risk management decision, individuals could reasonably select any action from a set of possible actions. Their 'decision space' is not in any way limited in this circumstance. However, a risk assessment or a determination of risk tolerance should make some possible actions less reasonable than others, which constricts the decision space available to decision makers. To help decision makers understand the role of a risk assessment and risk tolerance in risk management decisions, I developed an interactive R Shiny application that displays reasonable decision space. The decision space is modified by inputs of risk assessment (from scientific assessments or expert elicitation) and inputs of risk tolerance (from policy makers or decision makers). My visualization tool aims to help decision makers understand the role of scientific analyses that produce risk assessments, as well as the role of policy that produces risk tolerance in defining the decision space and the best actions to take given the decision context.

Energy Track

The practical and accessible application of Game Theory to strategic decision making

April 17, 2024 10:30am - April 17, 2024 11:15am

"Presentation not available"

Speaker: Dave Debacker (Open Options) & Christa Roemkens (Chevron)

Abstract:

This presentation will focus on the practical application of Game Theory to complex, multi-stakeholder, strategic decision-making. This session will be applicable to both technical and non-technical streams/attendees.

Using past project examples, Christa & Dave will share their thoughts on the experience and insights generated by using a preference-based game theory approach when dealing with highly complex, intractable problems. This involves using proven analytical methods and tools to consider the actions and reactions of key stakeholders - these could be competitors, regulators, customers, partners, suppliers, or any other party with the ability to materially impact a situation. It is important to note that this is a qualitative approach driven by challenging and constructive discussion, not cash flow modeling. The speakers will touch on the unique advantages and benefits in terms of efficiency, mitigating bias, managing uncertainty, aligning decision-making teams, and increasing decision speed. They will also explain how the synchronous and asynchronous elements work together and leverage the complementary roles of the client organization and the external consultant.

While the presentation will be based on specific examples, the speakers will pay special attention to calling out the parallels and analogies that make the process and lessons learned applicable to virtually all forms of complex, multi-stakeholder problems, regardless of industry or geography. Throughout, they will highlight how Game Theory-based decision-making fundamentally aligns with and supports Decision Quality (DQ).

Using Conversational Swarm AI for Optimized Group Decision Making

April 17, 2024 11:15am - April 17, 2024 12pm

Click here to download PDF copy of presentation

Speaker: David Baltaxe (Unanimous AI)

Abstract:

Making smart decisions is critically important to every organization. So how can we use AI to improve the decision-making process? Some companies are using generative AI to replace people with data- and algorithmic-driven decisions, but this takes people out of the loop and often leads to suboptimal results. At the 2023 SDP conference, we showcased a new technology called Swarm AI, which integrates humans and AI and connects groups of people together and amplifies their combined intelligence. Based on the remarkable properties of biological swarms, Swarm AI empowers networked human teams to quickly tap into their combined knowledge, wisdom, and insight to generate more accurate forecasts, estimations, and predictions. This leads to better, smarter decisions. The newest iteration of Swarm AI, called Conversational Swarm AI (CSI), represents a dramatic improvement. Unlike previous versions, CSI allows groups of any size to converse naturally using text chat and allows for answering open-ended questions and solving unstructured problems. This talk will review how CSI works and exciting commercial applications for the energy sector.

Pharma Track

Pharmaprojects: How it can be used from Business to Research

April 17, 2024 10:30am - April 17, 2024 11am

Click here to download PDF copy of presentation

Speaker: Harrison Miller (Alnylam Pharmaceuticals)

Abstract:

Probabilities of success are a key assessment for decision analytic strategy and portfolio models in drug development. Various studies have analyzed historical data using different methodologies and different data sets, but many of these are significantly out of date or are based on limited data sets. This talk will review an attempt to reproduce a landmark paper with a comprehensive, up-to-date data set, and will highlight several key learnings about the nuances, methodologies, data sets, changes over time, and how/when this information can be best leveraged in the context of drug development decision making.

Simple Bayesian Reference Class Forecasting for Binary and Continuous Business Variables

April 17, 2024 11am - April 17, 2024 11:30am

Click here to download PDF copy of presentation

Speaker: Shaun Comfort (Genentech, Inc.)

Abstract:

Various industries generate forecasts to facilitate portfolio investments and decision-making. Typical examples in pharma include binary probability of success estimates for clinical trials and peak sales estimates. The vast majority of these forecasts are generated using subject matter experts and/or quantitative models using detailed information about the project(s) under consideration. From the Heuristics and Biases perspective, these forecasts are based on an “inside view” of projects resulting in predictions that are non-regressive and miscalibrated, relative to actual outcomes. In 1977, Daniel Kahneman and Amos Tversky published a paper outlining a simple corrective procedure for combining inside-view forecasts with distributional data from relevant outcomes, to produce plausible forecasts that are closer to actual results. This approach termed has been termed “reference class forecasting” Flyvbjerg (2011). For this presentation, Dr. Comfort presents a simple Bayesian reformulation of Kahneman and Tversky’s approach to update inside-view forecasts with prior information to produce posterior probability distributions for binary and continuous business variables. He illustrates this with a practical example from his recent article in Foresight Issue #72 (https://forecasters.org/foresight/issues/) to generate posterior estimates of clinical program probability of success and 5-year peak revenues, from initial inside-view forecasts.

Platform Trial Designs for Sequential Treatment Evaluation

April 17, 2024 11:30am - April 17, 2024 12pm

Click here to download PDF copy of presentation

Speaker: Alex Kaizer (University of Colorado Anschutz Medical Campus)

Abstract:

The potential strengths of platform trials have been elucidated for oncology, infectious diseases, and other settings. In this talk we first present the lessons learned from a sequential platform trial used in the West Africa Ebola virus disease outbreak and how these may translate to the context of oncology research. For example, one shortcoming of the original design was that supplemental information from controls in previous trial segments was not utilized. We address this limitation by proposing an adaptive design methodology that facilitates information sharing within the trial. The design also allows the use of multisource adaptive randomization to target information balance within a trial segment if other segments of data are incorporated. Compared to the standard design, we demonstrate that MEMs with adaptive randomization can improve power with limited type-I error inflation. We conclude by discussing some of the future directions and potential applications of newer methodologies and applications for platform trial designs that both include and extend the lessons learned previously.

Natural Resources Track

Institutionalizing Collaboration and Multi-objective Decision Making in Water Resources Planning Processes within the Corps

April 17, 2024 10:30am - April 17, 2024 10:50am

Click here to download PDF copy of presentation

Authors: Michelle Hilleary, Hal Cardwell, Michael Deegan, Igor Linkov, Chris Cummings, Kat McCain

Speakers: Michelle Hilleary & Hal Cardwell (U.S. Army Corps of Engineers)

Abstract:

Collaboration in water resources management in the U.S. Army Corps of Engineers (USACE) has benefited from sustained strategic investment and network nurturing. Recent policy directives to look more comprehensively across categories of benefits are moving the Corps to employ more multi-objective decision making techniques in the tradeoffs analysis step of the Civil Works planning process. Applying collaborative techniques in tradeoffs analyses will facilitate better decision-making and increased realization of benefits in the public sphere.

The Corps’ Institute for Water Resources (IWR) was created in 1969 to analyze and anticipate changing water resources management conditions, and to develop planning methods and analytical tools to address economic, social, institutional, and environmental needs in water resources planning and policy. In 2008, the Collaboration and Public Participation Center (CPCX) was named a Corps Center of Expertise within IWR. CPCX is an authoritative source for senior Corps leaders, government agencies, civilian, and international leaders from related industries to study and confer on the tactical application of water resource management and conflict resolution. CPCX was a response to the growing need for alternative methods of dispute resolution in the management of our nation’s waters. USACE's recognition of the need for collaboration, partnering, and public participation in water resources decision making has been documented in both guidance and strategic plans.

CPCX provides technical assistance to USACE Districts and Divisions on collaborative processes, builds USACE collaborative capacity, publishes reports on environmental conflict resolution and collaborative processes, and manages the USACE’s Collaboration and Public Participation Community of Practice. CPCX’s work is focused on its four goals of capacity building, direct services, policy support, and innovative processes. CPCX improves the outcomes of USACE missions by supporting collaborative processes and ensuring that the interests of partners, stakeholders, and the public are addressed.

In accordance with the 2021 Comprehensive Benefits Memo, Corps study teams are directed to “analyze benefits in total and equally across a full array of benefits categories”. These public benefit categories include economic (national and regional), environmental (national and regional), and other social benefits that are valued by the federal government, non-federal sponsor, public, and interested stakeholders. These “buckets” of public benefits along with the guiding principles from the Principles, Requirements, and Guidelines (PR&G) will help inform evaluation and decision-making for water resources investments. Identifying plans that maximize net total benefits will require comparison in performance among various objectives and criteria. Decision analytic techniques may help provide transparency and clarity to distinguish between input of value preferences and technical information in informing a decision. Stakeholders involved in a planning study will bring their own values and preferences; therefore, using structured tradeoffs methods will be vital to both identify the plan that maximizes net public benefits and provide clarity and transparency on the eventual selection of the recommended plan.

Expanding tools and perspectives to include ecosystem service concepts in Superfund site management decisions

April 17, 2024 10:50am - April 17, 2024 11:05am

Click here to download PDF copy of presentation

Authors: Leah M. Sharpe, Matthew C. Harwell, Jim Harvey, Tammy Newcomer-Johnson, Gina Ferreira, Stephanie Kim, Bruce Pluta

Speaker: Leah Sharpe (US Environmental Protection Agency)

Abstract:

The U.S. Environmental Protection Agency’s (EPA) Superfund program is responsible for the assessment, cleanup, and reuse of some of the most contaminated sites in the United States. The primary goal of Superfund work is to protect human health and the environment and the work itself is regulatorily and legislatively proscribed. While the concept of ecosystem services (ES; i.e., the benefits that humans receive from nature) is not a part of Superfund processes, the EPA identified potential connections between ES concepts and remediation and redevelopment of contaminated sites in 2009 and has been applying those concepts in cleanups for over a decade. In addition to regulatory hurdles, the organizational separation of those developing ES tools and those responsible for managing Superfund sites was another challenge to the incorporation of ES concepts in management actions. An ongoing collaboration between tool developers, modelers, risk assessors, and project managers, co-led by researchers focusing on ES tools and risk assessors focused on contaminated site management, has made meaningful progress in incorporating ES concepts and tools in contaminated site cleanup and reuse. These advances include developing generic guidelines for incorporating ES into ecological risk assessments, identifying the value-added aspects provided by incorporation of ES, developing ES tools in response to manager needs, identifying concrete steps to support Superfund staff in effective incorporation of ES tools into their work processes, and conducting multiple case studies as practical demonstrations of this incorporation. This presentation will discuss how this collaboration works, the co-development of decision tools and frameworks, and how the incorporation of ES concepts impacted outcomes in case study examples.

Structured Decision-Making Framework for Managing Cyanobacterial Harmful Algal Blooms in New York State Parks

April 17, 2024 11:05am - April 17, 2024 11:20am

Click here to download PDF copy of presentation

Authors: Jennifer L. Graham, Gabriella Cebada Mora, Rebecca M. Gorney, Lianne C. Ball, Claudia Mengelt, and Michael C. Runge

Speaker: Jennifer Graham (USGS)

Abstract:

Cyanobacteria increasingly are a global water-quality concern because of the potential for these organisms to develop harmful blooms that affect ecologic, economic, and public health. Cyanobacterial harmful algal blooms (CyanoHABs) can lead to a decrease in water quality and may affect the recreational and ecological benefits of parks that include lakes. The New York State Office of Parks, Recreation and Historic Preservation (OPRHP) is a state agency charged with the operation of state parks and historic sites. Many New York State parks include lakes or other freshwater bodies, which can be susceptible to CyanoHABs. The OPRHP faces difficult decisions regarding prevention of and response to CyanoHABs. Decision analysis is often used to inform complex decisions regarding natural resource management. Structured decision making (SDM) breaks down complex decisions into their basic parts and reconstructs the problem into a framework that allows for collaborative examination and development of suitable actions. The U.S. Geological Survey partnered with OPRHP and the New York State Department of Environmental Conservation to develop a SDM template for managing CyanoHABs in OPRHP parks. Two parks, Moreau Lake State Park and Rockland Lake State Park, served as case studies to motivate and test the template. This presentation will describe how the principles of SDM can be used to navigate the challenges associated with managing CyanoHABs using the case studies as examples. Management objectives and strategies for CyanoHABs in OPRHP parks, strategies to evaluate consequences and manage trade-offs, and potential challenges to the implementation of preferred alternatives will be discussed.

Resource allocation for invasive species management on National Wildlife Refuges lands in the Midwestern United States

April 17, 2024 11:20am - April 17, 2024 11:35am

Click here to download PDF copy of presentation

Authors: Angela Romito & Max Post van der Burg

Speaker: Angela Romito (U.S. Fish and Wildlife Service)

Abstract:

A team comprised of United States Fish & Wildlife Service National Wildlife Refuge System (US FWS NWRS) biologists and refuge managers worked collaboratively to develop a transparent, coordinated process to allocate annual invasive species funding, better leverage relevant resources, and reduce the impact of non-native species, in turn, facilitating achievement of biological integrity, species diversity, and environmental health across the Midwest US FWS NWRS. They constrained their work to reflect refuge capacity for project implementation, variability in station priorities, adherence to funding directives, and relevant law and policy. Five fundamental objectives were identified by the team including, 1) maximizing the condition of refuge species of concern and their associated habitats, 2) project success, 3) efficiency, 4) maximizing the number of refuges that benefit from allocation and, 5) partner and landowner acceptance. To assess how well proposed projects performed relative to these objectives, measurable performance measures (15 total) were identified for each objective. A request for proposal (RFP) system was used to solicit projects from 70 Refuge Complexes and Wetland Management Districts across the Midwest NWRS. To reduce administrative burden, and subjectivity in proposal review, the RFP was designed (via Microsoft Forms) to allow proposal authors to score their projects according to the criteria selected by the model development team. The decision aid was used to produce four project portfolios, along with predictions of their expected conservation benefits, for consideration by regional decision-makers. The optimal portfolio resulted in the most projects funded and the highest cumulative conservation benefit. Decision-makers collectively chose to allocate funds to the projects in the optimal portfolio. The decision tool, along with the RFP system, will be used to allocate these funds in future years.

Energy Track

The Challenges and Lessons of Developing a Sustainable Decision-Making Culture

April 17, 2024 1pm - April 17, 2024 1:45pm

Click here to download PDF copy of presentation

Co-Presenting: Christa Roemkens, Mike Benefiel & Ellen Vernotzy (Chevron)

Abstract:

Creating a sustainable decision making culture is a continuous work-in-progress. Just as businesses and industries evolve to adapt to changing drivers, so the decision making culture must adapt too. It is shaped as much by a company's specific culture as by the trends in general and project management philosophy. Inevitably, components put in place with the best intentions take on a life of their own and can become obstacles rather than enablers of efficient and effective decision making. The response is often to blame the tools, rather than how they are being used - leading to a cycle of periodic upheaval that seldom leads to the desired improvements. How can we break this cycle? (Spoiler alert - there is no silver bullet).

Bridging the Gap – Translating Behavioral Economics Research to Daily Practice

April 17, 2024 1:45pm - April 17, 2024 2:30pm

Click here to download PDF copy of presentation

Speaker: Alison Zabik (Exxon Mobil)

Abstract:

As Decision professionals, we are designers. We craft and shape the elements of decision quality to support decision makers and stakeholders, in pursuit of organizational objectives. Decision professionals are obligated therefore to design and customize decision processes in support of the human needs of the decision maker and in consideration of the organizational behavior context of the enterprise.

This talk will translate behavioral economics literature into a decision quality practice. Focus will be on practical techniques that incorporate academic insights, such as mental accounting, time discounting and choice bracketing, into the day-to-day practice of decision quality. This talk will also briefly touch on organizational context ideas, translating insights from public policy literature to applications of decision quality practice in a multinational oil and gas context.

Pharma Track

Learning about Clinical Trials with Causal Inference and Causal Discovery

April 17, 2024 1pm - April 17, 2024 1:30pm

Click here to download PDF copy of presentation

Speaker: Gary Summers (GSK)

Abstract:

We’ve all heard that statistic’s mantra, “Correlation does not equal causation.” However, causal inference, a new set of statistical techniques, can identify causal relationships from data. Amazingly, it can derive experimental results and counterfactuals from observational data. DAGs, directed acyclic graphs, similar to influence diagrams, are the heart of causal inference. Using ten years of publicly available clinical trial data, Intelligencia and GSK are creating a DAG and studying causation in clinical trial designs, a prelude measuring the impact of decision choices, such as how enriching trials with patient selection biomarkers impact a trial’s probability of success. This presentation will introduce causal inference, DAGs, and our early results.

Dynamic Framing for Strategic Games in Biopharma

April 17, 2024 1:30pm - April 17, 2024 2pm

Click here to download PDF copy of presentation

Speakers: Steve Galatis & Paul Papayoanou (Decision Frameworks)

Abstract:

In many situations, the decisions of biopharma companies are interdependent with those of competitors and partners. Co-marketing and licensing deals, and other M&A transactions, require negotiation of win-win agreements. Launch strategy, pricing and market access, and in-market positioning, meanwhile, are best developed by thinking through complex action-reaction dynamics and key uncertainties. For such situations, game theory is the most appropriate tool, and Strategic Gaming is a practical application that is an extension of traditional decision analysis. When we frame issues and opportunities with Strategic Gaming, we need to capture the dynamics of interactions between two or more companies. The Dynamic Framing approach involves development of a game tree that incorporates the decisions of each of the key “players” and often the resolution of key uncertainties as well. We illustrate this approach with a case example in which a company can choose to build up its own sales force or negotiate a licensing agreement with another biopharma company.

Craftsmanship in Pharmaceutical Portfolio Management

April 17, 2024 2pm - April 17, 2024 2:30pm

Click here to download PDF copy of presentation

Speaker: Henry Conter (Roche)

Natural Resources Track

Roles and limitations of artificial intelligence and machine learning for the management of natural resources

April 17, 2024 1pm - April 17, 2024 1:25pm

Click here to download PDF copy of presentation

Speaker: Julien Martin (US Geological Survey)

Abstract:

Applications of artificial intelligence (AI) and Machine learning (ML) techniques to ecological research and management are growing rapidly. These techniques have the potential to increase the efficiency of data collection and analyses of ecological data to help inform management decisions. In this session, we will discuss the benefits and limitations of these methodologies for natural resource management. More specifically, we will examine the roles and limitations of AI and ML in the context of structured decision making. Structured decision making (SDM) for natural resource management is a method for analyzing a decision by decomposing the problem into components, which include: (1) management objectives; (2) potential management actions; (3) models to project the consequences of actions; (4) optimization methods and (4) monitoring. We will talk about the benefits that AI/ML can bring in the context of SDM, with a particular focus on the following components: models, optimization and monitoring. Examples of successful applications of AI/ML include image analysis and the processing of acoustic data to identify species or matching individuals. In turn this identification process can be used to model vital rates, abundance and distribution of organisms. The accounting of imperfect detection and misclassification errors is important for reliable inference from data collected or processed with AI/ML. We will describe SDM and adaptive management cases studies that could benefits from AI/ML techniques. Finally, we will discuss the challenges and limitations of applying AI/ML to natural resource management.

Which came first: the data or the decision? How AI/ML helps the USGS deliver decision-agnostic water data that informs every link in the decision quality chain

April 17, 2024 1:25pm - April 17, 2024 1:45pm

Click here to download PDF copy of presentation

Speaker: Katrina Alger (US Geological Survey)

Abstract:

The USGS has long been synonymous with delivering accurate and reliable observed water data through its extensive national networks of stream gages and hydrologic experts. The bureau has no management authority and instead partners with a wide variety of federal, state, and local partners to provide objective scientific research in the service of action. As a result, much of the data delivered by the Water Enterprise must be decision-agnostic, and yet the expectation is that its primary use will be in decision-making at multiple scales.

Congressional directives from the 2009 SECURE Water Act, combined with methodological and computational advancements, have driven USGS to expand water data delivery beyond gage measurements by providing modeled assessments of water availability for human and ecological needs, now and into the future. While some models are process-based, an increasing number are data-driven and use machine learning (ML) techniques to predict and forecast variety of metrics related to water quality, quantity, and use. Compared to point observations, models have the advantage of providing data that is spatially continuous by interpolating between observations, and can be used to understand trends and forecast future conditions. However, models constructed outside of a specific decision context can be challenging to use and interpret due to embedded assumptions or limitations of training datasets that are not always made explicit to decision-makers. Reconciling these issues with the tenets of decision quality (DQ) is challenging but imperative as ML methodology advances and becomes increasingly common in the production and delivery of data.

This talk will be both informational and aspirational by providing some examples of how the USGS is using ML models to advance understanding of the complete water cycle, describing how DQ intersects with internal efforts to establish best practices for ML model development, and offering some perspectives on how we can deliver data that empowers decision-makers by informing every link of the DQ chain.

Day 3

Energy Track

Application of Scenario Thinking for Offshore Wind Development in the US

April 18, 2024 8:30am - April 18, 2024 9:15am

Click here to download PDF copy of presentation

Speaker: Brendon Keinath (RWE Offshore Wind US)

Abstract:

In 2021, the Biden-Harris administration set a goal of deploying 30 gigawatts of offshore wind by 2030 and 110 gigawatts by 2050. The first commercial scale wind farms in the US are currently being installed off the East Coast and the industry is at a fever pitch. That said, there is still a significant amount of uncertainty around achieving these goals several decades out. In this case study, Decision Quality methods in Scenario Thinking were applied to understand key drivers (e.g., political, technical, supply chain, etc.) and develop future worlds to gain insights for circumstances where these goals are exceeded, met or missed in order to inform future planning.

Why decision quality is critical for resilient, sustainable and future-ready energy supply chains

April 18, 2024 9:15am - April 18, 2024 10am

Click here to download PDF copy of presentation

Speaker: Sri Vaidyanathan (Shell)

Abstract:

Global energy supply chains, historically driven by financial efficiency whose fragility was brought to focus during the pandemic. The pandemic and supply chain chokes of the recent years has exposed the vulnerability of these supply chains. Considered together with the need for decisive actions towards a net-zero future, structure and priorities within global supply chains are in urgent need of reconsideration.

Supply chains are essentially networks of various organizations, including suppliers, manufacturers, and retailers, aiming to deliver products or services to the end consumer. When these products cross multiple international boundaries like in complex-energy supply chains, the chain becomes global, bringing about interdependencies among the organizations.

While the laser focus on financial efficiency has offered benefits in the past, changing consumer preferences, investor and regulatory pressures will demand that together with financial efficiency decision-making within organizations must also include considerations of sustainability, human rights and other factors to push for resiliency and sustainability.

The future of supply chains should pivot towards robustness, founded on principles like sustainability, collaboration, transparency, and diversification. This is essential for tackling economic vulnerability, climate change, and societal disparities. Embracing this model could foster economic and societal well-being without compromising the environment. Recent research points towards the positive correlation between sustainability and resilience within supply chains. Collaboration, seen as a "team sport" in this context, can be immensely beneficial. Joint efforts by buyers and suppliers lead to energy supply chains being at the forefront of delivering low carbon energy solutions that the world needs.

This is where decision frameworks will be critical to drive responsible decision making. Decision frameworks in energy supply chains must be retooled from a singular focus on financial efficiency to seek win-win outcomes that balance financial efficiency with sustainability. Through collaborative approaches like proper opportunity framing and open dialogue with the supply chains, solutions can be found which are both sustainable and financially efficient. For instance, products can be fundamentally re-designed to use less materials or responsibly sourced materials. These decision frameworks can spur innovation in areas like wind turbine maintenance, designing more efficient electrolysers or sustainable fuels which are crucial for the future of energy.

However, transforming these complex supply chains isn't simple, given their historic focus on short-term gains, this is why strategic decision frameworks are crucial to drive a reshaped vision for global supply chains, emphasizing long-term collaboration and shared benefits and responsibilities. The integration of digital technology can further enhance transparency and traceability, pivotal for the chains' future success.

Pharma Track

An overview of the Descriptive Benefit-Risk Framework for New Drug Approval Decisions

April 18, 2024 8:15am - April 18, 2024 8:45am

Click here to download PDF copy of presentation

Speaker: Dr. Hong Yang (FDA, Center for Drug Evaluation and Research)

Abstract:

Medical product decision-making at FDA is a complex process involving large, multi-disciplinary teams, voluminous streams of scientific information, regulatory requirements, and diverse external stakeholders. Focusing on Agency decision-making at the time of initial marketing authorization, this session will provide an overview of the principles of benefit-risk assessment used by the three medical product centers and demonstrate how benefit-risk assessment, a subset of decision analysis, informs regulatory decision-making. Illustrative case examples showing the spectrum of decision analysis approaches used will be provided as well as information useful to product sponsors. Dr. Lackey from the Center for Drug Evaluation and Research (CDER), will provide an overview of the descriptive Benefit-Risk Framework for new drug approval decisions as well as the Benefit-Risk Guidance for Industry. This talk will present a case study showing how regulators approached the approval decision, how the principles in the Guidance are applied, and explore some key areas of concern and describe how those were resolved. Dr. Yang, from the Center for Biologics Evaluation and Research (CBER), will present a case example from a recent vaccine approval. This example will explore regulatory challenges and how the FDA Benefit-Risk Framework, real-world data and quantitative benefit-risk modeling are used to support the decision making. Dr. Gebben, from the Center for Devices and Radiologic Health (CDRH), will discuss how patient preference information is utilized by the Center as well as guidance for industry on collection and submission of preference information. This talk will highlight how patient preferences are integrated into regulatory decision-making, including the use of preference techniques (discrete choice experiments, threshold technique, etc.) to gain insights. The session will close with a panel discussion.

A Recent Vaccine Approval

April 18, 2024 8:45am - April 18, 2024 9:30am

Click here to download PDF copy of presentation

Speaker: Leila Lackey (FDA, Center for Drug Evaluation and Research)

Abstract:

Medical product decision-making at FDA is a complex process involving large, multi-disciplinary teams, voluminous streams of scientific information, regulatory requirements, and diverse external stakeholders. Focusing on Agency decision-making at the time of initial marketing authorization, this session will provide an overview of the principles of benefit-risk assessment used by the three medical product centers and demonstrate how benefit-risk assessment, a subset of decision analysis, informs regulatory decision-making. Illustrative case examples showing the spectrum of decision analysis approaches used will be provided as well as information useful to product sponsors. Dr. Lackey from the Center for Drug Evaluation and Research (CDER), will provide an overview of the descriptive Benefit-Risk Framework for new drug approval decisions as well as the Benefit-Risk Guidance for Industry. This talk will present a case study showing how regulators approached the approval decision, how the principles in the Guidance are applied, and explore some key areas of concern and describe how those were resolved. Dr. Yang, from the Center for Biologics Evaluation and Research (CBER), will present a case example from a recent vaccine approval. This example will explore regulatory challenges and how the FDA Benefit-Risk Framework, real-world data and quantitative benefit-risk modeling are used to support the decision making. Dr. Gebben, from the Center for Devices and Radiologic Health (CDRH), will discuss how patient preference information is utilized by the Center as well as guidance for industry on collection and submission of preference information. This talk will highlight how patient preferences are integrated into regulatory decision-making, including the use of preference techniques (discrete choice experiments, threshold technique, etc.) to gain insights. The session will close with a panel discussion.

How Patient Preference Information is Used by the Center for Devices and Radiological Health

April 18, 2024 9:30am - April 18, 2024 10am

Click here to download PDF copy of presentation

Author: Dr. Dave Gebben

Co-Presenting: Dr. Sara Eggers (FDA, Center for Drug Evaluation and Research)

Abstract:

Medical product decision-making at FDA is a complex process involving large, multi-disciplinary teams, voluminous streams of scientific information, regulatory requirements, and diverse external stakeholders. Focusing on Agency decision-making at the time of initial marketing authorization, this session will provide an overview of the principles of benefit-risk assessment used by the three medical product centers and demonstrate how benefit-risk assessment, a subset of decision analysis, informs regulatory decision-making. Illustrative case examples showing the spectrum of decision analysis approaches used will be provided as well as information useful to product sponsors. Dr. Lackey from the Center for Drug Evaluation and Research (CDER), will provide an overview of the descriptive Benefit-Risk Framework for new drug approval decisions as well as the Benefit-Risk Guidance for Industry. This talk will present a case study showing how regulators approached the approval decision, how the principles in the Guidance are applied, and explore some key areas of concern and describe how those were resolved. Dr. Yang, from the Center for Biologics Evaluation and Research (CBER), will present a case example from a recent vaccine approval. This example will explore regulatory challenges and how the FDA Benefit-Risk Framework, real-world data and quantitative benefit-risk modeling are used to support the decision making. Dr. Gebben, from the Center for Devices and Radiologic Health (CDRH), will discuss how patient preference information is utilized by the Center as well as guidance for industry on collection and submission of preference information. This talk will highlight how patient preferences are integrated into regulatory decision-making, including the use of preference techniques (discrete choice experiments, threshold technique, etc.) to gain insights. The session will close with a panel discussion.

Chair's Choice Track

MODA for Personal Decisions

April 18, 2024 8:30am - April 18, 2024 9am

Click here to download PDF copy of presentation

Speaker: Eric Johnson (GSK)

Abstract:

We often have many objectives when we face big personal decisions like buying a house or car. If the information and judgments are all available, the basic data and logic for a multi-objective analysis can be captured in a simple grid in a spreadsheet. But if we are in an iterative process of information framing and gathering, the considerations and links to source data may get lost in the shuffle (leading to rework), and the tool may not give high-level insights (leading to poorly focused investigation). In this talk, I show some design ideas for a MODA spreadsheet that address these issues, with examples from my own house and car purchase decisions.

Extending Good Decision Making Skills Beyond the Workplace | Facilitated Discussion with Q&A

April 18, 2024 9am - April 18, 2024 10am

Click here to download PDF copy of presentation

Co-Chairs: Amy Day (Clarity4Action.Org) & Stefanie McLaney (Decision Education Foundation)

Panel: Steve Begg (The University of Adelaide), Terry Karner (Astellas), Kuno Huisman (ASML)

Abstract:

Decision Professionals know the value of DQ in their own lives and careers, yet many would like to help others learn these effective tools. This session will bring together Decision Professionals who have acted on that impulse with encouraging results. Participants will gain insights from these professionals, who will share their experiences and methodologies in imparting decision-making skills to diverse groups such as youth, parents, and educators. The focus will be on practical applications within educational settings, community organizations, and through strategic partnerships. Additionally, the session will feature an interactive panel discussion, accompanied by Q&A designed to address specific questions, and facilitate a deeper understanding for attendees looking to adapt these practices in their respective areas of influence.

Energy Track

Leading through uncertainty in the energy sector: How organizational transformation / evolution is influencing decisions in the energy field

April 18, 2024 10:30am - April 18, 2024 11:15am

Click here to download PDF copy of presentation

Speaker: George Nsoribe (Chevron)

Abstract:

With the global clamor for lower carbon emissions, huge volatility in oil market prices, cost of exploration and production, technology changes and political factors in regions where oil is found in commercial quantities, the near- permanent pseudo war between USA and China to the actual war in Eastern Europe, It’s the dawn of a new era in the energy industry and realizing it can no longer be business as usual, industry chiefs need to transform themselves and their organizations to succeed. To further call out the situation, in 2015 the United Nations (UN) developed a set of sustainable development goals, one of which was to ensure access to affordable, reliable, sustainable, and modern energy for all by 2030 with one of the targets being to increase substantially the share of renewable energy in the global energy mix by then. This target puts pressure on country governments around the world to cut fossil fuel use and develop renewable energy sources. Now, with renewable energies currently being a very little portion of the mix for most global energy companies, it means the need for organizational transformation is urgent if the goal of remaining major contributors to the energy that powers the world and beat the competition must be achieved. This study will identify specific areas of uncertainties in this transformation, how the industry has responded as well as the impact it has brought. It will look at how decision making has been identified as one of the strategic imperatives alongside digital, performance management and others to help win in every environment. Based on this analysis, it will derive guidelines for firms and investors to reduce uncertainties on the path to increase renewables in the energy mix as well as support firms’ decision-making under these uncertainties.

Organizational Transformation Challenge: End-to-end Integration of DQ in Large Capital Projects

April 18, 2024 11:15am - April 18, 2024 12pm

Click here to download PDF copy of presentation

Speaker: Carl Spetzler (Strategic Decisions Group)

Abstract:

Large Capital Projects present unique challenges that are being addressed with IPD (Integrated Project Delivery). Full adoption and integration of DQ within IPD assures maximum value in large projects. The organizational transformation needs are unique since project organizations are in constant transition throughout the project life cycle. We will review what it takes to achieve success in this transformation.

Pharma Track

New Approaches to Pre- and Post- Revenue Asset Valuation in Drug Development

April 18, 2024 10:30am - April 18, 2024 11am

Click here to download PDF copy of presentation

Speaker: Michael Kane (Telperian)

Abstract:

Accurate valuations of both pre- and post-revenue drug assets plays an instrumental role in a broad range of operations and decisions the biotechnology and pharmaceutical sectors.They inform decisions made not only by pharmaceutical companies but also by a range of stakeholders including venture capitalists, private equity firms, and others. An accurate valuation provides a reliable forecast of the potential return on investment, which directly influences where and how capital is allocated within the industry. It forms the basis of strategic investment decisions, making it possible to identify the most promising opportunities and understand which investments are more likely to yield a satisfactory return. In this talk we present models designed to estimate future sales of post-revenue and pre-revenue assets. The former generates distributional estimates of each asset’s cumulative sales. Pre-revenue assets can then be valued by using sales estimates for similar post-revenue assets in the calculation of the net present value. When applied to Pfizer’s portfolio of post-revenue assets, this model accurately projects the company’s market capitalization—defined as the product of its share price and the number of outstanding shares—to within a 10% margin.

The New Zealand Drug Harms Study: Use of multi criteria decision analysis to consider social and individual harms from illegal drugs

April 18, 2024 11am - April 18, 2024 11:30am

Click here to download PDF copy of presentation

Speaker: Paul Gordon (Catalyze)

Abstract:

This presentation will show how an MCDA decision modelling approach, coupled with a collaborative process was used to determine and understand the range and extent of harms that 23 drugs has across New Zealand. The drug harms were evaluated by a diverse group of experts and considered both the whole-of-population harms and harms specific to youth (12-17 years old) across 17 distinct harms. The harms considered covered both harms to the user, and harms to others around the user.

The background and history of the approach and methodology will be presented, including a summary of a similar earlier study in Australia, and will explore how this approach could provide valuable insights into how to think about drug harms in general.

Prior Elicitation as input to Phase 3 PTS Calculation

April 18, 2024 11:30am - April 18, 2024 12pm

Click here to download PDF copy of presentation

Speaker: Niki Arya (AstraZeneca)

Abstract:

We utilized expert elicitation to determine the prior assumptions needed to calculate the probability of technical success (PTS) for a planned large outcome study. The primary endpoint for the large outcome study is a composite endpoint with subcomponents that are major events (deaths, hospitalizations, etc.). A presentation was put together summarizing the results from previous trials with our drug and other drugs within the same medication class. In these trials, a similar composite endpoint and the individual subcomponents were analyzed in slightly different populations to help shape prior assumptions about the effects of our drug on the primary endpoint proposed in our large outcome study. This presentation was shared with a panel of 5 experts in this field who then used these results to predict the expected treatment effect on the primary composite endpoint and each of its subcomponents in the population proposed for the planned large outcome study. These predictions were then pooled together and weighted appropriately based on the expertise of the panelist to form a prior distribution which was then used to determine the PTS for the planned large outcome study. In the presentation for the Society of Decision Professionals conference, we will go into the details of the expert elicitation process, how each expert’s predictions were incorporated into the prior assumptions, and how the prior assumptions were used to determine the PTS.

Chair's Choice Track

Augmenting AI with Decision Intelligence to Accelerate Business

April 18, 2024 10:30am - April 18, 2024 11am

Click here to download PDF copy of presentation

Speaker: Fadi Micaelian (Sparkdit)

Abstract:

We will explore in this use case how the leading aviation and aerospace giant was able to solve the problem of airplane swapping by dynamically optimizing their business objectives: Traveler Satisfaction, Airline Revenue, and Airline Expenses. The problem could not be tackled with traditional AI alone as it lacked both causality and the ability of taking into account the constantly changing market needs and requirements. We will also briefly touch on how the same AI augmentation technology, rooted in Decision Science, is yielding 2x better eCommerce recommendations than Content Filtering, or AI based Collaborative Filtering, without ever intruding on consumer privacy.

Explainable AI: How XAI Puts the End User Back in the Driver's Seat

April 18, 2024 11am - April 18, 2024 11:30am

Click here to download PDF copy of presentation

Speaker: Patrick Elder (ECS Federal)

Abstract:

In the era of Artificial Intelligence (AI) and machine learning, decision makers and decision analysts can be caught between the desire to leverage new technologies and the seemingly inherent opacity that comes with complex models. Without the ability to explain and ultimately justify the decision based on the results from AI models, these end-users can feel like passengers on a journey without a clear understanding of the route or destination. However, the advent of Explainable AI (XAI) promises to transform this landscape by providing transparency and interpretability in AI systems.

This session briefing at the Society of Decision Professionals Conference will delve into the thought-provoking article titled "Explainable AI: How XAI Puts the End User Back in the Driver's Seat" (https://ecstech.com/ecs-insight/blog/explainable-ai-how-xai-puts-the-end-user-back-in-the-drivers-seat/), authored by industry expert Patrick Elder. The session with explore concepts discussed in the article and how they can be applied to AI to improve the ability of decision makers and decision analysts to use these technologies appropriately and improve decision quality.

Join us in this illuminating session to gain a deeper understanding of how Explainable AI can help decision-making process improvement without sacrificing traceability. Whether you are a decision professional, data scientist, or simply interested in the transformative power of AI, this session promises to provide valuable insights and foster meaningful discussions on the future of responsible AI adoption.

Validation Challenges in AI-Based Decision Support

April 18, 2024 11:30am - April 18, 2024 12pm

Click here to download PDF copy of presentation

Speaker: Doug Samuelson (The Dupuy Institute)

Abstract:

Artificial Intelligence (AI), offers promising uses in decision-making, as support for both suggesting courses of action and evaluating likely outcomes. Especially with the recent increasing interest in conflicts that do not focus exclusively on direct military conflict, computer-supported assistance in assessing “what might happen if” is becoming more and more valuable, if not essential. However, use of such support involves some hazards well worth noting. Among these are: the AI’s extensive data requirements; the difficulty of assessing the credibility of the AI’s outputs; the difficulty, for some systems, of making minor adjustments and re-running the analysis (the very capability one would hope these systems would enhance); all too often, the opacity of the reasoning the AI employed; and some particular legal and operational difficulties of dealing with software providers. Most important, there is an intractable limitation: AI cannot infer context nor make inferences about matter entirely outside the data it as had the opportunity to ingest. Also, AI cannot be validated without an observation-based data set, more reliable than the AI, against which to compare the AI’s results. However, even observation-based data sets entail inherent uncertainty. Relying heavily and uncritically on such analyses has a high probability of leading to disaster. We discuss these issues in the context of some actual professional wargaming experiences.

Energy Track

DECISION QUALITY FOR EXPLORATION WELL PATH OPTIMIZATION - A CASE STUDY FROM THE VIENNA BASIN, AUSTRIA

April 18, 2024 1pm - April 18, 2024 1:45pm

Click here to download PDF copy of presentation

Speaker: Jost Püttmann & Anna Kruk-Michot (OMV)

Abstract:

OMV is a Vienna based energy company with an Upstream division and Exploration & Production activities in 13 countries. In November 2023 OMV faced a complex challenge when they commenced drilling a 5500m deep well, Strasshof T17, in the Vienna Basin, Austria. The project aimed to explore a complex subsurface structure, poorly imaged on seismic data. The initial plan involved drilling an expensive pilot hole before the final sidetrack to mitigate subsurface uncertainties.

To address this challenging situation, OMV applied a Decision Quality (DQ) framework which they executed on the critical path within one week. The workflow included defining stakeholders, framing the problem, establishing the drilling path, evaluating different subsurface scenarios, and presenting findings to the decision board.

The application of DQ analysis allowed to optimize the drilling strategy resulting in cost savings of 25%. Beyond the cost benefits, the DQ process fostered effective team building and alignment among diverse disciplines. Additionally, the systematic analysis approach yielded previously unavailable insights, validated long-debated assumptions, and garnered attention at the senior vice president (SVP) level, demonstrating its effectiveness and scalability.

Chair's Choice Track

Let’s Talk Strategy Development and Facilitation

April 18, 2024 1pm - April 18, 2024 1:30pm

Click here to download PDF copy of presentation

Speaker: Bill Haskett (Haskett Consulting International)

Abstract:

Do you really have a strategy? Strategy development and implementation support is a different flavour of decision support from that provided within most projects. While our Decision Quality principles remain intact, a shift that emphasizes greater context, positioning, and pathway/vehicle is required. The Objective emphasis shifts to access, materiality, positioning, control, and preservation. Hambrick and Fredrickson developed an excellent approach to strategy development in the early 2000’s citing five components (Location, Vehicles, Staging, Differentiation, and Economic Logic) but it deserves updating from our decision and facilitation perspective. As guides to strategy development, we must ensure a greater emphasis on effective implementation. For example, it isn’t good enough to only differentiate. Competitive advantage (offensive and defensive) must have direction and “stickiness”. The sub-elements of Materiality, Access, Differentiation, and Preservation lead to a critical success element, Control. All elements support objective fulfilment of a desired preferred/prioritized objectives.

Our task as Strategy and Decision Support professionals is to ensure the strategy is comprehensive, integrated, and logical. An updated objective-based strategy model will be developed and illustrated that will provide a logical approach to constructing and implementing successful strategies.

Back to Basics: Why Bayes Matters in Decision Analysis

April 18, 2024 2pm - April 18, 2024 2:30pm

Click here to download PDF copy of presentation

Speaker: Sheldon Bernard (A440 Management)

Abstract:

This conversation focuses on elucidating the fundamental role of the Bayesian framework in decision analysis, a crucial discipline and toolkit for navigating decision-making under uncertainty. Tracing back to our roots, we aim to unfold the connection between Bayesian Decision Theory and Inference and its relation to Decision Analysis. By offering thought-provoking insights (via case study) into the broader picture and practical applications of Bayesian Inference, we extend our exploration to its capacity for solving complex business problems with data. We will discuss how the Bayesian foundation provides the path to integrate Behavioral Economics, Decision Analysis, and Data Science. The aim is to provide perspective on how a Bayesian framework can help equip decision-makers with strategies to diminish bias and augment decision quality.