The Black Box vs. Human Rights: Why Human Services Needs SHAP
In community services, "computer says so" is an ethical failure. If we deploy predictive models to allocate resources and scrutinise efficacywe must defend exactly *how* those outputs are generated. If we can't explain it, we shouldn't use it, if we use nothing we deserve smack.
Traditional tools like ANOVA or MANOVA look at multicollinearity retrospectively within a constrained set. They struggle to show dynamic, systemic impact across complex service delivery landscapes.
**KernelSHAP (SHapley Additive exPlanations)** changes this by shifting from simple correlation to predictive application. Derived from cooperative game theory, it calculates the marginal contribution of every variable across every possible combination of inputs.
The standard weight allocation is calculated as:
* **M**: Total number of variables in the model.
* **|z'|**: Number of active features being tested in conjunction.
* **\binom{M}{|z'|}**: The binomial coefficient.
Instead of guessing, we get a tangible output and weight of what is driving an output, exact amounts.
### Governance vs. Surveillance
When used properly, SHAP enforces algorithmic accountability. When ignored, people get hurt.
* **The Reality:** In Ontario, explainable modeling helps unpack Intimate Partner Violence (IPV) risk assessments and population-level recidivism data to inform interventions. But let's be real: implementing this properly is highly impractical in a resource-scarce reality.
* **The Yikes:** Look at the NSW Police Suspect Target Management Plan (STMP). Without transparent attribution, predictive tools quickly default to missing axioms, systemic racism, and Orwellian over-policing.
Using SHAP isn't about getting fancy; it’s foundational data governance. If you deploy predictive models without this level of visibility, you are choosing to sit uncomfortably with unaccountable systems.
*Change my mind or challenge the math below.*
Shap - a better tool than an acronym thank gosh
