20 Cross-discipline collaboration

20.1 Same words, different meanings

You tell an engineer you’ve validated the model. They hear that you’ve checked its inputs against a schema; you mean you held out data and measured how it generalises. They ask whether it’s tested; you say yes, thinking of your evaluation metrics, while they mean a suite of pass/fail assertions on the code. Someone says the model is accurate, and one of you hears a number on a continuum while the other hears “it does what the spec says”. Nobody is wrong, and nobody is lying. You are speaking overlapping vocabularies in which the same words carry different meanings, and the gap is invisible precisely because the words are shared.

This vocabulary gap is the everyday texture of data science and software engineering working together, and it is the reason this book exists — and its companion. The friction between the two disciplines is rarely a gap in ability; it’s a gap in vocabulary, assumptions, and what each side considers the obvious default. This chapter is about working across that gap deliberately.

20.2 The words that collide

It’s worth naming the specific collisions, because a gap you can recognise mid-sentence is one you can close in the next sentence. Performance is the most expensive of them. To you it means how well the model predicts; to an engineer it means latency, throughput, and memory. A conversation in which one side reports that performance improved and the other hears that the service got faster can run for a surprisingly long time before anyone notices, and it will be noticed at the worst moment — typically when a model that gained two points of accuracy by tripling its inference time reaches a service with a latency budget.

Model is nearly as bad. Yours is a trained artefact with learned parameters; an engineer’s is a schema — the data model, the domain model, the classes that describe the entities in the system. Ask where the model is defined and you’ll be pointed at a file of type declarations. Regression splits the same way: you fit one, they run one. To an engineer, a regression is a previously-working behaviour that has broken, and “we found a regression” is a bug report rather than a modelling result. Bias carries at least three meanings across the two disciplines — the statistical property of an estimator, the fairness sense used about model outcomes, and the loose colloquial sense of a system being skewed — and because all three are legitimate, a discussion about “reducing bias” can proceed with each side agreeing enthusiastically to something different. Significant is the classic: you may mean a p-value below a threshold, and your listener almost certainly means “big enough to care about”, which is why a result described as significant but tiny sounds to them like a contradiction. And validation, as the opening noted, means checking inputs conform to a schema on one side and estimating generalisation on the other — with the extra hazard that inside machine learning, the validation set and the test set are already distinct things, so an engineer hearing “test set” reasonably assumes you mean a set of tests.

The fix isn’t to police anyone’s vocabulary; both usages are correct in their home discipline and neither side will abandon theirs. It’s to qualify the noun when you’re speaking across the boundary — inference latency, held-out accuracy, statistically significant, the trained artefact — which costs a word and removes the ambiguity at source. The more useful habit, though, is learning to notice the silence. Disagreements are safe, because they surface. The dangerous moment is the one where a discussion touching one of these words runs unusually smoothly and everyone nods; that easy agreement is often the sound of two people agreeing to different things, and it’s worth spending thirty seconds asking someone to restate what they think was decided.

20.3 Working across the divide

In a production data science effort you’ll work with software engineers (who own the systems your model runs inside), ML engineers and platform teams (who own the serving and pipeline infrastructure), and product managers (who own what the thing is for). Each cares about different properties — reliability, latency, maintainability, business value — and most of the friction concentrates at the handoff, the point where a model leaves the data scientist and becomes part of a larger system. The remedy is not better intentions; it’s shared vocabulary and explicit contracts, replacing assumed understanding with stated agreement.

Data Science Bridge

A shared interface between teams is a data contract. You already rely on the idea: two systems agree on a schema so that data can flow between them without either having to know the other’s internals. A team interface is the same construct applied to people — the data scientists and the engineers agree on exactly what the model produces (its output schema, its latency, how it behaves on bad input) and what the surrounding service expects, so neither side has to guess or reverse-engineer the other’s assumptions.

Where the analogy breaks down: a data contract is enforced by code — the schema validates, or the request fails. A team contract has a human dimension no schema captures: a shared understanding of intent, of what “good enough” means for this use case, of who is responsible when the model misbehaves at 3am. The schema pins down the bytes; it cannot pin down the agreement about ownership and intent, and that human half is where collaboration most often succeeds or fails.

20.4 The handoff

The moment a model passes to engineering for production is where the most can go wrong, because each side assumes the other knows things that were only ever in their head. A good handoff makes those things explicit. Engineering needs from you: the model artefact and exactly how to load it (Chapter 15), the input and output contract, the expected performance and its caveats, the known failure modes, and how the model should be monitored (Chapter 16). You need from them: where it will run, the latency budget, and how errors will surface. The contract at the centre of it all is something both sides can write down and validate:

from pydantic import BaseModel, Field, ValidationError

# The agreed interface between the data science team (which produces
# predictions) and the engineering service (which consumes them).
# Writing it down is the handoff: both sides build against this.
class ChurnPrediction(BaseModel):
    customer_id: int
    churn_probability: float = Field(ge=0, le=1)   # a probability, not a label
    model_version: str                              # so a bad version is traceable

# Engineering can validate every prediction against the contract.
valid = ChurnPrediction(customer_id=42, churn_probability=0.87, model_version="v2.1")
print(f"accepted: {valid.customer_id} -> {valid.churn_probability}")

try:
    ChurnPrediction(customer_id=42, churn_probability=1.7, model_version="v2.1")
except ValidationError as exc:
    print(f"rejected: {exc.errors()[0]['msg']}")

accepted: 42 -> 0.87
rejected: Input should be less than or equal to 1

The contract names what flows across the boundary and what counts as valid — a probability bounded in [0, 1], a version string so a misbehaving release is traceable. Agreeing it up front, in code both teams build against, is dramatically cheaper than discovering in production that the data scientist returned a label where the service expected a probability. A short handoff checklist (artefact, contract, performance and caveats, failure modes, monitoring, ownership) turns the implicit into the explicit before anyone is paged.

20.5 Meeting in the middle

Responsibilities overlap at the boundary, and pretending they don’t is how things fall through it. Who owns the feature pipeline — the data scientist who designed the features or the engineer who runs it? Who is on call when the model’s predictions go strange? The workable answer is shared ownership with clear interfaces: each side owns its part, the contract defines the seam, and both understand enough of the other’s world to collaborate rather than lob work over a wall.

That mutual understanding is the whole purpose of crossing the bridge — in either direction. Software engineering is built to eliminate uncertainty: to make behaviour repeatable, guarantees enforceable, results reproducible. Data science is built to work within uncertainty: to quantify it, model it, and decide well despite it. Each discipline has a rigour the other lacks, and each can find the other’s rigour baffling at first. This book has spent twenty chapters carrying engineering practices into data science; its companion, Thinking in Uncertainty, carries statistical thinking the other way, into software engineering. The two meet here, at the handoff — and the teams that work best are the ones where each side has learned enough of the other’s discipline to respect it and build against it.

Author’s Note

The data science–software engineering gap is usually described as a skills gap, but it’s more honestly a values gap, and seeing that is what makes collaboration work. Engineers optimise for reliability and reproducibility; data scientists optimise for insight and adaptability under uncertainty. Each can read the other’s priorities as a failing: the engineer sees the untested exploratory notebook as recklessness, and the data scientist sees the demand for tests around a throwaway experiment as bureaucracy. Both are correct within their own frame, and both are missing the half the other can see.

The point of this book — and of crossing the bridge in either direction — was never to convert one discipline into the other. It’s to hold both at once: to know when to explore loosely and when to engineer tightly, and to extend genuine respect to a colleague whose instinct points the opposite way to yours, because between the two of you, you cover ground that neither could alone. You don’t have to become a software engineer. You have to become a data scientist who can build things others can trust, and who can work with the people whose job is to run them — which is a larger, and more useful, thing to be.

20.6 Summary

Collaboration across the divide is built on shared vocabulary and explicit contracts:

The same words mean different things. “Test”, “validation”, “model”, “accuracy” carry different meanings to data scientists and engineers; the gap is invisible because the words are shared, and naming it is the first fix.
Friction concentrates at the handoff. Replace assumed understanding with explicit contracts — what the model produces, what the service expects — written down where both sides can build against them.
A team interface is a data contract plus a human one. The schema pins the bytes; intent, “good enough”, and ownership are the human half no schema captures.
Hold both kinds of rigour. Engineering eliminates uncertainty, data science quantifies it; the strongest teams respect both and have learned enough of each to meet in the middle.

Part 6 puts the whole book to work, carrying a model from a notebook all the way to production across three end-to-end projects, beginning with from notebook to production API.

20.7 Exercises

Map the vocabulary gaps on your own team: list three words that mean different things to you and to a software engineer (start with “test”, “validation”, and “model”), and write the two meanings side by side. Which of these gaps has caused an actual misunderstanding you can recall?
Write a handoff document for a model you would give to engineering: the artefact and how to load it, the input/output contract, the expected performance and its caveats, the failure modes, and how it should be monitored. What did you have to make explicit that you had never previously written down?
Define the interface between a model and the service that calls it as an explicit contract — a schema for the inputs and outputs, the latency budget, and the behaviour on bad input. Why is agreeing this in advance cheaper than discovering the mismatch in production?
Conceptual: A team takes the data-contract analogy at its word: they agree a schema for the model’s output, wire its validation into CI, and call the handoff done. Six weeks later they are in an incident review anyway. Describe a specific failure that passes every schema check and still breaks the collaboration — then say what would have had to be written down, and by whom, to prevent it.
Conceptual: This book and its companion frame the divide as two kinds of rigour — eliminating uncertainty versus quantifying it. Describe a concrete situation where the engineering instinct and the data science instinct pull in opposite directions, and how a team holding both would resolve it.

--- # Content: CC BY-NC-SA 4.0 | Code: MIT - see /LICENSE.md --- # Cross-discipline collaboration {#sec-collaboration} ## Same words, different meanings {#sec-same-words} You tell an engineer you've *validated* the model. They hear that you've checked its inputs against a schema; you mean you held out data and measured how it generalises. They ask whether it's *tested*; you say yes, thinking of your evaluation metrics, while they mean a suite of pass/fail assertions on the code. Someone says the model is *accurate*, and one of you hears a number on a continuum while the other hears "it does what the spec says". Nobody is wrong, and nobody is lying. You are speaking overlapping vocabularies in which the same words carry different meanings, and the gap is invisible precisely because the words are shared. This vocabulary gap is the everyday texture of data science and software engineering working together, and it is the reason this book exists — and its companion. The friction between the two disciplines is rarely a gap in ability; it's a gap in vocabulary, assumptions, and what each side considers the obvious default. This chapter is about working across that gap deliberately. ## The words that collide {#sec-words-that-collide} It's worth naming the specific collisions, because a gap you can recognise mid-sentence is one you can close in the next sentence. *Performance* is the most expensive of them. To you it means how well the model predicts; to an engineer it means latency, throughput, and memory. A conversation in which one side reports that performance improved and the other hears that the service got faster can run for a surprisingly long time before anyone notices, and it will be noticed at the worst moment — typically when a model that gained two points of accuracy by tripling its inference time reaches a service with a latency budget. *Model* is nearly as bad. Yours is a trained artefact with learned parameters; an engineer's is a schema — the data model, the domain model, the classes that describe the entities in the system. Ask where the model is defined and you'll be pointed at a file of type declarations. *Regression* splits the same way: you fit one, they run one. To an engineer, a regression is a previously-working behaviour that has broken, and "we found a regression" is a bug report rather than a modelling result. *Bias* carries at least three meanings across the two disciplines — the statistical property of an estimator, the fairness sense used about model outcomes, and the loose colloquial sense of a system being skewed — and because all three are legitimate, a discussion about "reducing bias" can proceed with each side agreeing enthusiastically to something different. *Significant* is the classic: you may mean a p-value below a threshold, and your listener almost certainly means "big enough to care about", which is why a result described as significant but tiny sounds to them like a contradiction. And *validation*, as the opening noted, means checking inputs conform to a schema on one side and estimating generalisation on the other — with the extra hazard that inside machine learning, the validation set and the test set are already distinct things, so an engineer hearing "test set" reasonably assumes you mean a set of tests. The fix isn't to police anyone's vocabulary; both usages are correct in their home discipline and neither side will abandon theirs. It's to qualify the noun when you're speaking across the boundary — inference latency, held-out accuracy, statistically significant, the trained artefact — which costs a word and removes the ambiguity at source. The more useful habit, though, is learning to notice the silence. Disagreements are safe, because they surface. The dangerous moment is the one where a discussion touching one of these words runs unusually smoothly and everyone nods; that easy agreement is often the sound of two people agreeing to different things, and it's worth spending thirty seconds asking someone to restate what they think was decided. ## Working across the divide {#sec-across-divide} In a production data science effort you'll work with software engineers (who own the systems your model runs inside), ML engineers and platform teams (who own the serving and pipeline infrastructure), and product managers (who own what the thing is for). Each cares about different properties — reliability, latency, maintainability, business value — and most of the friction concentrates at the *handoff*, the point where a model leaves the data scientist and becomes part of a larger system. The remedy is not better intentions; it's shared vocabulary and explicit contracts, replacing assumed understanding with stated agreement. ::: {.callout-note} ## Data Science Bridge A shared interface between teams is a data contract. You already rely on the idea: two systems agree on a schema so that data can flow between them without either having to know the other's internals. A team interface is the same construct applied to people — the data scientists and the engineers agree on exactly what the model produces (its output schema, its latency, how it behaves on bad input) and what the surrounding service expects, so neither side has to guess or reverse-engineer the other's assumptions. Where the analogy breaks down: a data contract is enforced by code — the schema validates, or the request fails. A *team* contract has a human dimension no schema captures: a shared understanding of intent, of what "good enough" means for this use case, of who is responsible when the model misbehaves at 3am. The schema pins down the bytes; it cannot pin down the agreement about ownership and intent, and that human half is where collaboration most often succeeds or fails. ::: ## The handoff {#sec-handoff} The moment a model passes to engineering for production is where the most can go wrong, because each side assumes the other knows things that were only ever in their head. A good handoff makes those things explicit. Engineering needs from you: the model artefact and exactly how to load it (@sec-deployment), the input and output contract, the expected performance *and its caveats*, the known failure modes, and how the model should be monitored (@sec-monitoring). You need from them: where it will run, the latency budget, and how errors will surface. The contract at the centre of it all is something both sides can write down and validate: ```{python} #| label: handoff-contract #| echo: true from pydantic import BaseModel, Field, ValidationError # The agreed interface between the data science team (which produces # predictions) and the engineering service (which consumes them). # Writing it down is the handoff: both sides build against this. class ChurnPrediction(BaseModel): customer_id: int churn_probability: float = Field(ge=0, le=1) # a probability, not a label model_version: str # so a bad version is traceable # Engineering can validate every prediction against the contract. valid = ChurnPrediction(customer_id=42, churn_probability=0.87, model_version="v2.1") print(f"accepted: {valid.customer_id} -> {valid.churn_probability}") try: ChurnPrediction(customer_id=42, churn_probability=1.7, model_version="v2.1") except ValidationError as exc: print(f"rejected: {exc.errors()[0]['msg']}") ``` The contract names what flows across the boundary and what counts as valid — a probability bounded in `[0, 1]`, a version string so a misbehaving release is traceable. Agreeing it up front, in code both teams build against, is dramatically cheaper than discovering in production that the data scientist returned a label where the service expected a probability. A short handoff checklist (artefact, contract, performance and caveats, failure modes, monitoring, ownership) turns the implicit into the explicit before anyone is paged. ## Meeting in the middle {#sec-meeting-middle} Responsibilities overlap at the boundary, and pretending they don't is how things fall through it. Who owns the feature pipeline — the data scientist who designed the features or the engineer who runs it? Who is on call when the model's predictions go strange? The workable answer is shared ownership with clear interfaces: each side owns its part, the contract defines the seam, and both understand enough of the other's world to collaborate rather than lob work over a wall. That mutual understanding is the whole purpose of crossing the bridge — in either direction. Software engineering is built to *eliminate* uncertainty: to make behaviour repeatable, guarantees enforceable, results reproducible. Data science is built to work *within* uncertainty: to quantify it, model it, and decide well despite it. Each discipline has a rigour the other lacks, and each can find the other's rigour baffling at first. This book has spent twenty chapters carrying engineering practices into data science; its companion, *Thinking in Uncertainty*, carries statistical thinking the other way, into software engineering. The two meet here, at the handoff — and the teams that work best are the ones where each side has learned enough of the other's discipline to respect it and build against it. ::: {.callout-tip} ## Author's Note The data science–software engineering gap is usually described as a skills gap, but it's more honestly a *values* gap, and seeing that is what makes collaboration work. Engineers optimise for reliability and reproducibility; data scientists optimise for insight and adaptability under uncertainty. Each can read the other's priorities as a failing: the engineer sees the untested exploratory notebook as recklessness, and the data scientist sees the demand for tests around a throwaway experiment as bureaucracy. Both are correct *within their own frame*, and both are missing the half the other can see. The point of this book — and of crossing the bridge in either direction — was never to convert one discipline into the other. It's to hold both at once: to know when to explore loosely and when to engineer tightly, and to extend genuine respect to a colleague whose instinct points the opposite way to yours, because between the two of you, you cover ground that neither could alone. You don't have to become a software engineer. You have to become a data scientist who can build things others can trust, and who can work with the people whose job is to run them — which is a larger, and more useful, thing to be. ::: ## Summary {#sec-cross-discipline-summary} Collaboration across the divide is built on shared vocabulary and explicit contracts: 1. **The same words mean different things.** "Test", "validation", "model", "accuracy" carry different meanings to data scientists and engineers; the gap is invisible because the words are shared, and naming it is the first fix. 2. **Friction concentrates at the handoff.** Replace assumed understanding with explicit contracts — what the model produces, what the service expects — written down where both sides can build against them. 3. **A team interface is a data contract plus a human one.** The schema pins the bytes; intent, "good enough", and ownership are the human half no schema captures. 4. **Hold both kinds of rigour.** Engineering eliminates uncertainty, data science quantifies it; the strongest teams respect both and have learned enough of each to meet in the middle. Part 6 puts the whole book to work, carrying a model from a notebook all the way to production across three end-to-end projects, beginning with *from notebook to production API*. ## Exercises {#sec-cross-discipline-exercises} 1. Map the vocabulary gaps on your own team: list three words that mean different things to you and to a software engineer (start with "test", "validation", and "model"), and write the two meanings side by side. Which of these gaps has caused an actual misunderstanding you can recall? 2. Write a handoff document for a model you would give to engineering: the artefact and how to load it, the input/output contract, the expected performance and its caveats, the failure modes, and how it should be monitored. What did you have to make explicit that you had never previously written down? 3. Define the interface between a model and the service that calls it as an explicit contract — a schema for the inputs and outputs, the latency budget, and the behaviour on bad input. Why is agreeing this in advance cheaper than discovering the mismatch in production? 4. **Conceptual:** A team takes the data-contract analogy at its word: they agree a schema for the model's output, wire its validation into CI, and call the handoff done. Six weeks later they are in an incident review anyway. Describe a specific failure that passes every schema check and still breaks the collaboration — then say what would have had to be written down, and by whom, to prevent it. 5. **Conceptual:** This book and its companion frame the divide as two kinds of rigour — eliminating uncertainty versus quantifying it. Describe a concrete situation where the engineering instinct and the data science instinct pull in opposite directions, and how a team holding both would resolve it.