AI engineering · Agentic delivery

Building Effective Agents in Practice: Delivering Jira and Crashlytics Fixes as Pull Requests

Khairul Alam·October 2026·~15 min read
AI agentsBuilding effective agentsJiraCrashlyticsCI/CD

A change can start as a Jira ticket, a plain spec, or a crash report, and the path to a pull request can be automated. This article uses the framework from Anthropic’s Building effective agents to show how: a simple workflow, with one agent step in the middle, checks around it, and a person at the end.

In brief
  • Based on Anthropic’s Building effective agents: start simple, use workflows where the path is known, and an agent only where the steps cannot be predicted.
  • Three inputs (Jira, plain spec, Crashlytics) become one task format, then follow the same pipeline.
  • The pipeline is a workflow with one agent step. RAG is a retrieval capability inside it, not an alternative.
  • Prompts are written by code from templates. The agent can open a draft PR; a person merges.
Contents
  1. The idea behind it
  2. The cycle
  3. Three ways in
  4. Crashlytics to fix
  5. The patterns applied
  6. Where the LLM is needed
  7. LLM, RAG or agent?
  8. Parts of the augmented LLM
  9. Automatic prompts
  10. Tool design for coding agents
  11. Push, review and the one-click button
  12. Principles, guardrails and rollout
  13. Takeaways and glossary

1. The idea behind it

This article applies Anthropic’s engineering essay Building effective agents (Dec 2024) to one job: delivering a code change. Its advice is to prefer “simple, composable patterns” over complex frameworks, and it separates two kinds of agentic systems:

The essay describes five workflow patterns and then the agent. Everything in this article is built from these six pieces.

Prompt chainingworkflow LLM gate LLM fixed steps, each feeds the next Routingworkflow In Route A B C classify, then specialise Parallelizationworkflow In LLM LLM LLM Join split or vote, then combine Orchestrator-workersworkflow Planner W1 W2 W3 Join subtasks decided at run time Evaluator-optimizerworkflow Generate Evaluate feedback loop until good enough Agentmodel directs its own loop LLM Tools real results as ground truth stops on done or a limit
Five workflows and the agent, from Building effective agents. Pick the simplest one that works.

The essay’s guidance: find the simplest solution and add complexity only when it clearly improves results. Agents trade latency and cost for flexibility, so use one where the number of steps cannot be predicted. Software delivery is a good fit because tests give the agent objective feedback, while a person still reviews the result.

2. The cycle

Start: one click, a Jira rule, or a Crashlytics alert Jira ticketstory or bug Plain spectext or Markdown Crashlyticscrash issue Intakeone format Promptauto-built AgentLLM + tools Checksplain code Push + PRdraft only unattended, in a sandbox, no IDE Reviewa person Merge, releaseyour pipeline Monitorcrash-free rate
The same line serves every input. The dashed arrow closes the loop: production data confirms the fix, or raises the next alert.

3. Three ways in

InputWhat starts itWhat intake fetchesWatch out for
Jira ticketA button, or an automation rule on a label such as ai-okSummary, description, type, acceptance criteria, linksVague tickets. Stop and ask when criteria are missing
Plain specA command or a workflow runThe text itself (optionally cleaned up by a model)No fields, so the goal must be extracted
Crashlytics issueA Firebase alert: new fatal issue, regression, or velocityStack trace, exception, app version, OS and device, user count, logsObfuscated traces and personal data (see next section)

Whatever the source, intake produces the same JSON, so every later step is shared:

{
  "source": "crashlytics",
  "id": "a1b2c3",
  "type": "bug",
  "goal": "Fix NullPointerException in LoginViewModel.onSubmit",
  "evidence": {
    "users": 4210, "version": "3.8.1", "os": "Android 13-14",
    "top_frame": "LoginViewModel.kt:57"
  },
  "acceptance": [
    "No crash when the email is null",
    "A unit test reproduces the crash first"
  ],
  "scope": ["app/src/main/java/com/example/login/"],
  "open_questions": []
}

4. Crashlytics to fix

A crash is a bug report written by production. The new part of this flow is the middle: the model works out where and why, then the agent decides and applies a fix. See also Firebase Crashlytics: find and fix crashes faster.

Alertfatal, regression Triageusers, own code Fetchtrace, logs LocateLLM: cause, files Fixagent: test, fix Checkstests, lint, build Push + PRlinks the issue Reviewa person decides Releaseyour pipeline Monitorcrash-free rate Cannot reproduce it with a test, or low confidence? Stop and post the analysis, no PR. plain code language model retrieval a person
The model decides the solution in the middle (Locate, then Fix). Code does everything before and after it.

How the crash data reaches the pipeline depends on your setup: Firebase alerts through a Cloud Function, a scheduled query on the BigQuery export, or an MCP server that exposes Crashlytics to the agent. Check the current Firebase documentation for what is available.

5. The patterns applied

The pipeline is a workflow with one agent step. Each pattern from the essay has a natural place in it.

PatternWhere it appears
Prompt chainingThe backbone: intake, prompt, agent, checks, PR. Gates between steps: completeness (missing criteria stop the run), triage (crash thresholds), diff-size guard
RoutingClassify the input (feature, bug, chore, crash) and pick the matching template. Optionally send routine tickets to a smaller model and hard ones to a stronger model
ParallelizationSectioning: tests, lint and the secret scan run side by side. Voting: several independent review prompts look for risky changes
Orchestrator-workersFor a large ticket touching many files, a planner splits the work at run time and workers handle each part. Use only when one agent run is not enough
Evaluator-optimizerThe checks judge the change and the failure follow-up prompt makes the agent improve it. The optional review pass is the same idea with a second model as evaluator
AgentThe coding step itself: the number of steps is unpredictable, each step gets ground truth from running the tests, and the run is capped by turns, time and cost
Start with the first row only

A chain of four steps plus one agent run already delivers most of the value. Add routing, parallel checks, a reviewer or an orchestrator one at a time, and only if measured results improve.

6. Where the LLM is needed

Triggerplain code Fetch inputplain code NormaliseLLM, optional Find contextretrieval Build prompttemplates Agent loopLLM + tools Checksplain code Review passLLM, optional Push + PRplain code Review, mergea person plain code language model retrieval a person Dashed = optional. The model is required in exactly one place: the agent loop.
Five steps are plain code, one is retrieval, three involve a model, one is a person.

Rule of thumb: use a model where the input is fuzzy (language, code that needs understanding) and the output needs judgement; use code where the answer is exact (pass or fail, a branch name, an API call). Never let a model grade its own tests.

7. LLM, RAG or agent?

The essay’s building block is the augmented LLM: a model with retrieval, tools and memory. RAG is the retrieval part of that block. A workflow adds predefined code paths around it, and an agent lets the model direct its own loop. This pipeline uses the last two: a workflow, with one agent step inside.

Single LLM callone prompt in,one answer out Augmented LLMretrieval (RAG)tools, memoryreads, can call tools Workflowpredefined code pathschain, route, parallel,orchestrate, evaluatepredictable Agentthe model decideseach next step,using tools in a loopflexible, costs more more autonomy, more cost, more risk This pipeline: a workflow with one agent step
Climb only as far as the task needs. RAG is a capability of the second step, not a rival to the fourth.

8. Parts of the augmented LLM

Whichever tool you use (Claude Code, Codex, or one you build), a coding agent is the essay’s augmented LLM running in a loop. Knowing the parts around it shows what to configure, what to restrict, and what to watch.

TriggersCLI, chat, CI, webhook Instructionsrules, AGENTS.md Retrieval, memoryrepo search, notes, history Toolsread, edit, shell, git, MCP Augmented LLMretrieval, tools, memoryin a loop = an agent Permissions, sandboxallowlist, isolated runner Guardrails, checksbudgets, tests, review Observabilitylogs, cost, audit trail Outputscommit, PR, comment Inputs (blue) feed the model, it acts through tools (green), and controls (orange, red) bound it.
The essay’s building block is the augmented LLM. A delivery pipeline wraps it in triggers, instructions, controls and outputs.
ComponentIn this pipeline
InstructionsCLAUDE.md or AGENTS.md plus the generated task prompt
ToolsFile read and edit, shell with an allowlist, git, an MCP server for Jira or Crashlytics
Retrieval and memoryThe agent’s own code search, a plan file, the failure log from the last attempt
Permissions and sandboxA throwaway runner, a narrow token, no production access
Guardrails and observabilityTurn, time and cost caps; independent checks; saved prompts, diffs and costs

The series Claude Code in scripts and CI and Codex in scripts and CI show how to configure these in real tools.

9. Automatic prompts

Nobody types a prompt in this cycle. Code writes up to six, from templates versioned in the repo.

Normaliseinput to task Compilemain prompt Agentedits code Checkstest, lint Reviewsecond look PR textdescription check fails: an automatic follow-up carries the error log Questions to the ticket author 1 2 3 4 5 6 Numbered circles = a prompt written by code. Nobody types any of them during the run.
Number 3 is the one people forget. It makes the loop correct itself.
#PromptFires whenTemplate
1Extract the task (optional)Input is free textextract-task.md
2Main task promptOnce per runtask-bug.md, task-feature.md
3Failure follow-upA check fails, mid-runfix-failure.md
4Review the diff (optional)All checks passreview-diff.md
5PR descriptionBefore the draft PRpr-description.md
6Questions for the authorSpec has gapsclarify.md

Two examples: the main prompt for a crash, and the automatic follow-up.

# task-bug.md (crash)                        # fix-failure.md (sent by the pipeline)
Read CLAUDE.md first.                        The checks failed after your change.
<crash id="{{id}}">{{evidence}}</crash>          Step: {{step}}
1. Name the likely root cause + confidence.  Output: {{log_tail}}
2. Write a test that FAILS because of it.    Fix the cause. Never weaken tests.
3. Make the smallest fix. Run all checks.    Attempt {{n}} of {{max}}.
Max 150 changed lines. Report: cause, files, test, doubts.

10. Tool design for coding agents

The essay calls the interface between the model and its tools the agent-computer interface (ACI) and says it deserves as much care as a human interface. It also notes that tool definitions need as much prompt engineering as the main prompt. Applied here:

Advice from the essayIn a delivery pipeline
Write tool descriptions like a docstring for a junior developer: examples, edge cases, boundariesName and describe each tool and script clearly (run_tests, open_pr) and say when not to use it
Choose formats close to natural text; avoid formats that need line counting or escapingPrefer simple edit tools over asking the model to emit large diffs or JSON-wrapped code
Poka-yoke: change the tool so mistakes are harderRequire absolute paths (the essay’s own fix for path errors), allowlist shell commands, block forbidden paths, give the reviewer read-only tools
Test the tools with many example inputs and watch the mistakesReplay past tickets and crashes through the pipeline and fix the tool, not only the prompt

11. Push, review and the one-click button

Verifieddiff Branchai/PROJ-123-... Commitid in message Pushbranch only Draft PRsummary, links CI checksmust be green Human reviewrequired Mergeby a person The agent’s token can push a branch and open a draft PR. Branch protection stops it merging.
Git and API calls only, so this part is reliable. Merging stays human.
Jira button or rulePROJ-123 Chat or command lineship PROJ-123 Crashlytics alertCloud Function, webhook One workflow GitHub Actionssandboxed runnerscoped secretstime and cost caps Result draft pull requestcomment on the issuerun log and cost
Different front doors, one workflow. The behaviour is the same no matter who or what triggers it.
name: ticket-to-pr
on:
  workflow_dispatch:
    inputs: { source: { description: "PROJ-123 or crashlytics:<issue>", required: true } }
permissions: { contents: write, pull-requests: write }
jobs:
  deliver:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - run: python3 tools/intake.py "${{ inputs.source }}" > task.json     # Jira, spec or Crashlytics
      - run: python3 tools/compile_prompt.py task.json > prompt.md          # templates, no model
      - run: claude -p "$(cat prompt.md)" --allowedTools "Read,Edit,Bash(npm test:*)" --max-turns 30 > result.json
      - run: ./tools/verify.sh                                               # tests, lint, diff guard, secret scan
      - run: ./tools/open_pr.sh task.json result.json                        # branch, commit, push, draft PR

This is a sketch, not a drop-in file. Check each tool’s current documentation for flags and permissions, and give each secret the narrowest scope.

12. Principles, guardrails and rollout

The essay ends on three principles. Here is what each means for this pipeline.

PrinciplePractice
SimplicityStart with a chain and one agent run. Call the CLI or API directly before adopting a framework, and add patterns only when measured results improve
TransparencyShow the plan (a plan file), keep the prompts, the output, the diff and the cost of every run, linked from the PR
A well-built ACINarrow, documented, tested tools (previous section)

The essay also warns that agents cost more and errors can compound, and recommends sandbox testing and guardrails, stopping conditions such as a maximum number of iterations, and human checkpoints.

You do not need an IDE to deliver a routine change, but you still need engineering judgement to approve it. The scarce skills become writing testable specs and reading diffs.

13. Takeaways and glossary

TermIn plain words
WorkflowLLMs and tools orchestrated through predefined code paths.
AgentAn LLM that directs its own process and tool use in a loop until done or limited.
Augmented LLMA model with retrieval, tools and memory.
RAGRetrieve relevant passages first, then have the model answer from them.
ACIAgent-computer interface: how tools are defined, described and presented to the model.
Poka-yokeDesigning a tool so that mistakes are hard to make.
Auto-promptA prompt built by code from a template and the state of the run.
MCPModel Context Protocol: a standard way to connect an agent to tools such as Jira or GitHub.
← Back to all writing