Go Easy on Hosted Services When Using AI Agents

AI agents are now part of how a lot of us write code, and that isn’t going away. But every time an agent “just checks” a file on a forge, it sends a request to someone’s server. Multiply that by every agent, every session, and every retry, and the people who keep that infrastructure running pay for it with their time, money, and sleep.

What Maintainers Are Saying

In Please stop externalizing your costs directly into my face, Drew DeVault writes that he spent 20-100% of his time in some weeks fighting LLM crawlers at SourceHut. They ignore robots.txt and hit expensive endpoints like git blame, every page of git log, and every commit in every repo, causing dozens of brief outages per week.

In Protecting our FLOSS commons from LLMs, Codeberg describes crawlers reading every issue filter variant, the whole Git history, and every file at every revision, even unchanged ones. Those requests trigger expensive database queries, and the defenses admins build against them end up limiting legitimate users too. Their advice is short: “Just run git clone and enjoy.”

Both would rather you didn’t use LLMs at all. If you do anyway, the least you can do is not make it worse: an agent browsing a forge interactively hits the same expensive pages those crawlers do.

The Idea

Fetch once, work locally. An agent reading a repository page by page through the web UI makes dozens of requests. One git clone does the same job.

Tell Your Agent

Agents follow their instruction file, so put the rule there. Here’s the section from my global ~/.claude/CLAUDE.md:

## Go easy on hosted services

- To read code, history, or docs from a hosted repo (GitHub, GitLab,
  Codeberg, etc.), prefer one `git clone` over repeated API calls,
  raw-file fetches, or crawling the web UI page by page. It's one
  request instead of dozens, and it reduces load on the admins who
  run those services.
  - Clone into `~/.cache/repos/<host>/<owner>/<repo>`, not the
    scratchpad or `/tmp`, so the clone survives reboots and later
    sessions can reuse it. Writing there is allowed without asking.
  - E.g. `git clone --no-checkout --depth 1 <url> ~/.cache/repos/<host>/<owner>/<repo>`,
    then explore locally with `git grep <pattern> HEAD`,
    `git show HEAD:<path>`, `git ls-tree -r HEAD`. Drop `--depth 1`
    when you need history.
  - Before cloning, check whether that path already exists. If it
    does, `git fetch` it instead of cloning again.
- Same idea for other services: download a dump, archive, or package
  once instead of scraping, reuse what you've already fetched, and
  don't poll.

The same works for AGENTS.md or whatever file your tool reads.

Keep Clones Somewhere That Lasts

Agents like to put temporary files in a scratchpad under /tmp. On most Linux systems /tmp is wiped on reboot or cleaned out after a few days. So the “reuse an existing clone” rule quietly stops working, and every restart means cloning everything again.

Point the agent at a directory that persists, like ~/.cache/repos/. A <host>/<owner>/<repo> layout keeps paths predictable, so a later session can find the clone and fetch it instead of downloading it again. If your instruction file restricts writes outside the project, add an explicit exception for this directory, or the agent will fall back to /tmp.

What’s in there is still third-party code, so treat it as untrusted: read it, but don’t run or build it without checking first.

Exploring a Clone Locally

--no-checkout skips the working tree; you read files straight from git’s objects.

repo=~/.cache/repos/codeberg.org/owner/repo

git clone --no-checkout --depth 1 https://codeberg.org/owner/repo.git "$repo"

git -C "$repo" ls-tree -r --name-only HEAD   # list files
git -C "$repo" grep -n "pattern" HEAD        # search
git -C "$repo" show HEAD:path/to/file        # read a file

git -C "$repo" fetch --depth 1 origin        # update later
git -C "$repo" show FETCH_HEAD:path/to/file

Summary

• AI usage is inevitable; dumping its costs on volunteers isn’t.
• Clone once, explore locally, fetch to update.
• Keep clones in a persistent directory, not /tmp, so they survive reboots.
• Write it into your agent’s instruction file so it’s the default.