<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:media="http://search.yahoo.com/mrss/">
  <channel>
    <title>Cold Starts · 仁才德</title>
    <link>https://jared.lynskey.co.nz/zh-cn/tags/cold-starts/</link>
    <description>关于软件工程与研发团队领导力的短文——绩效评估、招聘与团队扩张、CI/CD、DevOps 监控、智能体 AI。写于首尔。</description>
    <generator>Hugo</generator>
    <language>zh-CN</language>
    <copyright>© 2026 仁才德</copyright>
    <lastBuildDate>Sat, 07 Nov 2026 00:00:00 +1300</lastBuildDate>
    <atom:link href="https://jared.lynskey.co.nz/zh-cn/tags/cold-starts/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Fly.io 冷启动为什么慢：缩容到零后，从 33 秒降到 3 秒</title>
      <link>https://jared.lynskey.co.nz/zh-cn/posts/2026/2026-09-09-scale-to-zero-504s/</link>
      <guid isPermaLink="true">https://jared.lynskey.co.nz/zh-cn/posts/2026/2026-09-09-scale-to-zero-504s/</guid>
      <pubDate>Wed, 09 Sep 2026 00:00:00 +1200</pubDate>
      <dc:creator>Jared Lynskey</dc:creator>
      <category>Fly.io</category>
      <category>Django</category>
      <category>DevOps</category>
      <category>Cloudflare</category>
      <category>Performance</category>
      <category>Cold Starts</category>
      <category>Redis</category>
      <category>Debugging</category>
      <description>在 Fly.io 上用 auto_stop_machines 缩容到零，请求落到已停止的机器上就得等它启动。我的冷启动要 25 到 33 秒，这篇讲我怎么把它降到 3 秒左右。</description>
      <content:encoded><![CDATA[<p><strong>更新（2026 年 10 月 22 日）</strong>：这篇文章最核心的数字是错的。发布当天，我把这个 zone 的 504 按 user agent 分组，发现 9 月 5 日到 9 日的 7,344 个 504 全部来自 Cloudflare 自己的 Early Hints 探测请求（<code>bastion early hints</code> 和 <code>nginx-ssl early hints</code>）。这些请求从 Fly 源站拿不到任何响应，被记成 504，而且沿用真实请求的路径，看起来和用户侧故障一模一样。同一时期，真实客户端（应用、浏览器、curl、搜索引擎爬虫）一个 504 都没收到。源站本来就不发 preload 头，Early Hints 什么忙也没帮上，所以我当天就把它关了。下面的冷启动是真的，解决办法也依然有效，但 22% 从来不是我的用户遇到的数字。在把 Cloudflare 的 5xx 比例当成故障之前，先看看是哪些 user agent 在收到这些错误。</p>
<p><strong>更新（2026 年 11 月 4 日）</strong>：再更正一点。这些 504 并不是源站没能响应的探测请求，而是 Cloudflare 自己的 Early Hints 缓存查询，根本不会到达源站；这里的 504 只表示这个 URL 没有缓存任何内容。关掉 Early Hints 也不是最好的办法：发送 <code>Link</code> 响应头之后，这个功能开始生效，这些未命中也变成了命中。完整经过见《<a href="https://jared.lynskey.co.nz/zh-cn/posts/2026/cloudflare-504-early-hints/" >Cloudflare 504 错误其实不是故障：Early Hints 与 HTTP 103 详解</a>》。</p>
<p>先说结论：开启 <code>auto_stop_machines</code> 后，落到已停止的 Fly.io 机器上的请求要等整台机器启动完。我的机器冷启动要 25 到 33 秒，因为每次机器启动，entrypoint 都会跑一遍 Django 迁移和 <code>collectstatic</code>，而字节码从来没人缓存过。把这些发布相关的工作改成每个镜像只做一次（用以 <code>FLY_IMAGE_REF</code> 为键的 Redis 锁），再在构建时加上 <code>compileall</code>，冷启动降到了 9 秒左右。最后把 <code>auto_stop_machines</code> 从 <code>'stop'</code> 改成 <code>'suspend'</code>（为此得先去掉 swap，Fly 不允许挂起带 swap 的机器），唤醒只要 3 秒左右。</p>
<p>我本来只是去 Cloudflare 面板上看一个流量尖峰，结果在它下面发现了严重得多的东西：一天之内，应用处理的请求有 22% 是失败的。没有任何告警触发，健康检查全是绿的，日志看起来也完全正常。因为那些失败的请求，根本没有到达应用。</p>
<p>原因是我自己写的容器启动脚本，在完全错误的时机做着完全合理的事。</p>

<h2 class="relative group">尖峰是个幌子
    <div id="尖峰是个幌子" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e5%b0%96%e5%b3%b0%e6%98%af%e4%b8%aa%e5%b9%8c%e5%ad%90" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>我最初想解释的是这样一个小时：约 4,900 个请求，而中位数是每小时 121 个。独立访客 42 个。也就是每人 110 个请求，人类不是这么浏览网页的。</p>
<p>是扫描器。一个机器人在编造子域名，挨个向它们索要密钥文件：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">api-stage-asia.example.com/.env.local
</span></span><span class="line"><span class="cl">api-us-east-1-demo.example.com/.env
</span></span><span class="line"><span class="cl">api-eu-uat.example.com/wp-config.php~
</span></span><span class="line"><span class="cl">app-test-us-east-1.example.com/actuator/configprops
</span></span><span class="line"><span class="cl">prod-api-use2.example.com/.env.local
</span></span><span class="line"><span class="cl">relay.example.com/.env.production</span></span></code></pre></div></div>
<p>三十来个主机名，你能想到的每一种 <code>.env</code> 变体，外加冲着 Django 应用要的某个 Java 框架的 actuator 端点。它们全部拿到了指向规范主机的 301，什么也没被返回。乏味，而且确实无害。</p>
<p>不过既然分析页面开着，我顺手把同一天按状态码分组了。真正的问题在那里。</p>
<table>
	<thead>
			<tr>
					<th>状态码</th>
					<th>24 小时请求数</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>总计</td>
					<td>11,350</td>
			</tr>
			<tr>
					<td>504 Gateway Timeout</td>
					<td>2,474</td>
			</tr>
	</tbody>
</table>
<p>22%。而且不是均匀散布在整个站点，而是精确地压在最疼的地方：</p>
<table>
	<thead>
			<tr>
					<th>端点</th>
					<th>504</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>通知未读数</td>
					<td>621</td>
			</tr>
			<tr>
					<td>通知列表</td>
					<td>614</td>
			</tr>
			<tr>
					<td>消息未读数</td>
					<td>469</td>
			</tr>
			<tr>
					<td>信息流游标</td>
					<td>210</td>
			</tr>
	</tbody>
</table>
<p>这些都是移动端每 30 秒轮询一次的端点。所以这不是一个抽象的错误率数字，而是用户的手机整天、大概持续了好几周，都在悄悄加载不出未读角标。</p>

<h2 class="relative group">日志为什么看起来正常
    <div id="日志为什么看起来正常" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e6%97%a5%e5%bf%97%e4%b8%ba%e4%bb%80%e4%b9%88%e7%9c%8b%e8%b5%b7%e6%9d%a5%e6%ad%a3%e5%b8%b8" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>我的第一反应是应用变慢了。并没有。我从日志归档里翻出最糟糕那个小时里应用自己的请求日志：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">[2026-09-08 01:00:29] &#34;GET /api/notifications/unread/&#34; 304 61.615
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:29] &#34;GET /api/notifications/&#34;        304 63.568
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:59] &#34;GET /api/notifications/unread/&#34; 304 58.742
</span></span><span class="line"><span class="cl">[2026-09-08 01:00:59] &#34;GET /api/notifications/&#34;        304 105.438</span></span></code></pre></div></div>
<p>44 到 105 毫秒，清一色 304。源站很快。</p>
<p>这是关键线索，值得写成一条规则：<strong>如果边缘报告的错误是源站从没听说过的，那么请求是在到达之前就死掉了。</strong> 别再读应用日志，去看它前面的那一层。</p>
<p>在我这里就是 Fly.io 的代理，答案藏在机器的调度方式里。</p>

<h2 class="relative group">缩容到零是关于成本的承诺，不是关于延迟的
    <div id="缩容到零是关于成本的承诺不是关于延迟的" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e7%bc%a9%e5%ae%b9%e5%88%b0%e9%9b%b6%e6%98%af%e5%85%b3%e4%ba%8e%e6%88%90%e6%9c%ac%e7%9a%84%e6%89%bf%e8%af%ba%e4%b8%8d%e6%98%af%e5%85%b3%e4%ba%8e%e5%bb%b6%e8%bf%9f%e7%9a%84" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>我的配置是这样，乍看之下相当合理：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;stop&#39;</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_start_machines</span> <span class="p">=</span> <span class="kc">true</span>
</span></span><span class="line"><span class="cl">  <span class="nx">min_machines_running</span> <span class="p">=</span> <span class="mi">1</span></span></span></code></pre></div></div>
<p>空闲的机器停掉，来流量再启动，用多少付多少。对于流量有峰谷的小应用，这正是你想要的，也正因如此我定义了十台机器，平时却只有一台在跑。</p>
<p>它藏起来的账单是延迟。当一个请求打到一台已停止的机器上，总得有人等它启动完。如果启动比代理的耐心更久，这个人就会拿到 504。</p>
<p>那么我的机器启动要多久？我从来没量过。日志归档知道，因为启动脚本会打印每个阶段，而且每一行都带着机器 ID：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">WITH</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">SELECT</span><span class="w"> </span><span class="n">instance</span><span class="p">,</span><span class="w"> </span><span class="k">timestamp</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">         </span><span class="k">CASE</span><span class="w"> </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Starting app server%&#39;</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;begin&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Migrations complete%&#39;</span><span class="w"> </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;migrated&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">              </span><span class="k">WHEN</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">LIKE</span><span class="w">  </span><span class="s1">&#39;%&#34;GET /healthz%&#39;</span><span class="w">       </span><span class="k">THEN</span><span class="w"> </span><span class="s1">&#39;serving&#39;</span><span class="w"> </span><span class="k">END</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">phase</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">FROM</span><span class="w"> </span><span class="n">logs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">WHERE</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;myapp&#39;</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="k">timestamp</span><span class="p">::</span><span class="nb">date</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;2026-09-08&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="n">instance</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;begin&#39;</span><span class="p">)</span><span class="w">    </span><span class="k">AS</span><span class="w"> </span><span class="n">started</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;migrated&#39;</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">migrated</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">min</span><span class="p">(</span><span class="k">timestamp</span><span class="p">)</span><span class="w"> </span><span class="n">FILTER</span><span class="w"> </span><span class="p">(</span><span class="k">WHERE</span><span class="w"> </span><span class="n">phase</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;serving&#39;</span><span class="p">)</span><span class="w">  </span><span class="k">AS</span><span class="w"> </span><span class="n">first_serve</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="n">s</span><span class="w"> </span><span class="k">GROUP</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="n">instance</span></span></span></code></pre></div></div>
<p>在每一台机器上，答案都一致：</p>
<table>
	<thead>
			<tr>
					<th>阶段</th>
					<th>耗时</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>启动到迁移完成</td>
					<td>10 到 13 秒</td>
			</tr>
			<tr>
					<td>迁移到处理第一个请求</td>
					<td>8 到 19 秒</td>
			</tr>
			<tr>
					<td><strong>合计</strong></td>
					<td><strong>25 到 33 秒</strong></td>
			</tr>
	</tbody>
</table>
<p>然后我数了数这件事发生的频率：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="k">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">starts</span><span class="p">,</span><span class="w"> </span><span class="k">count</span><span class="p">(</span><span class="k">DISTINCT</span><span class="w"> </span><span class="n">instance</span><span class="p">)</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="n">machines</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">FROM</span><span class="w"> </span><span class="n">logs</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="k">WHERE</span><span class="w"> </span><span class="n">app</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;myapp&#39;</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="k">timestamp</span><span class="p">::</span><span class="nb">date</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;2026-09-08&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">AND</span><span class="w"> </span><span class="n">message</span><span class="w"> </span><span class="k">ILIKE</span><span class="w"> </span><span class="s1">&#39;%Starting app server%&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="c1">-- starts: 92, machines: 10</span></span></span></code></pre></div></div>
<p>一天 92 次冷启动。每一次都打开一个半分钟的窗口，任何被路由到那台机器的请求都会超时。22% 不再神秘了。</p>

<h2 class="relative group">原因就在我自己的 entrypoint 里
    <div id="原因就在我自己的-entrypoint-里" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e5%8e%9f%e5%9b%a0%e5%b0%b1%e5%9c%a8%e6%88%91%e8%87%aa%e5%b7%b1%e7%9a%84-entrypoint-%e9%87%8c" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>这是每一台机器在响应任何请求之前都要跑的东西：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="c1"># entrypoint.sh, run on every machine start</span>
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Running migrations...&#34;</span>
</span></span><span class="line"><span class="cl">python manage.py migrate --noinput
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">echo</span> <span class="s2">&#34;Running collectstatic in background...&#34;</span>
</span></span><span class="line"><span class="cl">python manage.py collectstatic --noinput <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">exec</span> <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span>    <span class="c1"># then start the app server</span></span></span></code></pre></div></div>
<p>带着「一天 92 次」再读一遍。</p>
<p><code>migrate</code> 是一次完整的 Django 启动：导入所有模型、连上 Postgres、查询迁移表，然后判定无事可做。在共享 CPU 上，为了什么也不做花掉十秒。<code>collectstatic</code> 是第二次完整的 Django 启动，之后还要把静态文件上传到对象存储，同时和正在启动的 Web 服务器抢同一个核心。应用服务器是第三次启动。</p>
<p>三个 Python 解释器，其中两个在重复已经做过的事，机器每醒来一次就来一遍。</p>
<p>当初这么写的理由是站得住脚的，你的代码大概也长得差不多：迁移总得在某处执行，Fly 的 <code>release_command</code> 曾经卡住过，而 entrypoint 是唯一保证会在部署时执行的地方。把部署工作放在这里是对的，只是把它放在<strong>每次机器启动</strong>时不对。而一旦你的机器开始自己停自己起，这两件事就不再是同一个事件了。</p>
<p>整个 bug 就是这一句，我认为它相当常见：</p>
<blockquote><p>属于<strong>发布</strong>的工作，被按<strong>机器启动</strong>执行了。缩容到零把一个事件变成了九十二个。</p>
</blockquote>
<h2 class="relative group">解法：把发布只认领一次
    <div id="解法把发布只认领一次" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e8%a7%a3%e6%b3%95%e6%8a%8a%e5%8f%91%e5%b8%83%e5%8f%aa%e8%ae%a4%e9%a2%86%e4%b8%80%e6%ac%a1" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>某个镜像的第八台机器不需要跑迁移。这个镜像只需要跑一次，其他机器只需要知道已经跑过了。</p>
<p>这就是一把每次部署都换名字的锁。Fly 用 <code>FLY_IMAGE_REF</code> 白送了这个名字，而 Redis 我本来就有。整个东西是一个在 Django 存在之前就运行的小脚本，所以常见情况下的答案是毫秒级，而不是一次框架启动：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="c1"># release_gate.py: runs before Django exists. Exit code 0 means &#34;prepare this release&#34;.</span>
</span></span><span class="line"><span class="cl"><span class="kn">import</span> <span class="nn">os</span><span class="o">,</span> <span class="nn">time</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="n">PREPARE</span><span class="p">,</span> <span class="n">SKIP</span> <span class="o">=</span> <span class="mi">0</span><span class="p">,</span> <span class="mi">1</span>
</span></span><span class="line"><span class="cl"><span class="n">LOCK_TTL</span><span class="p">,</span> <span class="n">WAIT</span><span class="p">,</span> <span class="n">POLL</span> <span class="o">=</span> <span class="mi">300</span><span class="p">,</span> <span class="mi">240</span><span class="p">,</span> <span class="mi">2</span>
</span></span><span class="line"><span class="cl"><span class="n">_sleep</span><span class="p">,</span> <span class="n">_monotonic</span> <span class="o">=</span> <span class="n">time</span><span class="o">.</span><span class="n">sleep</span><span class="p">,</span> <span class="n">time</span><span class="o">.</span><span class="n">monotonic</span>    <span class="c1"># aliases, so tests can replace them</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">release_id</span><span class="p">()</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Something that changes exactly when the deployed image does.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_IMAGE_REF&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_MACHINE_VERSION&#34;</span><span class="p">)</span> <span class="ow">or</span> <span class="s2">&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">claim</span><span class="p">(</span><span class="n">client</span><span class="p">,</span> <span class="n">release</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">int</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">or</span> <span class="ow">not</span> <span class="n">release</span><span class="p">:</span>   <span class="c1"># no Redis, or no idea which release: behave as before</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">PREPARE</span>
</span></span><span class="line"><span class="cl">    <span class="n">done_key</span><span class="p">,</span> <span class="n">lock_key</span> <span class="o">=</span> <span class="sa">f</span><span class="s2">&#34;release:</span><span class="si">{</span><span class="n">release</span><span class="si">}</span><span class="s2">:done&#34;</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&#34;release:</span><span class="si">{</span><span class="n">release</span><span class="si">}</span><span class="s2">:lock&#34;</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">done_key</span><span class="p">):</span>            <span class="c1"># somebody already did it for this image</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">SKIP</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="n">owner</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">getenv</span><span class="p">(</span><span class="s2">&#34;FLY_MACHINE_ID&#34;</span><span class="p">,</span> <span class="s2">&#34;unknown&#34;</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">    <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">lock_key</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="n">LOCK_TTL</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="n">PREPARE</span>                  <span class="c1"># we won the race; we do the work</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="c1"># Someone else is preparing. Wait for them, because serving requests</span>
</span></span><span class="line"><span class="cl">    <span class="c1"># against a half-migrated schema is worse than starting slowly.</span>
</span></span><span class="line"><span class="cl">    <span class="n">deadline</span> <span class="o">=</span> <span class="n">_monotonic</span><span class="p">()</span> <span class="o">+</span> <span class="n">WAIT</span>
</span></span><span class="line"><span class="cl">    <span class="k">while</span> <span class="n">_monotonic</span><span class="p">()</span> <span class="o">&lt;</span> <span class="n">deadline</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="n">_sleep</span><span class="p">(</span><span class="n">POLL</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">done_key</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">SKIP</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">client</span><span class="o">.</span><span class="n">set</span><span class="p">(</span><span class="n">lock_key</span><span class="p">,</span> <span class="n">owner</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="n">LOCK_TTL</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="n">PREPARE</span>              <span class="c1"># the owner died and its lock expired; take over</span>
</span></span><span class="line"><span class="cl">    <span class="k">return</span> <span class="n">PREPARE</span>                      <span class="c1"># waited long enough; do it ourselves</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># ... plus `done` (sets done_key) and a main() that treats any Redis error as PREPARE</span></span></span></code></pre></div></div>
<p>entrypoint 变成一个 <code>if</code>：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sh" data-lang="sh"><span class="line"><span class="cl"><span class="k">if</span> python3 release_gate.py claim<span class="p">;</span> <span class="k">then</span>
</span></span><span class="line"><span class="cl">    python manage.py migrate --noinput
</span></span><span class="line"><span class="cl">    python manage.py collectstatic --noinput <span class="p">&amp;</span>
</span></span><span class="line"><span class="cl">    python3 release_gate.py <span class="k">done</span>
</span></span><span class="line"><span class="cl"><span class="k">fi</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="nb">exec</span> <span class="s2">&#34;</span><span class="nv">$@</span><span class="s2">&#34;</span></span></span></code></pre></div></div>

<h2 class="relative group">每一种失败都必须落在「那就准备」上
    <div id="每一种失败都必须落在那就准备上" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e6%af%8f%e4%b8%80%e7%a7%8d%e5%a4%b1%e8%b4%a5%e9%83%bd%e5%bf%85%e9%a1%bb%e8%90%bd%e5%9c%a8%e9%82%a3%e5%b0%b1%e5%87%86%e5%a4%87%e4%b8%8a" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>比起代码，这一节更值得抄走。</p>
<p>这样一个门闩，站在你的用户和一次库迁移之间。如果它朝<strong>跳过</strong>的方向出错，你就会对着一个没迁移的数据库提供服务，那是真正的故障。如果它朝<strong>准备</strong>的方向出错，你只是把一次幂等的空迁移跑了两遍，浪费十秒。</p>
<p>这两个结果并不对称，所以代码也不能对称地对待它们。每一条不确定的路径都返回 <code>PREPARE</code>：</p>
<ul>
<li><strong>没有 Redis，或者 Redis 不可达。</strong> 准备。这正是旧行为，于是缓存故障退化成「慢」，而不是「坏」。</li>
<li><strong>环境里没有镜像引用。</strong> 准备。我不知道这是哪个发布，也就无从知道它是否就绪。</li>
<li><strong>锁的持有者在迁移途中死了。</strong> 它的锁过期，下一个等待者接管并准备。</li>
<li><strong>等待超过了超时时间。</strong> 还是准备。一台永远不提供服务的机器，比一次重复迁移更糟。</li>
</ul>
<p>反方向上我故意留了一处不对称：发现另一台机器确实握着锁的机器会<strong>等待</strong>，而不是开始服务。部署期间启动慢一点没关系，对着只迁移了一半的表结构接受查询则不行。</p>
<p>如果你把它建在一个会吞掉连接错误的缓存包装器上（我一开始就是这么做的），有一个陷阱。这种包装器在 Redis 不可达时让 <code>add()</code> 返回 <code>False</code>，而这读起来和「别人正持有锁」一模一样，那会让每台机器都跳过迁移——恰好是危险的那个方向。用回读锁来区分：一把没有任何人持有的锁，意味着缓存坏了，而不是你输掉了竞争。</p>

<h2 class="relative group">不用 Redis 也能测
    <div id="不用-redis-也能测" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e4%b8%8d%e7%94%a8-redis-%e4%b9%9f%e8%83%bd%e6%b5%8b" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>这段逻辑值得写单元测试，因为真正有意思的路径恰恰是你没法手工复现的。一个两方法的假对象就够了：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">class</span> <span class="nc">FakeRedis</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">    <span class="s2">&#34;&#34;&#34;Enough of a Redis client for the gate: get, and set with nx/ex.&#34;&#34;&#34;</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="kc">None</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">:</span> <span class="nb">dict</span><span class="p">[</span><span class="nb">str</span><span class="p">,</span> <span class="nb">str</span><span class="p">]</span> <span class="o">=</span> <span class="p">{}</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">get</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">):</span> <span class="k">return</span> <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">key</span><span class="p">)</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">set</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span><span class="p">,</span> <span class="n">nx</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">ex</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="k">if</span> <span class="n">nx</span> <span class="ow">and</span> <span class="n">key</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">:</span>
</span></span><span class="line"><span class="cl">            <span class="k">return</span> <span class="kc">None</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">store</span><span class="p">[</span><span class="n">key</span><span class="p">]</span> <span class="o">=</span> <span class="n">value</span>
</span></span><span class="line"><span class="cl">        <span class="k">return</span> <span class="kc">True</span></span></span></code></pre></div></div>
<p>然后按自己的节奏推动时间，去跑那些别扭的分支。这也是门闩调用模块级的 <code>_sleep</code> 和 <code>_monotonic</code> 别名，而不直接调用 <code>time.sleep</code> 的原因，这样测试替换它们时不会波及同进程里的其他线程：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-python" data-lang="python"><span class="line"><span class="cl"><span class="k">def</span> <span class="nf">test_a_waiter_takes_over_when_the_owner_disappears</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">    <span class="n">redis</span> <span class="o">=</span> <span class="n">FakeRedis</span><span class="p">()</span>
</span></span><span class="line"><span class="cl">    <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">release_gate</span><span class="o">.</span><span class="n">claim</span><span class="p">(</span><span class="n">redis</span><span class="p">,</span> <span class="s2">&#34;img-1&#34;</span><span class="p">),</span> <span class="n">release_gate</span><span class="o">.</span><span class="n">PREPARE</span><span class="p">)</span>   <span class="c1"># the owner takes the lock</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">def</span> <span class="nf">owner_dies</span><span class="p">(</span><span class="n">_seconds</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="n">redis</span><span class="o">.</span><span class="n">store</span><span class="o">.</span><span class="n">pop</span><span class="p">(</span><span class="s2">&#34;release:img-1:lock&#34;</span><span class="p">,</span> <span class="kc">None</span><span class="p">)</span>                              <span class="c1"># its lock expired</span>
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl">    <span class="k">with</span> <span class="n">patch</span><span class="o">.</span><span class="n">object</span><span class="p">(</span><span class="n">release_gate</span><span class="p">,</span> <span class="s2">&#34;_sleep&#34;</span><span class="p">,</span> <span class="n">owner_dies</span><span class="p">):</span>
</span></span><span class="line"><span class="cl">        <span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="n">release_gate</span><span class="o">.</span><span class="n">claim</span><span class="p">(</span><span class="n">redis</span><span class="p">,</span> <span class="s2">&#34;img-1&#34;</span><span class="p">),</span> <span class="n">release_gate</span><span class="o">.</span><span class="n">PREPARE</span><span class="p">)</span></span></span></code></pre></div></div>

<h2 class="relative group">有效果吗，一半
    <div id="有效果吗一半" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e6%9c%89%e6%95%88%e6%9e%9c%e5%90%97%e4%b8%80%e5%8d%8a" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>部署之后，停掉一台机器再启动，同时看着表。现在启动过程会说出自己的判断，然后立刻让路：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">15:23:34  Starting app server...
</span></span><span class="line"><span class="cl">15:23:36  release gate: this release is already prepared, starting straight away
</span></span><span class="line"><span class="cl">15:23:44  &#34;GET /healthz HTTP/1.1&#34; 200</span></span></code></pre></div></div>
<p>十秒，第二台量到十一秒。对比之前的 25 到 33 秒，这个窗口消失了三分之二，而门闩本身只花两秒，因为它从不导入 Django。</p>
<p>但十秒还是十秒。落在这十秒最开头的请求依然会失败。于是我去找剩下的部分。</p>

<h2 class="relative group">没有人在缓存字节码
    <div id="没有人在缓存字节码" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e6%b2%a1%e6%9c%89%e4%ba%ba%e5%9c%a8%e7%bc%93%e5%ad%98%e5%ad%97%e8%8a%82%e7%a0%81" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>镜像里有一行，几乎每一个 Python Dockerfile 都写着它：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="cl"><span class="k">ENV</span> <span class="nv">PYTHONDONTWRITEBYTECODE</span><span class="o">=</span><span class="m">1</span></span></span></code></pre></div></div>
<p>这是好建议。容器不该在运行时把 <code>.pyc</code> 涂进镜像层里。我没有想透的是这笔交易的另一半：如果运行时从不写字节码，而构建时也不写，那就没有任何东西被缓存，于是每一次进程启动都要把整棵依赖树从源码编译一遍。</p>
<p>我数了数：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">site-packages .py  files : 5,707
</span></span><span class="line"><span class="cl">site-packages .pyc files : 0</span></span></code></pre></div></div>
<p>Django、REST 框架、每一个库，在那 92 次日常启动里每次都重新编译。在生产机器上实测：</p>
<table>
	<thead>
			<tr>
					<th><code>django.setup()</code></th>
					<th>耗时</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>出厂状态</td>
					<td>5.15 秒</td>
			</tr>
			<tr>
					<td><code>compileall</code> 之后</td>
					<td>3.17 秒</td>
			</tr>
	</tbody>
</table>
<p>修复是一行，而且 <code>compileall</code> 是显式写入，所以那个环境变量拦不住它：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-dockerfile" data-lang="dockerfile"><span class="line"><span class="cl"><span class="k">RUN</span> python -m compileall -q /app/.venv/lib /app/myapp <span class="o">||</span> true</span></span></code></pre></div></div>
<p>十六秒的构建时间，只此一次，还在一个可缓存的层里。</p>
<p>顺手还发现了一个更安静的问题。<code>.dockerignore</code> 里写着：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">__pycache__/
</span></span><span class="line"><span class="cl">*.pyc</span></span></code></pre></div></div>
<p>Docker 是拿它们去匹配<strong>整条</strong>相对路径，而不是路径的每一段，所以没有锚定的模式只会在构建上下文的根目录生效。所有嵌套的 <code>__pycache__</code> 都被打进了镜像。里面有 218 个由笔记本上的 Python 3.15 编译出来的 <code>.pyc</code>，躺在一个解释器是 3.14 的镜像里，被安静地全部忽略。正确的写法是 <code>**/__pycache__/</code> 和 <code>**/*.pyc</code>。</p>
<p>诚实的结果：端到端只换来大约一秒，而不是两秒。十秒变成九秒。孤立的基准测试总是这样高估自己。</p>

<h2 class="relative group">挂起，以及挡住它的那一行
    <div id="挂起以及挡住它的那一行" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e6%8c%82%e8%b5%b7%e4%bb%a5%e5%8f%8a%e6%8c%a1%e4%bd%8f%e5%ae%83%e7%9a%84%e9%82%a3%e4%b8%80%e8%a1%8c" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>到了九秒，我已经没有可以再删的工作了。剩下的是一次无法避免的启动：Python 起来、Django 导入、WSGI 应用在一个共享核心上立起来。</p>
<p>那就别启动。Fly 可以给机器的内存做快照，而不是把它关掉：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="p">[</span><span class="nx">http_service</span><span class="p">]</span>
</span></span><span class="line"><span class="cl">  <span class="nx">auto_stop_machines</span> <span class="p">=</span> <span class="s1">&#39;suspend&#39;</span>   <span class="c"># was &#39;stop&#39;</span></span></span></code></pre></div></div>
<p>我去测试，Fly 拒绝了我，并给出了整件事里最有用的一条错误信息：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">failed to suspend VM: failed_precondition:
</span></span><span class="line"><span class="cl">Machines with swap cannot be suspended</span></span></code></pre></div></div>
<p>我的配置顶部有这么一段，注释是我自己写的，我也一直相信它：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-toml" data-lang="toml"><span class="line"><span class="cl"><span class="c"># Swap cushion so a memory spike swaps to disk instead of getting OOM-killed.</span>
</span></span><span class="line"><span class="cl"><span class="nx">swap_size_mb</span> <span class="p">=</span> <span class="mi">2048</span></span></span></code></pre></div></div>
<p>听起来是合理的预防。它真的在干活吗？每一台机器的回答都是：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">MemTotal:   985220 kB
</span></span><span class="line"><span class="cl">SwapTotal: 2097148 kB
</span></span><span class="line"><span class="cl">SwapFree:  2097148 kB</span></span></code></pre></div></div>
<p>一页都没有换出过。当天的日志里也没有 OOM kill——那 11 行匹配到 &ldquo;oom&rdquo; 的记录，是对一个内容哈希里恰好含有这三个字母的 JavaScript 包的请求。而真正的内存护栏在完全不同的地方，在应用服务器自己的配置里：任何 worker 一旦超过 800MB 就会被重启，远在触到 1GB 天花板之前。</p>
<p>所以这层缓冲是一份针对从未发生过的事件的保险，而且已经被另一套机制覆盖，它的保费是挂起能力。删掉。</p>
<table>
	<thead>
			<tr>
					<th>唤醒路径</th>
					<th>到能响应</th>
			</tr>
	</thead>
	<tbody>
			<tr>
					<td>今天早上的冷启动</td>
					<td>25 到 33 秒</td>
			</tr>
			<tr>
					<td>加了门闩和字节码之后的冷启动</td>
					<td>9 到 10 秒</td>
			</tr>
			<tr>
					<td><strong>从挂起恢复</strong></td>
					<td><strong>2.5 到 3.1 秒</strong></td>
			</tr>
	</tbody>
</table>
<p>测了三次，波动不超过半秒。唤醒之后每个端点都在 250 到 435 毫秒内返回 200。</p>

<h2 class="relative group">机器睡着时，你的套接字会怎样
    <div id="机器睡着时你的套接字会怎样" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e6%9c%ba%e5%99%a8%e7%9d%a1%e7%9d%80%e6%97%b6%e4%bd%a0%e7%9a%84%e5%a5%97%e6%8e%a5%e5%ad%97%e4%bc%9a%e6%80%8e%e6%a0%b7" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>不会有好事，而且没有钩子可以补救。Fly 冻结整个虚拟机：进程收不到信号，下坠途中什么也关不掉。它握着的连接在醒来时仍然留在内存里，指向对端几分钟前就已经丢弃的套接字。</p>
<p>这个问题不在下去的时候解，而在上来的时候解，而且多半取决于你之前是否已经做对了选择：</p>
<ul>
<li><strong>数据库连接按请求打开</strong>（<code>conn_max_age = 0</code>），没有活得够久、够得着变质的连接。</li>
<li><strong>Redis 缓存是降级而不是抛异常。</strong> 包装器捕获连接错误并返回一次未命中。</li>
</ul>
<p>我亲眼看到第二条挣到了它的位置。某次恢复时，日志里出现了完全在预料之中的失败：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">redis.exceptions.ConnectionError: Error while reading from fly-...</span></span></code></pre></div></div>
<p>而它所在的那个请求，用 3.6 秒返回了 200——死掉的缓存意味着慢一点的页面，而不是错误页面。之后的恢复什么都没记。</p>
<p>如果你的缓存客户端在连接死掉时会抛异常，挂起就会把每一次唤醒变成一串 500。在打开这个开关<strong>之前</strong>确认，而不是之后。</p>

<h2 class="relative group">那个骗了我的测试
    <div id="那个骗了我的测试" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e9%82%a3%e4%b8%aa%e9%aa%97%e4%ba%86%e6%88%91%e7%9a%84%e6%b5%8b%e8%af%95" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>我第一次测挂起，结论是这功能糟透了：502，30 秒之后，而且连着两次。</p>
<p>我用 <code>fly-force-instance-id</code> 把请求钉在了某一台机器上。用来压测单个实例时这是个好头部，用在这里则糟糕透顶：强制指定实例会绕过本该把它唤醒的代理逻辑。Fly 其实早就告诉我了，只要我把它当成答案而不是错误来读：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">machine was recently stopped and is unavailable to service request</span></span></code></pre></div></div>
<p>有意义的数字来自真实流量走的那条路。经由边缘发出 40 个并发请求，远超 12 的软上限，逼得代理必须扩容到挂起的机器上：</p>
<div class="highlight-wrapper"><div class="highlight"><pre tabindex="0" class="chroma"><code class="language-text" data-lang="text"><span class="line"><span class="cl">status codes:  40 × 200
</span></span><span class="line"><span class="cl">slowest:       2.1s</span></span></code></pre></div></div>
<p>两个教训，贵的是第二个。去测量用户走的那条路，而不是方便插桩的那条路。以及，当你的工具拒绝你的时候，去读那句拒绝——&ldquo;Machines with swap cannot be suspended&rdquo; 这一句，打开了整个下午。</p>

<h2 class="relative group">我建议你在自己的应用里检查什么
    <div id="我建议你在自己的应用里检查什么" class="anchor"></div>
    
    <span
        class="absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100 select-none">
        <a class="text-primary-300 dark:text-neutral-700 !no-underline" href="#%e6%88%91%e5%bb%ba%e8%ae%ae%e4%bd%a0%e5%9c%a8%e8%87%aa%e5%b7%b1%e7%9a%84%e5%ba%94%e7%94%a8%e9%87%8c%e6%a3%80%e6%9f%a5%e4%bb%80%e4%b9%88" aria-label="锚点">#</a>
    </span>
    
</h2>
<p>如果你跑在任何会停掉空闲实例的平台上，下面这些问题我花了半天，本可以省下好几周的静默失败：</p>
<ol>
<li><strong>冷启动要多久？</strong> 不是容器跑起来要多久，而是到它能响应一个真实请求要多久。如果你没法用秒数直接回答，就在需要它之前先量出来。</li>
<li><strong>你的 entrypoint 里，哪些按启动执行的事其实属于部署？</strong> 迁移、静态文件收集、缓存预热、索引构建。做一次都没问题，做九十二次都很贵。</li>
<li><strong>有东西在缓存你的字节码吗？</strong> 只有 <code>PYTHONDONTWRITEBYTECODE</code> 而构建里没有 <code>compileall</code>，答案就是没有。</li>
<li><strong>你能用挂起代替停止吗？</strong> 如果不能，是什么挡着？我这边是一个从未被碰过的 swap。</li>
<li><strong>边缘看得见、源站看不见的错误存在吗？</strong> 把两边比一比。那个差值不是统计口径的怪癖，而是死在中间的请求，你的应用日志永远不会显示它们。</li>
</ol>
<p>顺带一提，这段时间里健康检查一直是绿的。它总是绿的：健康检查跑在一台已经起来的机器上，它没法报告这台机器存在之前的那三十秒。</p>
]]></content:encoded>
      <media:content url="https://jared.lynskey.co.nz/en/posts/2026/2026-09-09-scale-to-zero-504s/featured.jpg" medium="image" type="image/jpeg" />
    </item>
  </channel>
</rss>
