Making Alibaba OSS Actually Behave Like S3 — 'S3-Compatible' Is Not a Contract

Shellby’s sync is bring-your-own-storage: fill in an S3-compatible bucket, and multi-device config syncs through it. The remote holds one encrypted ciphertext object plus a version tag (ETag), with conditional writes for optimistic concurrency. On AWS S3 and Cloudflare R2, everything works. Then a user filled in Alibaba OSS, and sync failed outright.

Tracing that ticket down led to a string of “S3-compatible” pits. This post strings them together — each says the same thing: “S3-compatible” only means the protocol looks alike; the addressing constraints, optional features, error carriers, and connection lifecycles can all differ.

Pit one: a 403, and the body is empty

The first symptom is the most discouraging: connecting to OSS returns 403, with an empty body — you can’t tell if it’s a signature error, a permission error, or a bad key.

The root cause is addressing style. S3 object addresses have two forms: path-style (endpoint/bucket/key) and virtual-hosted (bucket.endpoint/key, bucket in the domain). AWS and R2 accept both, while Alibaba OSS only accepts virtual-hosted — a path-style request gets 403’d by its routing layer before it ever reaches authentication, which is why there’s no error body.

The fix auto-detects the addressing style: a DNS-hostname endpoint (not IP, not localhost) with a DNS-compliant bucket → virtual-hosted; an IP/localhost endpoint or an illegal bucket name (e.g. with an underscore) → path-style, preserving MinIO and local testing.

bool _computeVirtualHosted() {
  final host = endpoint.split(':').first;
  final isIp = RegExp(r'^\d{1,3}(\.\d{1,3}){3}$').hasMatch(host) || host == 'localhost';
  final dnsSafeBucket = RegExp(r'^[a-z0-9][a-z0-9.-]{1,61}[a-z0-9]$').hasMatch(bucket);
  return !isIp && dnsSafeBucket;
}

But there’s a linked hidden pit here that earns you another 403 if you miss it: change the addressing style, and the SigV4 canonical request must change too. In virtual-hosted mode the signed path is just /key; in path-style it’s /bucket/key, and the host header comes from the URI as bucket.endpoint. If the signed string and the actual request disagree, OSS returns SignatureDoesNotMatch.

Pit two: 400 NotImplemented — it doesn’t support conditional writes

With addressing fixed, PUT 400s again, this time with text in the body: NotImplemented.

Our sync engine uses conditional writes for optimistic concurrency: first upload with If-None-Match: * (require the object not to exist), updates with If-Match: "etag" (compare-and-swap, to avoid overwriting someone else’s write). OSS’s S3-compatible API doesn’t support these conditional headers, and 400s when you include them.

You can’t drop optimistic concurrency for every backend just because one doesn’t support it. The fix is a runtime feature-probe + downgrade: the provider holds a _conditionalPut flag; the first PUT carries the condition headers; if it returns 400/501 with a body containing NotImplemented, it sets the flag false, retries once without the headers, and does plain writes thereafter. On R2/AWS the flag stays true, optimistic concurrency unchanged.

var resp = await _send('PUT', key, body, headers(conditional: _conditionalPut));
if (_conditionalPut &&
    (resp.statusCode == 400 || resp.statusCode == 501) &&
    resp.body.contains('NotImplemented')) {
  _conditionalPut = false;                       // probed unsupported, permanently downgrade
  resp = await _send('PUT', key, body, headers(conditional: false));
}

The downgrade is only safe because the upper layer has fallback semantics: with CAS gone, concurrent conflicts fall to the engine’s record-level LWW (last-write-wins) + tombstone deterministic merge, and the “pull-merge-then-land” order guarantees the other side’s update isn’t lost. Without a merge fallback upstream, you can’t casually downgrade CAS — that would just permit lost updates.

Pit three: the error code swallowed by the status code

The first two pits were quick to locate thanks to something added along the way: surfacing the real error from the body.

Early on the provider showed only the status code on any non-2xx — s3 put 403, s3 get 500 — smearing signature, permission, and key problems all into one “403”. But the real cause for S3/OSS is in the body XML’s <Code> / <Message>. Add a parser:

static String _errBody(http.Response resp) {
  String? tag(String t) => RegExp('<$t>(.*?)</$t>', dotAll: true).firstMatch(resp.body)?.group(1)?.trim();
  final code = tag('Code'), msg = tag('Message');
  if (code == null && msg == null) return '${resp.statusCode}';   // fall back to bare status code
  return '${resp.statusCode} ${code ?? ''}${msg != null ? ': $msg' : ''}';
}

The interesting part: pit one’s “403 empty body” lands exactly in the if (code == null && msg == null) fallback branch — and “the body is empty” is itself a diagnostic signal: likely the routing layer rejected it before auth, i.e. the addressing is wrong. Without surfacing the original error, this pit stays misdiagnosed as a key or signature problem forever.

Pit four: the connection gets closed, -1005

After the downgrade, a flaky class of failure appears: Flutter throws Connection closed before full header was received, Apple throws -1005 network connection lost. And it specifically targets the request after a prior 400 error response.

The root cause is connection lifecycle: OSS actively closes the keep-alive connection after a response (especially an error response), but the client’s connection pool doesn’t know, and the next request reuses that dead connection and fails transiently.

The fix adds transient retry, but the boundary is drawn hard — retry only network-level exceptions, not HTTP error responses:

for attempt in 0..<maxAttempts {
  if attempt > 0 { try? await Task.sleep(nanoseconds: 150_000_000 * UInt64(attempt)) }
  do {
    let (data, resp) = try await session.data(for: request)
    return (data, http)
  } catch let e as SyncProviderError { throw e                    // non-network → no retry
  } catch let e as URLError where e.code != .cancelled { lastError = e }  // transient → retry
}

A few deliberate choices: 4xx/5xx are normal returns, not exceptions, so they never enter the retry branch — 500s aren’t retried either, avoiding pounding a server’s deterministic error as if transient; user cancellation (.cancelled) is explicitly excluded; max 3 attempts, backoff 150ms × attempt; both stacks aligned (this pit was waded through in Flutter first, then ported verbatim to Apple, extracting the retry into a shared HTTPTransport for S3/WebDAV).

The confidence to retry rests on idempotency: PUT is a full-object overwrite of the same key (not an append), and each retry uses a freshly constructed request, so resending once has no side effect. Idempotent writes are what make retry safe — that’s the precondition.

Takeaways

• “S3-compatible” is a marketing word, not a contract. The consensus only goes as far as SigV4 signing plus basic GET/PUT; addressing style, conditional writes, error carrier, and connection lifecycle all need writing to the most conservative assumption, with runtime probing per backend, not compile-time “everyone’s like AWS”;
• A compatibility layer must surface the original error. Swapping the status code for the body’s <Code>/<Message> is what quickly revealed “this is actually addressing, not auth.” Even “the body is empty” is a signal — it often means the request was rejected by the routing layer before auth;
• Probe and downgrade optional features, but the downgrade needs a fallback. Probing once with a 400/NotImplemented and permanently downgrading to plain writes beats pre-configuring a per-vendor capability table; but only if the upper layer has an LWW merge fallback — otherwise dropping CAS just permits lost updates;
• Idempotent writes are what let you retry; retry only network exceptions. Full-object overwrite + a fresh request each time = idempotent; keep 4xx/5xx out of retry so you don’t pointlessly pound a server’s deterministic error. Incremental backoff + a cap of 3 + excluding user cancellation is a “sufficient, not excessive” default;
• The same pit must be aligned across every end. Behavioral consistency of a cross-platform provider has to be actively maintained — one end fixed while another runs bare is the easiest debt a multi-platform product accrues.