2. Design Dropbox

Topics: Handling Large Blobs, Resumable Uploads, File Synchronization

Intro

Dropbox is a cloud file storage service. Users upload files from their devices, download them elsewhere, share them with other users, and keep local folders synchronized with the cloud.

Final Design

						 +-----------------------+
						 | Web / Mobile / Desktop|
						 | Sync Client            |
						 +-----------+-----------+
									 |
						 +-----------v-----------+
						 | API Gateway /         |
						 | Load Balancer         |
						 +-----------+-----------+
									 |
						 +-----------v-----------+
						 | File Service          |
						 | auth, metadata, URLs  |
						 +-----+-----------+-----+
							   |           |
				+--------------v--+     +--v----------------+
				| Metadata DB     |     | Event Queue       |
				| files, ACLs,    |     | change fanout     |
				| versions, cursor|     +--+--------------+-+
				+-----------------+        |              |
										   |              v
							 +-------------v--+     +-----+--------+
							 | Blob Storage   |     | Sync / Push  |
							 | multipart data |     | WS or SSE    |
							 +-------+--------+     +--------------+
									 |
							  +------v------+
							  | CDN         |
							  | downloads   |
							  +-------------+

Functional Requirements ::4::

Features needed to satisfy the needs of the user.

In Scope

1. Users can upload files from web, mobile, and desktop devices.
2. Users can download files they own or that have been shared with them.
3. Users can share files with other users and view files shared with them.
4. A sync client detects local and remote changes and synchronizes files across devices.

Out of Scope

1. Editing files inside Dropbox.
2. Previewing files without downloading them.
3. User account creation and authentication flows; clients are assumed authenticated.
4. User storage quotas and billing.

Non Functional Requirements ::4::

Specifications about how the system operates, rather than the actions it provides.

Core Requirements And Benchmarks

1. Availability is preferred over immediate cross-region consistency. A remote change may take a short time to appear, but it must eventually synchronize.
2. Support files up to 50 GB without routing file bytes through application servers.
3. Protect file contents and metadata in transit and at rest; only authorized users can obtain access to a file.
4. Be durable and recoverable after infrastructure failure or accidental corruption.
5. Minimize upload, download, and synchronization time; support progress and resume.

Out of Scope

1. Strong, instantaneous consistency across every device and region.
2. File version history and user-configurable retention policies.
3. Antivirus and malware scanning.

General Estimations

These estimates show why a single request or application server should not carry a large file. User counts and average file sizes are unspecified, so total storage and request volume need workload assumptions before capacity planning.

A 50 GB file over a 100 Mbps connection takes about 4,000 seconds, or 1.1 hours, before protocol overhead and retries:

50 GB * 8 bits/byte / 100 Mbps = 4,000 seconds ~= 1.1 hours

With 8 MB chunks, a 50 GB upload is about 6,250 parts. A failed part can be retried without retransmitting the rest of the file. Parallel uploads can use available bandwidth more effectively, subject to client and service limits.

Set Up

Define the core data and API contract before building each workflow.

Entities

1. User: the authenticated owner or recipient of a file.
2. FileMetadata: file ID, owner, name, size, MIME type, object key, state, and current generation.
3. Blob: immutable file bytes, stored separately from metadata.
4. Permission: file ID, user ID, and access level.
5. ChangeEvent: an ordered record that lets a user's devices find changes since a cursor.
6. UploadSession: multipart upload ID, expected parts, and current state.

API ::4::

User identity comes from the authenticated request context, not a user ID supplied in the request body. The service authorizes every operation before issuing a URL.

1. Initiate Upload

POST /files/uploads
Body:
{
	"name": "project.zip",
	"size": 50000000000,
	"mime_type": "application/zip",
	"content_hash": "optional-file-fingerprint"
}

Response: HTTP 201 Created
{
	"file_id": "file-123",
	"upload_id": "upload-456",
	"part_size": 8000000
}

The client requests short-lived URLs for a small batch of parts when it is ready to upload them, rather than receiving thousands of URLs at initiation:

POST /files/{file_id}/uploads/{upload_id}/part-urls
Body: { "part_numbers": [1, 2, 3] }
Response: { "part_urls": [{ "part_number": 1, "url": "..." }, ...] }

2. Complete Upload

POST /files/{file_id}/uploads/{upload_id}/complete
Body:
{
	"parts": [
		{ "part_number": 1, "etag": "..." },
		{ "part_number": 2, "etag": "..." }
	]
}

Response: HTTP 200 OK
{
	"file_id": "file-123",
	"status": "available"
}

3. Get Download URL

GET /files/{file_id}/download-url

Response: HTTP 200 OK
{
	"download_url": "short-lived signed CDN URL",
	"expires_at": "..."
}

4. Share And Synchronize

POST /files/{file_id}/permissions
Body: { "user_id": "user-789", "access": "read" }

GET /files/shared
GET /files/changes?cursor={opaque_cursor}
Response: { "changes": [...], "next_cursor": "..." }

Change records include file ID, generation, event type, and updated metadata. An opaque cursor is safer than a timestamp alone: timestamps can tie, and clients need a stable point from which to continue after reconnecting.

Guiding Obstacles

The requirements point to four design challenges: safely moving large blobs, delivering downloads quickly, efficient sharing queries, and never losing sync changes.

1. Upload Files Up To 50 GB

Challenge In English

• A single request can exceed browser, gateway, and server limits and can run for hours.
• A connection failure should not force the client to restart from byte zero.
• Clients must not be trusted to claim that data was uploaded when it was not.

Solved Below

• Upload()
  • Direct-to-blob-storage presigned multipart upload
  • Part-level retry, progress, and completion verification

2. Fast And Reliable Downloads

Challenge In English

• Downloading through the File Service doubles network transfer and consumes its bandwidth.
• A single storage region adds latency for geographically distant users.
• Signed links must not bypass authorization checks or remain valid indefinitely.

Solved Below

• Download()
  • Direct signed download URL
  • CDN caching near the user

3. Sharing And Access Control

Challenge In English

• File owners need fast access to their files, and recipients need fast access to shared files.
• Storing every recipient in one file record makes recipient-to-file lookup expensive.
• Every generated download URL must follow a current permission check.

Solved Below

• Share()
  • Separate permission records indexed by user and file
  • Authorization before issuing signed URLs

4. Keep Devices In Sync

Challenge In English

• Devices may be offline, disconnect, or miss real-time notifications.
• A local watcher can report duplicate events or several writes for one save.
• Concurrent changes need deterministic handling even though full version history is out of scope.

Solved Below

• Sync()
  • Durable change log and cursor-based catch-up
  • WebSocket/SSE notifications as a low-latency hint

1. Upload() From Client To Cloud

Bare Bones

Start with the simplest design that satisfies the upload requirement, then fix its limits.

Components

1. Client: selects a file and sends it to the service.
2. File Service: authenticates the user, accepts bytes, and writes metadata.
3. Metadata Database: stores file properties and ownership.
4. Blob Storage: stores the raw file contents.

Upload Request

Action

Implements POST /files/uploads from Set Up > API > 1. Initiate Upload. The client eventually receives a file ID and an available file record.

Steps

1. Client requests an upload.
2. File Service authenticates and validates metadata and file size.
3. Service writes the file bytes to storage.
4. Service saves or updates FileMetadata.
5. Service reports success or failure to the client.

Diagram

1. Upload request()                         5. HTTP response()
						 +--------+             On Success: 201 Created
						 | Client |             { "file_id": "file-123" }
						 +---+----+             On Failure: 4xx / 5xx
							 |                  { "error": "..." }
							 v
					+--------+---------+
2. Validate()       |   File Service   |
3. Upload bytes()   +----+---------+---+
						 |         |
						 v         v
				  +------+---+  +--+----------------+
				  | Blob     |  | Metadata DB      |
				  | Storage  |  | fileId, owner,   |
				  +----------+  | name, size, state|
								+-------------------+

What Breaks

• The service carries the entire file through its network interface and memory/disk path.
• A 50 GB request is too long-lived for common request timeouts and payload limits.
• If the connection fails late, the client may have to resend the entire file.
• Writing bytes and metadata are separate operations, so either can succeed alone.

Bottleneck 1: Large, Resumable Uploads

Bad Solution: One Large POST Through The Service

Approach

Send the whole file in one request and let the File Service write it to Blob Storage.

Challenges

• Uploads can outlive client, gateway, and server timeouts.
• A retry retransmits bytes already sent.
• The application service pays the bandwidth and scaling cost for every file byte.

Good Solution: Direct Upload With A Presigned URL

Approach

The File Service authorizes the upload and generates a short-lived URL scoped to a server-generated object key. The client sends bytes directly to Blob Storage.

How It Works

1. The service creates a metadata row in uploading state and an object key that the client cannot choose.
2. The client requests short-lived presigned URLs for a small batch of parts.
3. The client uploads directly to Blob Storage and requests another batch as needed.
4. The client calls the completion endpoint; the service verifies the object and marks the file available.

Why It Works

• File bytes bypass the application servers.
• Signed URLs limit access to a specific object and operation for a short time.
• The service remains the control plane for authorization and metadata.

Challenges

• One interrupted upload still needs to start over unless storage supports multipart uploads.
• A file object can exist while metadata still says uploading, or metadata can refer to missing bytes.
• Presigned URLs are bearer credentials until they expire, so keep their lifetime short.

Great Solution: Multipart Upload With Resume

Approach

Split the file on the client into bounded parts, for example 8 MB each. The service starts a multipart upload and issues presigned URLs in small batches for parts the client is about to send. The client uploads parts in parallel within a limit, records successful part numbers and ETags, and retries only failed parts.

Client                   File Service              Blob Storage
  |                            |                         |
  | POST /files/uploads        |                         |
  |--------------------------->| CreateMultipartUpload  |
  |                            |------------------------>|
	| fileId + uploadId          |                         |
	|<---------------------------|                         |
	| POST part-urls [1..N]      |                         |
	|--------------------------->|                         |
	| signed URLs for [1..N]     |                         |
	|<---------------------------|                         |
  | PUT part 1, part 2, ... directly via signed URLs    |
  |----------------------------------------------------->|
  | POST /complete + part ETags|                         |
  |--------------------------->| ListParts / verify      |
  |                            | CompleteMultipartUpload|
  |                            |------------------------>|
  |                            | mark available + log change
  |<---------------------------|                         |

Why It Works

• Progress is measurable per part; failed parts can be retried independently.
• The client can pause and resume using the persisted upload session.
• Parallelism can improve throughput without increasing the size of each request.

Integrity And State

• Treat the service's completion operation as authoritative. Client-reported ETags are input, not proof.
• Verify uploaded parts with the storage provider's multipart listing API, then complete the upload.
• Mark metadata available only after the storage provider confirms assembly of the final object.
• Make completion idempotent so a retried request does not create duplicate file generations.
• Expire abandoned multipart sessions and remove orphaned objects with a cleanup job.
• Storage event notifications can trigger reconciliation, but part uploads generally do not each produce a completed-object event.

Challenges

• The service must persist upload IDs and part size/state so a client can resume.
• Multipart APIs have provider-specific limits on part count and size; choose part sizes to stay below them.
• A content fingerprint helps identify identical content but is not a file ID or an authorization credential.
• Encrypt or otherwise protect local upload-session state because it contains resumable session details.

Upload API Contract

POST /files/uploads
Response: HTTP 201 Created
{
	"file_id": "file-123",
	"upload_id": "upload-456",
	"part_size": 8000000,
	"part_urls": ["...", "..."]
}

POST /files/file-123/uploads/upload-456/complete
Body: { "parts": [{ "part_number": 1, "etag": "..." }] }
Response: HTTP 200 OK
{ "file_id": "file-123", "status": "available" }

Possible failures include 400 for invalid metadata or parts, 403 for an unauthorized user, 409 for an invalid upload state, and 5xx for a storage or service failure. The service returns success only after the final object is confirmed.

Data Model

FileMetadata
{
	file_id: UUID (primary key)
	owner_id: UUID
	name: string
	size_bytes: integer
	mime_type: string
	object_key: string
	current_generation: integer
	status: uploading | available | failed | deleted
	created_at: timestamp
	updated_at: timestamp
}

UploadSession
{
	upload_id: string (primary key)
	file_id: UUID
	owner_id: UUID
	storage_upload_id: string
	part_size_bytes: integer
	expires_at: timestamp
	status: active | completed | aborted
}

2. Download() From Cloud To Client

Bare Bones

Action

Implements GET /files/{file_id}/download-url. The service checks access and returns a temporary URL; the client downloads the bytes directly from storage.

Steps

1. Client requests a download URL.
2. File Service checks ownership or a current share permission.
3. Service returns a short-lived signed URL.
4. Client downloads the object directly from storage or the CDN.

What Breaks

• Proxying file bytes through the File Service doubles transfer and consumes service bandwidth.
• A single-region object store can be slow for users far away.
• Caching private objects without authorization-aware keys can expose one user's file to another.

Bottleneck 1: Download Latency And Throughput

Good Solution: Direct Signed Downloads

The File Service verifies the caller and issues a short-lived URL for the exact object and generation. The service never streams the file itself. Large downloads can use HTTP Range requests to resume or fetch byte ranges in parallel.

Great Solution: CDN For Private Content

Place a CDN in front of Blob Storage. The service returns a CDN signed URL or cookie; the CDN fetches the object on a cache miss and serves repeat requests from a nearby edge.

Challenges

• A signed URL is a bearer token: anyone holding it can use it until it expires.
• Use short expirations, private cache configuration, and keys that include object generation.
• Purge or version cache entries when content changes or access is revoked; never rely on stale cached authorization.
• CDN caching trades freshness and revocation speed against lower latency and storage egress cost.

Download API Contract

GET /files/{file_id}/download-url
Response: HTTP 200 OK
{
	"download_url": "https://cdn.example.com/signed/...",
	"expires_at": "..."
}

HTTP 403 Forbidden: caller does not have access
HTTP 404 Not Found: file does not exist or is unavailable

3. Share() With Other Users

Bare Bones

Action

An owner grants another user access to a file. The recipient can list files shared with them and request a download URL if permission allows it.

Bad Solution: Recipient List On The File

Store all recipient IDs inside the file metadata. This makes checking one file's recipients easy, but listing all files shared with a user requires scanning many files.

Good Solution: Permission Table

Store one record per file and recipient, indexed in both access patterns:

Permissions
| file_id (PK) | user_id (SK) | access | created_at |
|--------------|--------------|--------|------------|
| file-1       | user-2       | read   | ...        |
| file-1       | user-3       | read   | ...        |

UserSharedFiles
| user_id (PK) | file_id (SK) |
|--------------|--------------|
| user-2       | file-1       |
| user-3       | file-1       |

The second access path can be a database index or a materialized relation, depending on the database. If both are stored separately, update them transactionally or derive the recipient list from one canonical permission relation.

Share Steps

1. Owner sends POST /files/{file_id}/permissions with recipient and access level.
2. Service verifies the caller owns the file or can manage its permissions.
3. Service writes the permission and a change event for the recipient.
4. Recipient's shared-files query returns the file metadata they are allowed to see.
5. A later download URL request checks permission again before signing.

Challenges

• Permission changes must take effect for newly issued URLs immediately; existing signed URLs may remain usable until expiry.
• Sharing many files or users can create hot partitions; paginate and batch safely.
• Revocation and deletion should update permission records and emit sync events.

4. Sync() Across Devices

Bare Bones

Each desktop/mobile sync agent watches its local folder, queues changes, and uploads them. For remote changes, each client periodically asks for changes since its last sync.

Local To Remote

1. File system watcher reports a local create, update, rename, or delete.
2. Client debounces duplicate events and computes file identity/content metadata.
3. Client uploads changed content, using multipart upload for large files.
4. On successful completion, the service advances the file generation and appends a change event.

File-system watcher events are hints, not a perfect log. The client should periodically reconcile its local folder against its last known manifest to catch missed events.

Remote To Local

Bad Solution: Poll Every File

Checking each file individually causes many requests and does not scale with a user's file count.

Good Solution: Poll A Change Feed

The client calls GET /files/changes?cursor=.... The service returns an ordered page of events visible to that user and a next cursor. The client applies changes and persists the cursor only after the page is safely processed.

Great Solution: Push Plus Durable Catch-Up

Keep one WebSocket or SSE connection per active device/session. Push a notification when a relevant change occurs, then let the client fetch the authoritative event from the change feed. On reconnect, the client always resumes from its durable cursor; notifications are an optimization and can be duplicated or missed.

Local file watcher -> Sync Agent -> Upload API -> Blob Storage
											-> Metadata + Change Log
													   |
									 WebSocket/SSE hint |
													   v
Other Device <- Change Feed(cursor) <- Sync Service / File Service

Conflicts And Idempotency

• Each file update carries the generation the client last observed (optimistic concurrency).
• If the generation matches, create the next generation and publish a change event.
• If another device already wrote a newer generation, return a conflict so the client can reconcile.
• A simple product policy can keep both copies by saving the incoming update as a conflict copy; silently overwriting remote data risks data loss.
• Use idempotency keys for retried uploads and change application so reconnects do not create duplicate generations.
• Full version history is out of scope, but immutable object generations make recovery from partial failures safer.

Efficient Sync Of Large Files

Fixed-size chunks work for resumable upload, but they are poor for delta sync: inserting one byte near the start shifts later boundaries. Content-defined chunking (CDC) uses a rolling hash to find boundaries from content, so a small edit usually changes only nearby chunks. Store a manifest of chunk hashes and reuse unchanged chunks when supported.

This reduces transferred bytes but adds manifest storage, hashing CPU, and deduplication complexity. Start with multipart whole-file upload; add CDC when measured workloads justify it.

Deep Dives

1. How To Make Uploads And Downloads Fast

• Direct client-to-object-storage transfer keeps blob bytes off application servers.
• Multipart uploads allow bounded retries, progress reporting, and parallel transfer.
• Adapt part size and concurrency to bandwidth and device memory; too much parallelism can hurt.
• A CDN moves frequently requested downloads closer to users.
• CDC can reduce sync bytes when files change, while compression is useful mainly for compressible formats.
• Compression happens before encryption; already compressed media usually gains little.

2. How To Secure Files

1. Use HTTPS for client/API and signed storage transfers.
2. Encrypt blobs at rest with managed keys; tightly control key and bucket policies.
3. Keep buckets private. Generate short-lived, object-scoped signed URLs only after authorization.
4. Use server-generated object keys; never trust a client-selected path as an access-control boundary.
5. Check permissions when issuing each URL and make CDN caching private and generation-aware.
6. Treat fingerprints and ETags as integrity/resume metadata, not authorization secrets.

Signed URLs are bearer credentials: a recipient can forward one while it is valid. Short expiry limits exposure but does not prevent sharing. Stronger controls can require an authenticated CDN session or additional request restrictions.

3. How To Recover From Failure

• Blob storage provides durability features, but enable versioning/replication or backups according to recovery objectives.
• Keep metadata backups and a reconciliation process that detects metadata with missing objects and orphaned objects.
• Use an upload state machine (uploading, available, failed, deleted) and idempotent transitions.
• Retry transient storage operations with bounded exponential backoff; do not mark a file available before completion is confirmed.
• Retain change-log events long enough for offline devices, and return a full manifest resync if a cursor has expired.

Putting It Together

The File Service is the control plane: it authenticates requests, enforces ownership and sharing permissions, stores metadata, creates upload sessions, and issues signed URLs. Clients move file bytes directly to and from Blob Storage, with multipart upload for large files and a CDN for downloads. A metadata database stores file records and permissions; a durable change log supports cursor-based synchronization. Push notifications reduce sync delay, while polling/cursor catch-up provides correctness after disconnects.

The main trade-off is operational complexity: object storage, multipart state, permissions, CDN signing, and change delivery are more pieces than proxying bytes through one server. They are justified by the 50 GB file limit, resumability, security boundary, and scale of file transfer traffic.