# Moving your files from S3

Copy the files of a bucket at AWS, or at any other S3 service, into the bucket of an environment with brewless storage:import. Nothing is removed.


An application that comes from Laravel Vapor, or from anywhere else on AWS, keeps its uploads in an S3 bucket. `brewless storage:import` copies them into the bucket of an environment (see [A bucket for files](/applications/applications-and-environments#a-bucket-for-files)): you say where the files are now and give a key that may read them; where they go follows from the environment.

```bash
brewless storage:import production --from-env=../old-app/.env
```

## What does storage:import do?

It copies every file of the source bucket, or of one folder in it, into the environment's bucket. Nothing is removed, neither at the source nor in the environment's bucket, and nothing at the source is changed. A file that is in the environment's bucket already, with the same size and time, is left alone: running the import again copies only what is new or changed since, so you can copy everything now and once more just before you move your domain.

It needs an environment with a bucket for files and a key of its own for it, which Brewless makes when it sets the environment up (its Scaleway key needs IAMManager for that). Without them it says so and starts nothing.

## Which value goes where

The names are those of Laravel's `s3` disk in `config/filesystems.php`. For the **source** you give them; the **destination** is what Brewless already set in the environment's variables when it made the bucket.

| Laravel variable | Source: how you give it | Example at AWS | Destination (set by Brewless) |
| --- | --- | --- | --- |
| `AWS_BUCKET` | the argument `s3://<bucket>[/<folder>]`, `SOURCE_AWS_BUCKET` or `--from-env` | `my-vapor-bucket` | `<application>-<environment>-<first 8 characters of the project>` |
| `AWS_DEFAULT_REGION` | `--source-region`, `SOURCE_AWS_DEFAULT_REGION`, `--from-env` or the profile; at AWS it may be left out | `eu-west-1` | the region of the environment, such as `fr-par` |
| `AWS_ENDPOINT` | `--source-endpoint`, `SOURCE_AWS_ENDPOINT` or `--from-env`; **leave it out for AWS** | none | `https://s3.fr-par.scw.cloud` |
| `AWS_USE_PATH_STYLE_ENDPOINT` | `--source-path-style`, `SOURCE_AWS_USE_PATH_STYLE_ENDPOINT` or `--from-env` | `false` | `false` |
| `AWS_ACCESS_KEY_ID` | `SOURCE_AWS_ACCESS_KEY_ID`, `--from-env`, `--source-profile` or asked | `AKIA…` | the key of the identity `brewless-<application>-<environment>-files` |
| `AWS_SECRET_ACCESS_KEY` | `SOURCE_AWS_SECRET_ACCESS_KEY`, `--from-env`, `--source-profile` or asked | | idem |
| `AWS_URL` | not needed to copy | your CloudFront or S3 address | change or remove it after the move |

When a value is given in more than one way, the first of these wins: the option, then `SOURCE_AWS_*`, then `--source-profile`, then `--from-env`, and last the question in the terminal. The command says where it took the key from, never the key. `AWS_*` variables of your own shell are not used: they often belong to another AWS account. Run by a script without a terminal, a missing key stops the command.

A key is never an option on the command line, where it would end up in your shell's history.

### From the .env of your old application

```bash
brewless storage:import production --from-env=../old-app/.env
```

reads `AWS_BUCKET`, `AWS_DEFAULT_REGION`, `AWS_ENDPOINT`, `AWS_USE_PATH_STYLE_ENDPOINT`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` from that file and nothing else. The file is read on your machine and its values are not printed.

> [!NOTE]
> The `.env` of an application on Vapor usually has no `AWS_ACCESS_KEY_ID` or `AWS_SECRET_ACCESS_KEY`: on Vapor the application runs with an IAM role, and Vapor sets `AWS_BUCKET` itself. Make a key that may read the bucket (below) and give it as `SOURCE_AWS_*`. The name of the bucket is under `storage` in `vapor.yml`, which `brewless init --from-vapor` reads too.

### With variables

```bash
export SOURCE_AWS_ACCESS_KEY_ID=AKIA…
export SOURCE_AWS_SECRET_ACCESS_KEY=…
brewless storage:import production s3://my-vapor-bucket/uploads --source-region=eu-west-1
```

`/uploads` copies only that folder; without it the whole bucket is copied. A file keeps its name, folder included: `uploads/a.jpg` stays `uploads/a.jpg`.

### With a profile of the AWS CLI

```bash
brewless storage:import production s3://my-vapor-bucket --source-profile=migration
```

takes the key of that profile from `~/.aws/credentials` and its region from `~/.aws/config`.

## A key that may only read, at AWS

Do not use the key of your AWS account. Make one that may only read this bucket, and remove it once you have moved:

1. In the AWS console, under **IAM**, **Users**, create a user such as `brewless-migration`, without access to the console.
2. Give it an inline policy, with your bucket's name:

   ```json
   {
       "Version": "2012-10-17",
       "Statement": [
           { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-vapor-bucket" },
           { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-vapor-bucket/*" }
       ]
   }
   ```

3. Under **Security credentials**, create an access key for "Application running outside AWS". That is `SOURCE_AWS_ACCESS_KEY_ID` and `SOURCE_AWS_SECRET_ACCESS_KEY`.

The region of the bucket is in the S3 console, or `aws s3api get-bucket-location --bucket my-vapor-bucket` says it (an empty answer is `us-east-1`). Left out, the command asks AWS.

## Other S3 services

Anything that speaks S3 can be the source: give its endpoint, and the region it expects.

| Service | `--source-endpoint` | `--source-region` |
| --- | --- | --- |
| DigitalOcean Spaces | `https://ams3.digitaloceanspaces.com` | `ams3` |
| Cloudflare R2 | `https://<account id>.r2.cloudflarestorage.com` | `auto` |
| Hetzner Object Storage | `https://fsn1.your-objectstorage.com` | `fsn1` |
| Another Scaleway project | `https://s3.nl-ams.scw.cloud` | `nl-ams` |
| MinIO and the like | your address, with `--source-path-style` | what it is set up with |

## In your project, or from your machine

**By default the copy runs in your own Scaleway project.** The command starts it and Brewless runs a serverless job there with rclone: the files go from your old provider straight to Object Storage, never through Brewless and not through your machine. You can close your laptop while it runs. Both keys, the one for the source and the environment's own key for its bucket, reach the job in one secret of your Secret Manager (`<application>-<environment>-import`, under the path `/brewless`); when the copy ends, however it ends, that version of the secret is deleted. Brewless itself keeps no key: the operation only notes which version of the secret it was. A copy may run for up to a day; one that needs longer stops, and started again it carries on with what is left. Only one copy into an environment's bucket runs at a time.

**With `--local` the copy runs on your machine.** The command reads every file from the source and puts it into the environment's bucket itself, with the environment's key for its bucket fetched through Brewless as `env:pull` does. The source key never leaves your machine, but every file passes through it, so it takes as long as your connection does and the terminal has to stay open. A file larger than 5 GB is skipped and named at the end: copy those in your project. Brewless only notes in the audit trail that a copy started and how it ended, with its totals.

| | In your project (default) | `--local` |
| --- | --- | --- |
| Where the files pass | from provider to provider | through your machine |
| Where the source key goes | into your own Secret Manager, deleted afterwards | nowhere: it stays on your machine |
| Terminal open | no | yes, until it is done |
| Files larger than 5 GB | copied | skipped |
| A file that is there already | left alone (same size and time) | left alone (same name and size) |

## Following a copy

The command follows the copy as it goes: how many files of how many, and how much of how much. Stop following with `Ctrl+C`: a copy in your project goes on, and

```bash
brewless storage:status production
```

shows the last copy into that environment's bucket, while it runs and after it ended. The copy is also on the environment's page in the console, among its operations, where it can be cancelled. Cancelling stops the job and deletes the keys; what was copied stays.

`--dry-run` copies nothing: in your project it counts the files in the source and their size; with `--local` it also says how many of them the environment's bucket has already.

## After the move

- **Leave the old `AWS_*` lines out of the variables you push.** `brewless env:push` replaces the environment's variables as a whole. A `.env` from your old application with its own `AWS_BUCKET`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` would point the application back at the old bucket. Start from `brewless env:pull` instead. See [Environment variables](/running/variables).
- **`AWS_URL`** pointed at your old bucket or CloudFront. Change it, or remove it, and deploy.
- **Copy once more** just before you move your domain: only what changed since is copied.
- **Remove the key** you made at AWS for the move.
- **Your database** moves with `brewless db:import`; see [Moving your database](/applications/moving-a-database).

> [!WARNING]
> AWS bills the data that leaves it. Copying a bucket costs what AWS charges per GB of outgoing traffic; `--dry-run` says how much there is first.

What is not copied: who may read a file at AWS (an object that was public there is private in the environment's bucket, as everything in it is) and older versions of a file. rclone copies a file's type and its other metadata along with it.