Moving your files from S3

An application that comes from Laravel Vapor, or from anywhere else on AWS, keeps its uploads in an S3 bucket. brewless storage:import copies them into the bucket of an environment (see A bucket for files): you say where the files are now and give a key that may read them; where they go follows from the environment.

brewless storage:import production --from-env=../old-app/.env

What does storage:import do?#

It copies every file of the source bucket, or of one folder in it, into the environment's bucket. Nothing is removed, neither at the source nor in the environment's bucket, and nothing at the source is changed. A file that is in the environment's bucket already, with the same size and time, is left alone: running the import again copies only what is new or changed since, so you can copy everything now and once more just before you move your domain.

It needs an environment with a bucket for files and a key of its own for it, which Brewless makes when it sets the environment up (its Scaleway key needs IAMManager for that). Without them it says so and starts nothing.

Which value goes where#

The names are those of Laravel's s3 disk in config/filesystems.php. For the source you give them; the destination is what Brewless already set in the environment's variables when it made the bucket.

When a value is given in more than one way, the first of these wins: the option, then SOURCE_AWS_*, then --source-profile, then --from-env, and last the question in the terminal. The command says where it took the key from, never the key. AWS_* variables of your own shell are not used: they often belong to another AWS account. Run by a script without a terminal, a missing key stops the command.

A key is never an option on the command line, where it would end up in your shell's history.

From the .env of your old application#

brewless storage:import production --from-env=../old-app/.env

reads AWS_BUCKET, AWS_DEFAULT_REGION, AWS_ENDPOINT, AWS_USE_PATH_STYLE_ENDPOINT, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY from that file and nothing else. The file is read on your machine and its values are not printed.

Note

The .env of an application on Vapor usually has no AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY: on Vapor the application runs with an IAM role, and Vapor sets AWS_BUCKET itself. Make a key that may read the bucket (below) and give it as SOURCE_AWS_*. The name of the bucket is under storage in vapor.yml, which brewless init --from-vapor reads too.

With variables#

export SOURCE_AWS_ACCESS_KEY_ID=AKIA…
export SOURCE_AWS_SECRET_ACCESS_KEY=…
brewless storage:import production s3://my-vapor-bucket/uploads --source-region=eu-west-1

/uploads copies only that folder; without it the whole bucket is copied. A file keeps its name, folder included: uploads/a.jpg stays uploads/a.jpg.

With a profile of the AWS CLI#

brewless storage:import production s3://my-vapor-bucket --source-profile=migration

takes the key of that profile from ~/.aws/credentials and its region from ~/.aws/config.

A key that may only read, at AWS#

Do not use the key of your AWS account. Make one that may only read this bucket, and remove it once you have moved:

1. In the AWS console, under IAM, Users, create a user such as brewless-migration, without access to the console.

2. Give it an inline policy, with your bucket's name:

   {
       "Version": "2012-10-17",
       "Statement": [
           { "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::my-vapor-bucket" },
           { "Effect": "Allow", "Action": "s3:GetObject", "Resource": "arn:aws:s3:::my-vapor-bucket/*" }
       ]
   }
   

3. Under Security credentials, create an access key for "Application running outside AWS". That is SOURCE_AWS_ACCESS_KEY_ID and SOURCE_AWS_SECRET_ACCESS_KEY.

The region of the bucket is in the S3 console, or aws s3api get-bucket-location --bucket my-vapor-bucket says it (an empty answer is us-east-1). Left out, the command asks AWS.

Other S3 services#

Anything that speaks S3 can be the source: give its endpoint, and the region it expects.

In your project, or from your machine#

By default the copy runs in your own Scaleway project. The command starts it and Brewless runs a serverless job there with rclone: the files go from your old provider straight to Object Storage, never through Brewless and not through your machine. You can close your laptop while it runs. Both keys, the one for the source and the environment's own key for its bucket, reach the job in one secret of your Secret Manager (<application>-<environment>-import, under the path /brewless); when the copy ends, however it ends, that version of the secret is deleted. Brewless itself keeps no key: the operation only notes which version of the secret it was. A copy may run for up to a day; one that needs longer stops, and started again it carries on with what is left. Only one copy into an environment's bucket runs at a time.

With --local the copy runs on your machine. The command reads every file from the source and puts it into the environment's bucket itself, with the environment's key for its bucket fetched through Brewless as env:pull does. The source key never leaves your machine, but every file passes through it, so it takes as long as your connection does and the terminal has to stay open. A file larger than 5 GB is skipped and named at the end: copy those in your project. Brewless only notes in the audit trail that a copy started and how it ended, with its totals.

Following a copy#

The command follows the copy as it goes: how many files of how many, and how much of how much. Stop following with Ctrl+C: a copy in your project goes on, and

brewless storage:status production

shows the last copy into that environment's bucket, while it runs and after it ended. The copy is also on the environment's page in the console, among its operations, where it can be cancelled. Cancelling stops the job and deletes the keys; what was copied stays.

--dry-run copies nothing: in your project it counts the files in the source and their size; with --local it also says how many of them the environment's bucket has already.

After the move#

• Leave the old AWS_* lines out of the variables you push. brewless env:push replaces the environment's variables as a whole. A .env from your old application with its own AWS_BUCKET, AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY would point the application back at the old bucket. Start from brewless env:pull instead. See Environment variables.
• AWS_URL pointed at your old bucket or CloudFront. Change it, or remove it, and deploy.
• Copy once more just before you move your domain: only what changed since is copied.
• Remove the key you made at AWS for the move.

Warning

AWS bills the data that leaves it. Copying a bucket costs what AWS charges per GB of outgoing traffic; --dry-run says how much there is first.

What is not copied: who may read a file at AWS (an object that was public there is private in the environment's bucket, as everything in it is) and older versions of a file. rclone copies a file's type and its other metadata along with it.