Tag Archives: Amazon S3

Upload an Image to S3 Using Post and a Presigned Url

Hi everyone,

Today I’ve been converting my “PUT” upload to S3 to a “POST”. The main motivator for this was to restrict the file size of uploads using a signed policy. Unfortunately this was a pretty tedious process and the error responses from S3 were very vague. Thankfully it’s working now and here’s what I needed to do!

Here’s an excerpt from my nodejs lambda function that generates the presigned url:

/* Creates a pre-signed upload url and then stores a reference in a table */
exports.createUploadUrl = async params => {

    var { databaseHandler, bucketHandler } = params;

    // Create id for upload
    var uploadId = uuidv4();

    // Retrieve pre-signed url
    var bucketDataPromise = createPresignedPostPromise({
        Bucket: process.env.BUCKET_UPLOADS,
        Expires: 60 * 60,
        Conditions: [            
            ["content-length-range", 0, 300000], // 300kb
            [ "eq", "$key", uploadId],
            [ "eq", "$Content-Type", 'image/jpeg' ],
        ]
    });

    // var ddbData = await ddbDataPromise;
    var bucketData = await bucketDataPromise;

    // Wait for database handler to complete operation and then return
    return Helpers.generateResponse({
        data: {
            uploadData: bucketData,
            additionalFields: {
                key: uploadId,
                "Content-Type": 'image/jpeg',
            },
        },
        statusCode: 200
    });
}

You can then retrieve the upload details using a request like the following (Python):

resp = requests.put("https://f86caxqe9f.execute-api.ap-southeast-2.amazonaws.com/Prod/images", data=open(path, 'rb'))

This will return a response similar to the following:

{
    "messages": [],
    "data": {
        "uploadUrl": {
            "url": "YOUR UPLOAD URL",
            "fields": {
                "bucket": "YOUR BUCKET",
                "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
                "X-Amz-Credential": "XXX",
                "X-Amz-Date": "20190107T125044Z",
                "X-Amz-Security-Token": "SECURITY_TOKEN",
                "Policy": "YOUR BASE64 ENCODED POLICY",
                "X-Amz-Signature": "SIGNATURE"
            }
        },
        "uploadId": "UPLOAD_ID"
    }
}

And once you have those details you can use them to upload the image:

# Attempt to retrieve upload url etc
json = resp.json()
data = json["data"]
uploadUrl = data["uploadData"]["url"]
uploadFields = data["uploadData"]["fields"]
uploadFields.update(data["additionalFields"])

try:
    print("Uploading image...",end='')
    headers = {'Content-Type': 'multipart/form-data' }
    files = { 'file': open(path, 'rb')}
    resp = requests.post(uploadUrl, data=uploadFields, files=files)

    # Only show content if there's an error
    if resp.status_code == 200:
        print("Uploaded")
    else:  
        print("ERROR")                          
        print(repr(resp))
        print(repr(resp.content))

except Exception as e:
    print("\r\nFailed to upload image.")
    print("Upload data:")
    print(repr(uploadFields))
    print("Exception:")
    print(repr(e))
    traceback.print_exc()

Hopefully that’s been able to help you out, but feel free to let me know in the comments below if you need more info!

Thanks to these links for the info:
https://stackoverflow.com/a/35923104/522859
https://stackoverflow.com/a/49311831/522859

Advertisements

AWS CLI Copy from Bucket – An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

Hi everyone,

I ran into the following error while trying to copy files from one bucket to another using the AWS CLI:

An error occurred (AccessDenied) when calling the CopyObject operation: Access Denied

Thankfully this one is pretty self-explanatory. My user was missing the permissions required to view the bucket.

To see your user: aws configure list
To add permissions: view the IAM Management Console

Thanks,
Chris

AWS CLI Copy to Bucket – Could not connect to the endpoint URL

Hi everyone,

I ran into the following error while attempting to copy files to a bucket via the AWS CLI:

The solution turned out to be pretty straight-forward. The region on the cli-config was set incorrectly. For example, mine was set to “Sydney” instead of “ap-southeast-2”. Type “aws configure list” to check what yours is set to.

Thanks to the following stackoverflow post for the info: https://stackoverflow.com/a/40411174/522859

Warning: S3::putObject(): RequestTimeTooSkewed – CarrierWave (Ruby on Rails)

Hey everyone,

Another issue I came across while working with CarrierWave. After rebooting a VM Amazon started too complain “request time too skewed”:

Warning: S3::putObject(): RequestTimeTooSkewed The difference between the request time and the current time is to large. in x on line 14

This one is another simple fix, just make sure your system time is correct. I’d been using a VM and it had run forward by a few hours. Setting it back manually resolved the error.

Amazon S3 Images Expiring – CarrierWave (Ruby on Rails)

Hey everyone,

After working with CarrierWave for a couple of days a few of my images started disappearing. It took a while to work out what was causing it, but thankfully the fix is pretty easy:


#Set fog_public in your initialiser to true
config.fog_public = false

#Should be
config.fog_public = true

Alternatively, if you’d like them to keep expiring but want to adjust their availability you can use the following config option:

#Number of seconds
config.fog_authenticated_url_expiration = 60000 

There’s a bit of info in the following StackOverflow post: http://stackoverflow.com/a/13757841/522859

Excon::Errors::SocketError Broken pipe (Errno::EPIPE) – Ruby on Rails

Hey everyone,

I was having a bit of trouble with CarrierWave on Amazon S3 today. When attempting to upload files that were larger than ~150kb I received one of the following errors (depending on config):

getaddrinfo: Name or service not known carrierwave
Excon::Errors::SocketError in PhotosController#create
Broken pipe (Errno::EPIPE)

 

Despite the vague error, the solution was fairly simple. The region configured in my initialiser was different to the one my bucket was created in.

In order to find out which region you need, logon to your AWS console and browse to an uploaded image. Check the endpoint URL (properties > static website hosting) and simply copy the region. For examples:

Endpoint: testbucket123321.s3-website-us-west-2.amazonaws.com
Region: us-west-2

A couple of final tips if this doesn’t work for you:

  • You need the region codes, not the name. For instance, “Oregon” won’t work
  • Don’t forget to restart your app after making changes to the initialiser
  •