Tag Archives: upload

Create a pre-signed upload url for AWS S3 using Golang

Leave a reply

Hi everyone,

This is just a quick post on how to create a pre-signed upload url for AWS S3 using Golang.

The generate the presigned url, you’ll need something like the following:

package main

import (
	"fmt"
	"github.com/joho/godotenv"
	log "github.com/sirupsen/logrus"
	"os"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {

	// Load env vars
	err := godotenv.Load(".env")
	if err != nil {
		log.Fatalf("Error loading .env file: %v", err)
	}

	// Load the bucket name
	s3Bucket := os.Getenv("S3_BUCKET")
	if s3Bucket == "" {
		log.Fatal("an s3 bucket was unable to be loaded from env vars")
	}

	// Prepare the S3 request so a signature can be generated
	svc := s3.New(session.New())
	r, _ := svc.PutObjectRequest(&s3.PutObjectInput{
		Bucket: aws.String(s3Bucket),
		Key:    aws.String("test-file.jpg"),
	})

	// Create the pre-signed url with an expiry
	url, err := r.Presign(15 * time.Minute)
	if err != nil {
		fmt.Println("Failed to generate a pre-signed url: ", err)
		return
	}

	// Display the pre-signed url
	fmt.Println("Pre-signed URL", url)
}

Note that we’re using godotenv to load AWS environment variables containing a few AWS keys. You can get godotenv by running the following:

go get github.com/joho/godotenv

I then have a file called “.env” sitting in the root of my working directory:

S3_BUCKET=<YOUR_BUCKET_NAME>
AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY_ID>
AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_ACCESS_KEY>
AWS_REGION=<YOUR_AWS_REGION>

Once you’ve got all of that setup you can run the script, it should output a link in your console window similar to the following:

Pre-signed URL https://YOUR_BUCKET.s3.YOUR_REGION.amazonaws.com/test-file.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=CREDS&X-Amz-Date=20210717T073809Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=GENERATED_SIG

To test the url you can use something like Postman with the following configuration:

Simply pasting the url into the path should populate the headers for you. As for the body, select “binary” and browse for an image. When you’re ready, click “Send”.

You should get a 200 OK response and now be able to see your uploaded image in your destination bucket. Unless you’ve changed the key it should be under the name “test-file.jpg”.

One of the main advantages of using a pre-signed url is that it allows you to upload images directly to AWS and bypass your backend server completely. You can also use it to sign image retrievals. This allows you to give the links a limited life-span – great for preventing hot-linking.

Thanks to the following GitHub post for pointing me in the right direction: https://github.com/aws/aws-sdk-go/issues/467#issuecomment-171468806

Dragging a File from File Explorer Causes Google Chrome to Crash

Leave a reply

Hey everyone,

A team member found an interesting bug today that caused chrome to go non-responsive. The cause was simply dragging a file from the file explorer onto a drag and drop upload area. It is reproducible on every site I’ve tested, including Gmail and Dropbox.

Reproduce

  • Go to https://mail.google.com
  • Click compose
  • Click attach files (close any of the annoying popups that appear)
  • DRAG a number of files from the file selector window to the new message screen (they will start to upload) (IMPORTANT: these need to be dragged from the file selector popup, NOT a new explorer window)
  • While they are still uploading hit cancel on the file browser window. Browser will go non-responsive

A bug has been reported here: https://code.google.com/p/chromium/issues/detail?id=388661

Amazon S3 Images Expiring – CarrierWave (Ruby on Rails)

Leave a reply

Hey everyone,

After working with CarrierWave for a couple of days a few of my images started disappearing. It took a while to work out what was causing it, but thankfully the fix is pretty easy:


#Set fog_public in your initialiser to true
config.fog_public = false

#Should be
config.fog_public = true

Alternatively, if you’d like them to keep expiring but want to adjust their availability you can use the following config option:

#Number of seconds
config.fog_authenticated_url_expiration = 60000

There’s a bit of info in the following StackOverflow post: http://stackoverflow.com/a/13757841/522859

Excon::Errors::SocketError Broken pipe (Errno::EPIPE) – Ruby on Rails

Leave a reply

Hey everyone,

I was having a bit of trouble with CarrierWave on Amazon S3 today. When attempting to upload files that were larger than ~150kb I received one of the following errors (depending on config):

getaddrinfo: Name or service not known carrierwave
Excon::Errors::SocketError in PhotosController#create
Broken pipe (Errno::EPIPE)

 

Despite the vague error, the solution was fairly simple. The region configured in my initialiser was different to the one my bucket was created in.

In order to find out which region you need, logon to your AWS console and browse to an uploaded image. Check the endpoint URL (properties > static website hosting) and simply copy the region. For examples:

Endpoint: testbucket123321.s3-website-us-west-2.amazonaws.com
Region: us-west-2

A couple of final tips if this doesn’t work for you:

  • You need the region codes, not the name. For instance, “Oregon” won’t work
  • Don’t forget to restart your app after making changes to the initialiser

    •  

    Not Recognised by the ‘identify’ command – Paperclip

    5 Replies

    Just a quick post on another problem I ran into while setting up PaperClip. When attempting to submit a form WITHOUT an image I received the following error message:

     
Assets asset /tmp/stream20120218-22611-dgnkr-0.exe is not recognized by the 'identify' command.

    Apparently the most common cause of this issue is one of two things:

    #1: You don’t have RMagick installed
    The easiest fix for this is to simply install the rmagick gem. Simply add rmagick to your Gemfile and run bundle install. Restart your server and you should be ready to go.

    #2: Your command_path isn’t set correctly
    Run the following command from your console:

    chris@chris-VirtualBox:~/site$ which identify
/usr/bin/identify

    Then add the path that is returned to your environment settings:

    Paperclip.options[:command_path] = "/usr/bin"

    #3: If all else fails

    If none of the above seem to be causing your issue, try adding the following to your model:

     def asset?
    !(asset_content_type =~ /^image.*/).nil?
  end

    This will simply prevent the file being processed if it’s not an image, avoiding the issue all together. Hopefully this’ll help someone out, let me know if you come across any other solutions.

    UPDATE:
    Apparently a more recent cause of this issue is an update to the cocaine gem, rolling back to version 0.3.2 appears to resolve the issue. Thanks to Ahmed and Fabrice for bringing it to my attention:
    https://github.com/thoughtbot/paperclip/issues/1038
    http://stackoverflow.com/a/12760881/522859

    PaperClip Issues – Ruby on Rails

    1 Reply

    I finally decided to replace all my existing code to handle images with Paperclip. I was following the screencast by Emerson Lackey, #134 Paperclip, however I ran into a couple of issues. Thankfully they were all very easily fixed, and probably wouldn’t have occurred at all if I’d simply watched the whole screencast instead of trying to rush on ahead.

    Issue #1:
    The first issue I encountered was that my asset fields weren’t appearing on the page:

    I spent way too much time trying to work this one out, especially considering how obvious the solution is – the screencast is simply missing an “=” after the initial “<%" in "<% f.fields_for". Simply amend it as follows:

    Issue #2:
    The second one is actually encountered and addressed by Emerson himself. Unfortunately I started trying to find a solution before seeing his, so just in case someone else does a Google search I’ll include the issue here:

    undefined method `symbolize_keys!' for "/system/assets/1/original/mack_truck.jpg?1329550205":String

    The problem here is simply that the parenthesis are in the wrong place, simply amend your code as follows:

    Anyway, hopefully this helps someone else out – let me know if there are any issues.

    Update: Don’t name your model assets!
    Unfortunately I followed the tutorials naming example and called my model Asset. While this may have been fine in earlier version of rails it causes quirky conflicts with the asset pipeline. I strongly encourage you to use a different name i.e. Uploads