Tag Archives: AWS

Create a pre-signed upload url for AWS S3 using Golang

Hi everyone,

This is just a quick post on how to create a pre-signed upload url for AWS S3 using Golang.

The generate the presigned url, you’ll need something like the following:

package main

import (
	"fmt"
	"github.com/joho/godotenv"
	log "github.com/sirupsen/logrus"
	"os"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"
)

func main() {

	// Load env vars
	err := godotenv.Load(".env")
	if err != nil {
		log.Fatalf("Error loading .env file: %v", err)
	}

	// Load the bucket name
	s3Bucket := os.Getenv("S3_BUCKET")
	if s3Bucket == "" {
		log.Fatal("an s3 bucket was unable to be loaded from env vars")
	}

	// Prepare the S3 request so a signature can be generated
	svc := s3.New(session.New())
	r, _ := svc.PutObjectRequest(&s3.PutObjectInput{
		Bucket: aws.String(s3Bucket),
		Key:    aws.String("test-file.jpg"),
	})

	// Create the pre-signed url with an expiry
	url, err := r.Presign(15 * time.Minute)
	if err != nil {
		fmt.Println("Failed to generate a pre-signed url: ", err)
		return
	}

	// Display the pre-signed url
	fmt.Println("Pre-signed URL", url)
}

Note that we’re using godotenv to load AWS environment variables containing a few AWS keys. You can get godotenv by running the following:

go get github.com/joho/godotenv

I then have a file called “.env” sitting in the root of my working directory:

S3_BUCKET=<YOUR_BUCKET_NAME>
AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY_ID>
AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_ACCESS_KEY>
AWS_REGION=<YOUR_AWS_REGION>

Once you’ve got all of that setup you can run the script, it should output a link in your console window similar to the following:

Pre-signed URL https://YOUR_BUCKET.s3.YOUR_REGION.amazonaws.com/test-file.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=CREDS&X-Amz-Date=20210717T073809Z&X-Amz-Expires=900&X-Amz-SignedHeaders=host&X-Amz-Signature=GENERATED_SIG

To test the url you can use something like Postman with the following configuration:

Simply pasting the url into the path should populate the headers for you. As for the body, select “binary” and browse for an image. When you’re ready, click “Send”.

You should get a 200 OK response and now be able to see your uploaded image in your destination bucket. Unless you’ve changed the key it should be under the name “test-file.jpg”.

One of the main advantages of using a pre-signed url is that it allows you to upload images directly to AWS and bypass your backend server completely. You can also use it to sign image retrievals. This allows you to give the links a limited life-span – great for preventing hot-linking.

Thanks to the following GitHub post for pointing me in the right direction: https://github.com/aws/aws-sdk-go/issues/467#issuecomment-171468806

ElasticSearch on AWS – Anonymous is not authorized to perform es:ESHttpGet

Hi everyone,

I am trying out ElasticSearch on AWS and ran into the following error while trying to access the provided Kabana endpoint:

{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}

This turned out to be pretty simply, I just needed to whitelist my IP. Go to your search domain in the aws console, click access and finally select “Modify access policy”:

Then all you need to do is add another statement that gives your current ip access:

{ 
"Version": "2012-10-17", 
"Statement": [
{
 "Effect": "Allow",
"Principal": {
 "AWS": "*"
},
"Action": [
"es:ESHttp*"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"192.0.2.0/24"
]
}
},
"Resource": "arn:aws:es:us-west-1:987654321098:domain/test-domain/*"
}
]
}

Feel free to comment below if you hit any issues, but I also found the following links pretty helpful:

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html#es-ac-types-ip

https://aws.amazon.com/premiumsupport/knowledge-center/anonymous-not-authorized-elasticsearch/

Cognito Hosted UI User Pool – Google not showing

Hi everyone,

I’m implementing Cognito User Pools for an app and currently adding social providers (Google, Facebook, etc).

The setup process seems pretty straight forward, however the social options did not appear on my hosted ui.

It turned out that I’d missed the last step in the documentation:
– Go to “App Client Settings” (left hand menu under App integration)
– Look for “Enabled Identity Providers” and check any that you want to show

I found this a little unintuitive as I’d expected it to show once it was enabled in the “Identity Providers” section. I probably just need to learn to read ALL of the docs!

hosted-ui.JPG

The full documentation is available here: https://docs.aws.amazon.com/cognito/latest/developerguide/cognito-user-pools-social-idp.html

And the bulk of the configuration is performed here:
google-congito

Access to fetch from origin has been blocked by CORS policy – AWS SAM Local

Hi everyone,

I’ve been using AWS SAM local lately and ran into a bit of an issue with CORS. It took a looong time to find a solution that worked for all of my local scenarios so hopefully this will be able to help someone else out.

Access to fetch at 'http://127.0.0.1:3000' from origin 'http://localhost:3001' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

The error above is fairly typical when working with APIs. The request will work from tools such as postman and fiddler but browsers will block it. These are a few good links that explain why CORS is necessary:
https://medium.com/@electra_chong/what-is-cors-what-is-it-used-for-308cafa4df1a
https://stackoverflow.com/a/29167709/522859

As for the solution, add the following to your template.yml:

  PreFlightFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: functions/generic/cors.getPreflightHandler
      Runtime: nodejs8.10
      Events:
        GetEvent:
          Type: Api
          Properties:
            Path: /pages
            Method: options
            RestApiId: !Ref XXXApi
            Auth:
              Authorizer: NONE

If you haven’t already defined your api in your template.yml file there is a default/omitted one created for you. There are a few examples on the AWS github: https://github.com/awslabs/serverless-application-model/blob/release/v1.8.0/examples/2016-10-31/api_cognito_auth/template.yaml

The next thing to do is to create a handler for the options request:


/* Handles retrieving a specific page */
exports.getPreflightHandler = async (event, context, callback) => {
    callback(null, { body: {} }, headers: { 'content-type': 'application/json', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Headers': 'Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token', 'Access-Control-Allow-Methods': 'OPTIONS,GET,POST,PUT,PATCH,DELETE', 'Access-Control-Allow-Credentials': true }, statusCode: 200 })
};

I’ve moved all mine out to some helper methods but the above should be enough to get it working for you. Hopefully the AWS team will have a simpler solution soon but if you run into any issues in the meantime please let me know!

AWS SAM Request Extremely Slow – Fix

Hi everyone,

I’m currently using AWS SAM CLI with NodeJS and was surprised to find that the requests were significantly slower when run locally. Luckily, I came across a post that suggested adding –skip-pull-image to your start-api command:

sam local start-api --skip-pull-image

This brought my requests down to under a second. Thanks to the following link for the info: https://github.com/awslabs/aws-sam-cli/issues/134#issuecomment-406786704

Overriding Global Variables – AWS SAM Local

Hi everyone,

Today I’ve added local overrides to the global variables in my template.yml file. This was pretty finicky – a lot of conflicting suggestions out there unfortunately. These are the settings that ended up working for me.

local-env-var-overrides.json:

{
    "Parameters": {
        "RDS_HOSTNAME_TEST": "192.168.0.31"
    }
}

template.yaml

Globals:
  Function:
    Environment:
      Variables:        
        RDS_HOSTNAME_TEST:
          Type: String
          Default: 'DEFAULT'

aws sam local command:

sam local start-api --env-vars "local-env-var-overrides.json"

create.js

exports.createTrapHandler = async (event, context, callback) => {

	try {

        console.log("RDS_HOSTNAME_TEST: " + process.env.RDS_HOSTNAME_TEST)
        // Outputs: 192.168.0.31

This link ended up being pretty useful: https://www.npmjs.com/package/aws-sam-local

Get User Id in Lambda node.js

Hi everyone,

A quick post on where to find the user id (sub) in a lambda requested that has been authenticated with a congito authorizer.

You’ll be able to find everything you need in the event object under requestContext > authorizer > claims:


exports.viewContextHandler = async (event, context, callback) => {
    console.log(JSON.stringify(event.requestContext));
}

"requestContext": {
    "resourceId": "XXXXX",
    "authorizer": {
        "claims": {
            "at_hash": "XXXXX",
            "sub": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
            "aud": "XXXXX12341234512345XXXXX",
            "email_verified": "true",
            "token_use": "id",
            "auth_time": "1547371205",
            "iss": "https://cognito-XXXXX.com/XXXXX",
            "cognito:username": "XXXXX-XXXXX-XXXXX-XXXXX-XXXXX",
            "exp": "Sun Jan 13 10:20:05 UTC 2019",
            "iat": "Sun Jan 13 09:20:05 UTC 2019",
            "email": "XXXXX@XXXXX.XXXXX"
        }
    },
}

Cognito Auth with AWS SAM

Hi everyone,

I’ve spent today implementing Cognito with AWS SAM and it took quite a while to work out what needed to be done – unfortunately there’s a lot of conflicting doco out there. Posting a sample template just in case it’s able to help anyone else out.

The first thing to do is to explicitly define you rest API. By default AWS SAM will generate one with a default logical id of ServerlessRestApi. You’ll need to override this:

Resources:
  # See links for more info
  # Referencing cognito authorizer: https://github.com/awslabs/serverless-application-model/issues/512#issuecomment-411284092
  # Logical id is auto generated: https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  # Sample template.yml: https://github.com/awslabs/serverless-application-model/blob/release/v1.8.0/examples/2016-10-31/api_cognito_auth/template.yaml
  MyCustomApi:
    Type: "AWS::Serverless::Api"
    Properties:
      StageName: Prod
      Auth: # We will eventually define other auth options here such as Usage Plans/Api Keys, AWS_IAM, and Resource Policies
        DefaultAuthorizer: MyCustomCognitoAuthorizer
        Authorizers:
          MyCustomCognitoAuthorizer:
            UserPoolArn: !GetAtt MyCustomCognitoUserPool.Arn # Can be a string, or array
            # Identity: # Optional
              # Header: ... # Optional; Default: Authorization
              # ValidationExpression: ...  # Optional; ensures the request header matches a pattern before checking in with the Authorizer endpoint; is there a default we can set for Cognito User Pools Auth?

You’ll also need to create a user pool and client:

# Creating a cognito user pool - https://github.com/awslabs/serverless-application-model/blob/master/examples/2016-10-31/api_cognito_auth/template.yaml
  MyCustomCognitoUserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      UserPoolName: !Ref CognitoUserPoolName
      # LambdaConfig:
        # PreSignUp: !GetAtt PreSignupLambdaFunction.Arn
      Policies:
        PasswordPolicy:
          MinimumLength: 8
      UsernameAttributes:
        - email
      Schema:
        - AttributeDataType: String
          Name: email
          Required: false

  MyCustomCognitoUserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref MyCustomCognitoUserPool
      ClientName: !Ref CognitoUserPoolClientName
      GenerateSecret: false

You then add the api id and the auth attribute to each of your function properties. If you’ve used the default authorizer property when defining the associated api you can override it by using authorizer: none.

GetBreedFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: breed.getBreedHandler
      Policies: arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess
      Runtime: nodejs8.10
      Environment:
        Variables:
          TABLE_CAT_BREED: !Ref CatBreedTable
      Events:
        GetEvent:
          Type: Api
          Properties:
            Path: /breed/{breedId}
            Method: get
            RestApiId: !Ref MyCustomCustomApi
            Auth:
              Authorizer: NONE

Hopefully that’s able to help you out. If you run into any trouble I found the following link pretty useful: https://github.com/awslabs/serverless-application-model/blob/release/v1.8.0/examples/2016-10-31/api_cognito_auth/template.yaml

AWS SAM not re-creating DynamoDb Table that was manually deleted – CodeStar

Hi everyone,

I ran into a bit of an issue after deleting a DynamoDb table via the AWS Console that had been created via CloudFormation (using AWS SAM). After deleting it I had expected it to be re-created automatically on the next deploy. Unfortunately this didn’t happen.

I came across the following AWS article that does a pretty good job of summarising the issue: https://aws.amazon.com/premiumsupport/knowledge-center/failing-stack-updates-deleted/

To fix it, I removed all references to the table from my template.yml file (this includes the table definition and any !Ref tags). After pushing this changeset I returned all of the references and re-pushed.

Let me know if you have any issues!

Cheers,
Chris

mocha tests/* sh: 1: mocha: Permission denied – AWS CodeBuild with Node.js

Hi everyone,

I ran into the following error while running a Node.js build with AWS CodeBuild:

mocha tests/* sh: 1: mocha: Permission denied

To resolve this I removed node_modules from my repository and added it to .gitignore:

node_modules/

Thanks to the following links for the info:
Add node_modules to gitignore: https://stackoverflow.com/a/29820869/522859
Misc background issues: https://github.com/mochajs/mocha/issues/1487