Category Archives: Web Development

Breaking the Internet

this.query is not a function – error when using promisify with mysql

Hi everyone,

I ran into the following error while attempting to promisify my node.js mysql transactions:

TypeError: this.query is not a function
    at rollback (/var/task/node_modules/mysql/lib/Connection.js:179:15)
    at rollback (internal/util.js:230:26)

This took a while to track down but it turns out that I needed to bind the connection after promisifying the function. Instead of:

util.promisify(connection.query);

Add bind to the end of it:

util.promisify(connection.query).bind(connection)

Thanks to the following stackoverflow post for the insight: https://stackoverflow.com/a/51690276/522859

Advertisements

Overriding Global Variables – AWS SAM Local

Hi everyone,

Today I’ve added local overrides to the global variables in my template.yml file. This was pretty finicky – a lot of conflicting suggestions out there unfortunately. These are the settings that ended up working for me.

local-env-var-overrides.json:

{
    "Parameters": {
        "RDS_HOSTNAME_TEST": "192.168.0.31"
    }
}

template.yaml

Globals:
  Function:
    Environment:
      Variables:        
        RDS_HOSTNAME_TEST:
          Type: String
          Default: 'DEFAULT'

aws sam local command:

sam local start-api --env-vars "local-env-var-overrides.json"

create.js

exports.createTrapHandler = async (event, context, callback) => {

	try {

        console.log("RDS_HOSTNAME_TEST: " + process.env.RDS_HOSTNAME_TEST)
        // Outputs: 192.168.0.31

This link ended up being pretty useful: https://www.npmjs.com/package/aws-sam-local

Upload an Image to S3 Using Post and a Presigned Url

Hi everyone,

Today I’ve been converting my “PUT” upload to S3 to a “POST”. The main motivator for this was to restrict the file size of uploads using a signed policy. Unfortunately this was a pretty tedious process and the error responses from S3 were very vague. Thankfully it’s working now and here’s what I needed to do!

Here’s an excerpt from my nodejs lambda function that generates the presigned url:

/* Creates a pre-signed upload url and then stores a reference in a table */
exports.createUploadUrl = async params => {

    var { databaseHandler, bucketHandler } = params;

    // Create id for upload
    var uploadId = uuidv4();

    // Retrieve pre-signed url
    var bucketDataPromise = createPresignedPostPromise({
        Bucket: process.env.BUCKET_UPLOADS,
        Expires: 60 * 60,
        Conditions: [            
            ["content-length-range", 0, 300000], // 300kb
            [ "eq", "$key", uploadId],
            [ "eq", "$Content-Type", 'image/jpeg' ],
        ]
    });

    // var ddbData = await ddbDataPromise;
    var bucketData = await bucketDataPromise;

    // Wait for database handler to complete operation and then return
    return Helpers.generateResponse({
        data: {
            uploadData: bucketData,
            additionalFields: {
                key: uploadId,
                "Content-Type": 'image/jpeg',
            },
        },
        statusCode: 200
    });
}

You can then retrieve the upload details using a request like the following (Python):

resp = requests.put("https://f86caxqe9f.execute-api.ap-southeast-2.amazonaws.com/Prod/images", data=open(path, 'rb'))

This will return a response similar to the following:

{
    "messages": [],
    "data": {
        "uploadUrl": {
            "url": "YOUR UPLOAD URL",
            "fields": {
                "bucket": "YOUR BUCKET",
                "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
                "X-Amz-Credential": "XXX",
                "X-Amz-Date": "20190107T125044Z",
                "X-Amz-Security-Token": "SECURITY_TOKEN",
                "Policy": "YOUR BASE64 ENCODED POLICY",
                "X-Amz-Signature": "SIGNATURE"
            }
        },
        "uploadId": "UPLOAD_ID"
    }
}

And once you have those details you can use them to upload the image:

# Attempt to retrieve upload url etc
json = resp.json()
data = json["data"]
uploadUrl = data["uploadData"]["url"]
uploadFields = data["uploadData"]["fields"]
uploadFields.update(data["additionalFields"])

try:
    print("Uploading image...",end='')
    headers = {'Content-Type': 'multipart/form-data' }
    files = { 'file': open(path, 'rb')}
    resp = requests.post(uploadUrl, data=uploadFields, files=files)

    # Only show content if there's an error
    if resp.status_code == 200:
        print("Uploaded")
    else:  
        print("ERROR")                          
        print(repr(resp))
        print(repr(resp.content))

except Exception as e:
    print("\r\nFailed to upload image.")
    print("Upload data:")
    print(repr(uploadFields))
    print("Exception:")
    print(repr(e))
    traceback.print_exc()

Hopefully that’s been able to help you out, but feel free to let me know in the comments below if you need more info!

Thanks to these links for the info:
https://stackoverflow.com/a/35923104/522859
https://stackoverflow.com/a/49311831/522859

Cognito Auth with AWS SAM

Hi everyone,

I’ve spent today implementing Cognito with AWS SAM and it took quite a while to work out what needed to be done – unfortunately there’s a lot of conflicting doco out there. Posting a sample template just in case it’s able to help anyone else out.

The first thing to do is to explicitly define you rest API. By default AWS SAM will generate one with a default logical id of ServerlessRestApi. You’ll need to override this:

Resources:
  # See links for more info
  # Referencing cognito authorizer: https://github.com/awslabs/serverless-application-model/issues/512#issuecomment-411284092
  # Logical id is auto generated: https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
  # Sample template.yml: https://github.com/awslabs/serverless-application-model/blob/release/v1.8.0/examples/2016-10-31/api_cognito_auth/template.yaml
  MyCustomApi:
    Type: "AWS::Serverless::Api"
    Properties:
      StageName: Prod
      Auth: # We will eventually define other auth options here such as Usage Plans/Api Keys, AWS_IAM, and Resource Policies
        DefaultAuthorizer: MyCustomCognitoAuthorizer
        Authorizers:
          MyCustomCognitoAuthorizer:
            UserPoolArn: !GetAtt MyCustomCognitoUserPool.Arn # Can be a string, or array
            # Identity: # Optional
              # Header: ... # Optional; Default: Authorization
              # ValidationExpression: ...  # Optional; ensures the request header matches a pattern before checking in with the Authorizer endpoint; is there a default we can set for Cognito User Pools Auth?

You’ll also need to create a user pool and client:

# Creating a cognito user pool - https://github.com/awslabs/serverless-application-model/blob/master/examples/2016-10-31/api_cognito_auth/template.yaml
  MyCustomCognitoUserPool:
    Type: AWS::Cognito::UserPool
    Properties:
      UserPoolName: !Ref CognitoUserPoolName
      # LambdaConfig:
        # PreSignUp: !GetAtt PreSignupLambdaFunction.Arn
      Policies:
        PasswordPolicy:
          MinimumLength: 8
      UsernameAttributes:
        - email
      Schema:
        - AttributeDataType: String
          Name: email
          Required: false

  MyCustomCognitoUserPoolClient:
    Type: AWS::Cognito::UserPoolClient
    Properties:
      UserPoolId: !Ref MyCustomCognitoUserPool
      ClientName: !Ref CognitoUserPoolClientName
      GenerateSecret: false

You then add the api id and the auth attribute to each of your function properties. If you’ve used the default authorizer property when defining the associated api you can override it by using authorizer: none.

GetBreedFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: breed.getBreedHandler
      Policies: arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess
      Runtime: nodejs8.10
      Environment:
        Variables:
          TABLE_CAT_BREED: !Ref CatBreedTable
      Events:
        GetEvent:
          Type: Api
          Properties:
            Path: /breed/{breedId}
            Method: get
            RestApiId: !Ref MyCustomCustomApi
            Auth:
              Authorizer: NONE

Hopefully that’s able to help you out. If you run into any trouble I found the following link pretty useful: https://github.com/awslabs/serverless-application-model/blob/release/v1.8.0/examples/2016-10-31/api_cognito_auth/template.yaml

Error: Invalid value for “–parameter-overrides”: ParameterKey=TABLE_NAME,ParameterValue=CatBreeds is not in valid format. It must look something like ‘ParameterKey=KeyPairName,ParameterValue=MyKey ParameterKey=InstanceType,ParameterValue=t1.micro’ – AWS SAM CLI

Hi everyone,

I ran into the following error while using the AWS SAM Cli:

Error: Invalid value for "--parameter-overrides": ParameterKey=PURPLE_FROG,ParameterValue=CatBreeds is not in valid format. It must look something like 'ParameterKey=KeyPairName,ParameterValue=MyKey ParameterKey=InstanceType,ParameterValue=t1.micro'

This is the command I was running:

sam local start-api --parameter-overrides "ParameterKey=PURPLE_FROG,ParameterValue=CatBreeds"

The error message is unfortunately very misleading. After a fair bit of testing I found that the cause is actually the underscore in the parameter key and has nothing to do with the command format. Renaming all of the parameters to something like “PURPLEFROG” works.

Cheers,
Chris

4: /codebuild/output/tmp/script.sh: pip: not found – Node.js and CodeStar

Hi everyone,

I ran into the following CodeBuild error after upgrading my build environment from the default nodejs8.10 to nodejs10.14:

4: /codebuild/output/tmp/script.sh: pip: not found

This one was a little confusing, but thankfully fairly easy to fix. In your buildspec.yml file update the pip steps to reference pip3 instead of pip:

// Original
commands:
      # Install dependencies needed for running tests
      - npm install

      # Upgrade AWS CLI to the latest version
      - pip install --upgrade awscli
// Modified
commands:
      # Install dependencies needed for running tests
      - npm install

      # Upgrade AWS CLI to the latest version
      - pip3 install --upgrade awscli

mocha tests/* sh: 1: mocha: Permission denied – AWS CodeBuild with Node.js

Hi everyone,

I ran into the following error while running a Node.js build with AWS CodeBuild:

mocha tests/* sh: 1: mocha: Permission denied

To resolve this I removed node_modules from my repository and added it to .gitignore:

node_modules/

Thanks to the following links for the info:
Add node_modules to gitignore: https://stackoverflow.com/a/29820869/522859
Misc background issues: https://github.com/mochajs/mocha/issues/1487