Tag: python3

Upload an Image to S3 Using Post and a Presigned Url

Hi everyone,

Today I’ve been converting my “PUT” upload to S3 to a “POST”. The main motivator for this was to restrict the file size of uploads using a signed policy. Unfortunately this was a pretty tedious process and the error responses from S3 were very vague. Thankfully it’s working now and here’s what I needed to do!

Here’s an excerpt from my nodejs lambda function that generates the presigned url:

/* Creates a pre-signed upload url and then stores a reference in a table */
exports.createUploadUrl = async params => {

    var { databaseHandler, bucketHandler } = params;

    // Create id for upload
    var uploadId = uuidv4();

    // Retrieve pre-signed url
    var bucketDataPromise = createPresignedPostPromise({
        Bucket: process.env.BUCKET_UPLOADS,
        Expires: 60 * 60,
        Conditions: [            
            ["content-length-range", 0, 300000], // 300kb
            [ "eq", "$key", uploadId],
            [ "eq", "$Content-Type", 'image/jpeg' ],
        ]
    });

    // var ddbData = await ddbDataPromise;
    var bucketData = await bucketDataPromise;

    // Wait for database handler to complete operation and then return
    return Helpers.generateResponse({
        data: {
            uploadData: bucketData,
            additionalFields: {
                key: uploadId,
                "Content-Type": 'image/jpeg',
            },
        },
        statusCode: 200
    });
}

You can then retrieve the upload details using a request like the following (Python):

resp = requests.put("https://f86caxqe9f.execute-api.ap-southeast-2.amazonaws.com/Prod/images", data=open(path, 'rb'))

This will return a response similar to the following:

{
    "messages": [],
    "data": {
        "uploadUrl": {
            "url": "YOUR UPLOAD URL",
            "fields": {
                "bucket": "YOUR BUCKET",
                "X-Amz-Algorithm": "AWS4-HMAC-SHA256",
                "X-Amz-Credential": "XXX",
                "X-Amz-Date": "20190107T125044Z",
                "X-Amz-Security-Token": "SECURITY_TOKEN",
                "Policy": "YOUR BASE64 ENCODED POLICY",
                "X-Amz-Signature": "SIGNATURE"
            }
        },
        "uploadId": "UPLOAD_ID"
    }
}

And once you have those details you can use them to upload the image:

# Attempt to retrieve upload url etc
json = resp.json()
data = json["data"]
uploadUrl = data["uploadData"]["url"]
uploadFields = data["uploadData"]["fields"]
uploadFields.update(data["additionalFields"])

try:
    print("Uploading image...",end='')
    headers = {'Content-Type': 'multipart/form-data' }
    files = { 'file': open(path, 'rb')}
    resp = requests.post(uploadUrl, data=uploadFields, files=files)

    # Only show content if there's an error
    if resp.status_code == 200:
        print("Uploaded")
    else:  
        print("ERROR")                          
        print(repr(resp))
        print(repr(resp.content))

except Exception as e:
    print("\r\nFailed to upload image.")
    print("Upload data:")
    print(repr(uploadFields))
    print("Exception:")
    print(repr(e))
    traceback.print_exc()

Hopefully that’s been able to help you out, but feel free to let me know in the comments below if you need more info!

Thanks to these links for the info:
https://stackoverflow.com/a/35923104/522859
https://stackoverflow.com/a/49311831/522859

AttributeError: module ‘serial’ has no attribute ‘SerialException’ – Python 3.5

Hi everyone,

A quick fix for an error I ran into while updating to Pyhton 3.5.2:

AttributeError: module 'serial' has no attribute 'SerialException'

I needed to add pyserial:

pi@raspberrypi:~/Desktop/piscripts/get-commands $ sudo pip3.5 install pyserial
Collecting pyserial
  Downloading pyserial-3.4-py2.py3-none-any.whl (193kB)
    100% |████████████████████████████████| 194kB 530kB/s
Installing collected packages: pyserial
Successfully installed pyserial-3.4

Thanks to the following link: https://stackoverflow.com/a/47562659/522859

Could not find a version that satisfied the requirement smbus/python3-smbus

Hi everyone,

I ran into a bit of an issue installing smbus on Python3.5. It looks like all you have to do is use smbus2:

pip3.5 install smbus2

Collecting smbus2
Downloading smbus2-0.2.0.tar.gz
Building wheels for collected packages: smbus2
Running setup.py bdist_wheel for smbus2 … done
Stored in directory: /root/.cache/pip/wheels/90/71/b4/9f90d8e2d0349ab55fef07169a81bd8f925965f16174e2f809
Successfully built smbus2
Installing collected packages: smbus2
Successfully installed smbus2-0.2.0

Thanks to the following link: https://www.raspberrypi.org/forums/viewtopic.php?t=192909

Pillow Install Fails – Python 3.5.2

Hi everyone,

Pillow install fails

I ran into a bit of an issue trying to install Pillow on Raspbian with Python 3.5.2. The full error is below, but the solution was simply to install the following:

sudo apt-get install libjpeg8-dev
sudo pip3.5 install pillow

sudo pip3.5 install pillow
Collecting pillow
Using cached Pillow-4.3.0.tar.gz
Requirement already satisfied: olefile in /usr/local/lib/python3.5/site-packages (from pillow)
Installing collected packages: pillow
Running setup.py install for pillow … error
Complete output from command /usr/local/bin/python3.5 -u -c “import setuptools, tokenize;__file__=’/tmp/pip-build-inj0l_9q/pillow/setup.py’;f=getattr(tokenize, ‘open’, open)(__file__);code=f.read().replace(‘rn’, ‘n’);f.close();exec(compile(code, __file__, ‘exec’))” install –record /tmp/pip-xp_x4w_5-record/install-record.txt –single-version-externally-managed –compile:
running install
running build
running build_py
creating build
creating build/lib.linux-armv7l-3.5
creating build/lib.linux-armv7l-3.5/PIL
copying PIL/MicImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageMath.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/WebPImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImagePath.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageGrab.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/SunImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/__init__.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageEnhance.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageMorph.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/XpmImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/EpsImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/FtexImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/WmfImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/FitsStubImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageDraw2.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/IptcImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/_binary.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/MpoImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/Hdf5StubImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/Image.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/MspImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageWin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/GimpPaletteFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageTk.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PSDraw.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/McIdasImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/SgiImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageStat.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageChops.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PdfImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/features.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PcfFontFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PcxImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageOps.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImagePalette.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/TgaImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageSequence.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/_util.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/TiffTags.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/BmpImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/DdsImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ExifTags.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PyAccess.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PpmImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageCms.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PaletteFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/XVThumbImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/IcoImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageFilter.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageFont.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/GifImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/GbrImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/XbmImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PngImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/FontFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/IcnsImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/TiffImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/_tkinter_finder.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/DcxImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/OleFileIO.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/SpiderImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/JpegImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/GribStubImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/MpegImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PalmImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/BdfFontFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/GimpGradientFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PsdImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/WalImageFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PcdImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/version.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/FpxImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageQt.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/Jpeg2KImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageDraw.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/GdImageFile.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ContainerIO.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/CurImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/FliImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageMode.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/JpegPresets.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/PixarImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageColor.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/BufrStubImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageTransform.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImageShow.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/TarIO.py -> build/lib.linux-armv7l-3.5/PIL
copying PIL/ImtImagePlugin.py -> build/lib.linux-armv7l-3.5/PIL
running egg_info
writing dependency_links to Pillow.egg-info/dependency_links.txt
writing requirements to Pillow.egg-info/requires.txt
writing top-level names to Pillow.egg-info/top_level.txt
writing Pillow.egg-info/PKG-INFO
warning: manifest_maker: standard file ‘-c’ not found

reading manifest file ‘Pillow.egg-info/SOURCES.txt’
reading manifest template ‘MANIFEST.in’
warning: no files found matching ‘*.sh’
no previously-included directories found matching ‘docs/_static’
warning: no previously-included files found matching ‘.coveragerc’
warning: no previously-included files found matching ‘codecov.yml’
warning: no previously-included files found matching ‘.editorconfig’
warning: no previously-included files found matching ‘.landscape.yaml’
warning: no previously-included files found matching ‘.travis’
warning: no previously-included files found matching ‘.travis/*’
warning: no previously-included files found matching ‘appveyor.yml’
warning: no previously-included files found matching ‘build_children.sh’
warning: no previously-included files found matching ‘tox.ini’
warning: no previously-included files matching ‘.git*’ found anywhere in distribution
warning: no previously-included files matching ‘*.pyc’ found anywhere in distribution
warning: no previously-included files matching ‘*.so’ found anywhere in distribution
writing manifest file ‘Pillow.egg-info/SOURCES.txt’
running build_ext

The headers or library files could not be found for jpeg,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
https://pillow.readthedocs.io/en/latest/installation.html

Traceback (most recent call last):
File “/tmp/pip-build-inj0l_9q/pillow/setup.py”, line 787, in
zip_safe=not (debug_build() or PLATFORM_MINGW), )
File “/usr/local/lib/python3.5/distutils/core.py”, line 148, in setup
dist.run_commands()
File “/usr/local/lib/python3.5/distutils/dist.py”, line 955, in run_commands
self.run_command(cmd)
File “/usr/local/lib/python3.5/distutils/dist.py”, line 974, in run_command
cmd_obj.run()
File “/usr/local/lib/python3.5/site-packages/setuptools/command/install.py”, line 61, in run
return orig.install.run(self)
File “/usr/local/lib/python3.5/distutils/command/install.py”, line 539, in run
self.run_command(‘build’)
File “/usr/local/lib/python3.5/distutils/cmd.py”, line 313, in run_command
self.distribution.run_command(command)
File “/usr/local/lib/python3.5/distutils/dist.py”, line 974, in run_command
cmd_obj.run()
File “/usr/local/lib/python3.5/distutils/command/build.py”, line 135, in run
self.run_command(cmd_name)
File “/usr/local/lib/python3.5/distutils/cmd.py”, line 313, in run_command
self.distribution.run_command(command)
File “/usr/local/lib/python3.5/distutils/dist.py”, line 974, in run_command
cmd_obj.run()
File “/usr/local/lib/python3.5/distutils/command/build_ext.py”, line 338, in run
self.build_extensions()
File “/tmp/pip-build-inj0l_9q/pillow/setup.py”, line 577, in build_extensions
raise RequiredDependencyException(f)
__main__.RequiredDependencyException: jpeg

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “”, line 1, in
File “/tmp/pip-build-inj0l_9q/pillow/setup.py”, line 799, in
raise RequiredDependencyException(msg)
__main__.RequiredDependencyException:

The headers or library files could not be found for jpeg,
a required dependency when compiling Pillow from source.

Please see the install instructions at:
https://pillow.readthedocs.io/en/latest/installation.html

—————————————-
Command “/usr/local/bin/python3.5 -u -c “import setuptools, tokenize;__file__=’/tmp/pip-build-inj0l_9q/pillow/setup.py’;f=getattr(tokenize, ‘open’, open)(__file__);code=f.read().replace(‘rn’, ‘n’);f.close();exec(compile(code, __file__, ‘exec’))” install –record /tmp/pip-xp_x4w_5-record/install-record.txt –single-version-externally-managed –compile” failed with error code 1 in /tmp/pip-build-inj0l_9q/pillow/

Thanks to the comment on the following Stackoverflow post for the solution: https://stackoverflow.com/a/34631976/522859

Updating to Python 3.5.1 on RaspberryPi

Hi everyone,

Just a quick post on how to update to Python 3.5.1 the normal upgrade process isn’t working for you:


cd ~
wget https://www.python.org/ftp/python/3.5.1/Python-3.5.1.tgz
tar -zxvf Python-3.5.1.tgz
cd Python-3.5.1
./configure && make && sudo make install

You may need to restart once this is complete. Run the following to confirm that the version is 3.5.1:

pi@raspberrypi:~ $ python3 --version
Python 3.5.1

Check out the following link for more info: https://stackoverflow.com/a/37079319/522859

Check Tensorflow Version – Raspbian

Hey everyone,

Just a quick post on how to check the Tensorflow version on Raspbian with a one-liner (assumes Python 3.x):


python3 -c 'import tensorflow as tf; print(tf.__version__)'

Check out this Stackoverflow post for more info: https://stackoverflow.com/a/38549357/522859