AWS troubleshooting - Lamba deployment package file permissions

When creating your own Lambda deployment packages be aware of the permissions on the files before zipping them. Lambda requires the files to have read access for all users, particularly "other", if this is missing you will receive a non-obvious error when trying to call the function. The fix is simple enough, perform a 'chmod a+r *' before creating your zip file. If the code is visible in the inline editor adding an empty line and saving will also fix the problem, presumably by overwriting the file with the correct permissions.

Below are some examples of errors you will see in the various languages if read permissions are missing. Hopefully this post will have saved you some time debugging.

Java CloudWatch logs:

--

Class not found: example.Hello: class java.lang.ClassNotFoundException

java.lang.ClassNotFoundException: example.Hello

at java.net.URLClassLoader$1.run(URLClassLoader.java:370)

at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:361)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:348)

Caused by: java.io.FileNotFoundException: /var/task/example/Hello.class (Permission denied)

at java.io.FileInputStream.open0(Native Method)

at java.io.FileInputStream.open(FileInputStream.java:195)

at java.io.FileInputStream.<init>(FileInputStream.java:138)

at sun.misc.URLClassPath$FileLoader$1.getInputStream(URLClassPath.java:1251)

at sun.misc.Resource.cachedInputStream(Resource.java:77)

at sun.misc.Resource.getByteBuffer(Resource.java:160)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:454)

at java.net.URLClassLoader.access$100(URLClassLoader.java:73)

at java.net.URLClassLoader$1.run(URLClassLoader.java:368)

... 7 more

--

Java execution result (testing from console):

{

  "errorMessage": "Class not found: example.Hello",

  "errorType": "class java.lang.ClassNotFoundException"

}

Python CloudWatch logs:

--

Unable to import module 'python-hi': No module named python-hi

--

Python execution result (testing from console):

{

  "errorMessage": "Unable to import module 'python-hi'"

}

Node CloudWatch logs:

--

module initialization error: Error

    at Error (native)

    at Object.fs.openSync (fs.js:549:18)

    at Object.fs.readFileSync (fs.js:397:15)

    at Object.Module._extensions..js (module.js:415:20)

    at Module.load (module.js:343:32)

    at Function.Module._load (module.js:300:12)

    at Module.require (module.js:353:17)

    at require (internal/module.js:12:17)

--

Node execution result (testing from console):

{

  "errorMessage": "EACCES: permission denied, open '/var/task/node-hi.js'",

  "errorType": "Error",

  "stackTrace": [

    "Object.fs.openSync (fs.js:549:18)",

    "Object.fs.readFileSync (fs.js:397:15)",

    "Object.Module._extensions..js (module.js:415:20)",

    "Module.load (module.js:343:32)",

    "Function.Module._load (module.js:300:12)",

    "Module.require (module.js:353:17)",

    "require (internal/module.js:12:17)"

  ]

}

AWS Tip of the day: Tagging EC2 reserved instances

A quick post pointing out that EC2 reserved instances actually support tagging. This functionality is only available on the command line of via the API and not via the console but it still allows to you tag your reservations making it easier to keep track of why a reserved instance was purchased and what component it was intended for. Of course the reservation itself is not actually tied to a running instance in any way, it is merely a billing construct that is applied to any matching instances running in your account but if you are making architectural changes or considering different instance types for specific workloads or components the tags allow you (and your team) to see why the reservation was originally purchased. So for example if you are scaling up the instance sizes of a specific component, lets say from m4.large to m4.xlarge, you can check your reserved instance tags and modify the reservations associated with the component to ensure you continue to benefit from the purchase.

The tagging of reserved instances works the same as tagging other EC2 resources through the AWS CLI's ec2 create-tags command and specifying the reserved instances ID as the resource ID. You can find the reserved instance ID using the CLI's ec2 describe-reserved-instances command. Using an actual example, lets start off finding a reservation:

$ aws ec2 describe-reserved-instances

{

    "ReservedInstances": [

        {

            "ReservedInstancesId": "3d092b71-5243-4e5e-b409-86df342282ab", 

            "OfferingType": "No Upfront", 

            "AvailabilityZone": "eu-west-1c", 

            "End": "2017-08-12T04:48:58.000Z", 

            "ProductDescription": "Linux/UNIX", 

            "UsagePrice": 0.0, 

            "RecurringCharges": [

                {

                    "Amount": 0.01, 

                    "Frequency": "Hourly"

                }

            ], 

            "Start": "2016-08-12T04:48:59.763Z", 

            "State": "active", 

            "FixedPrice": 0.0, 

            "CurrencyCode": "USD", 

            "Duration": 31536000, 

            "InstanceTenancy": "default", 

            "InstanceType": "t2.micro", 

            "InstanceCount": 1

        }

    ]

}

Next, lets add a tag indicating that this reservation is intended for the "production" stack.:

$ aws ec2 create-tags --resources 3d092b71-5243-4e5e-b409-86df342282ab --tags Key=Stack,Value=production

Checking the result:

$ aws ec2 describe-reserved-instances

{

    "ReservedInstances": [

        {

            "ReservedInstancesId": "3d092b71-5243-4e5e-b409-86df342282ab", 

            "OfferingType": "No Upfront", 

            "AvailabilityZone": "eu-west-1c", 

            "End": "2017-08-12T04:48:58.000Z", 

            "ProductDescription": "Linux/UNIX", 

            "Tags": [

                {

                    "Value": "production", 

                    "Key": "Stack"

                }

            ], 

            "UsagePrice": 0.0, 

            "RecurringCharges": [

                {

                    "Amount": 0.01, 

                    "Frequency": "Hourly"

                }

            ], 

            "Start": "2016-08-12T04:48:59.763Z", 

            "State": "active", 

            "FixedPrice": 0.0, 

            "CurrencyCode": "USD", 

            "Duration": 31536000, 

            "InstanceTenancy": "default", 

            "InstanceType": "t2.micro", 

            "InstanceCount": 1

        }

    ]

}

Great we have a tag but what if we have hundreds of reservations, a long list of reservations is not particularly useful for quickly identifying the reservations related to a component or stack. The CLI's query and output functionality can help here:

$ aws ec2 describe-reserved-instances --query 'ReservedInstances[*].{AZ:AvailabilityZone,Type:InstanceType,Expiry:End,stack:Tags[?Key==`Stack`][?Value==`production`]}' --output=table
--------------------------------------------------------
|               DescribeReservedInstances              |
+-------------+----------------------------+-----------+
|     AZ      |          Expiry            |   Type    |
+-------------+----------------------------+-----------+
|  eu-west-1c |  2017-08-12T04:48:58.000Z  |  t2.micro |
+-------------+----------------------------+-----------+
Not quite the console view but easy enough to see that we have one reservation for the "production" Stack.