Simple Storage Service (S3)¶

S3 is a cloud storage solution used primarily for assets (documents, images, other files), not data. While similar to Google Drive, S3 is a more configurable storage service commonly used for enterprise applications. It’s also typically used in conjunction with other AWS services, like DynamoDB.

S3 offers no user-facing GUI by default, and it has no concept of folders. Instead, assets are stored within buckets as objects.

Buckets are where assets are stored within the S3 cloud. Buckets function less like folders to be navigated between, and more as storage areas that are distinct from one another. See AWS docs’s Buckets overview for more information.
Objects are the individual assets stored within those buckets. Each object has a unique identifier known as a key. See AWS docs’s Objects overview for more information.

Buckets have a flat data structure, meaning there is no directory tree. Despite this, it is possible to have the appearance of a directory tree by grouping objects as “folders.” This simply means using forward slash characters “/” within an object’s key. Skuid can also interpret these objects as folders. Additionally, when updating an S3 bucket in the AWS console, AWS creates 0-byte folder objects that represent folders. Skuid does not create these files in its UI, though it can parse them if they were created in the AWS console.

For more general information, see Amazon’s Introduction to Amazon S3.

Configuration¶

Using the Amazon S3 data source type requires setting the appropriate AWS permissions, an AWS authentication provider, appropriate settings on your S3 buckets, and the creation of a data source.

Creating an AWS authentication provider¶

Before configuring data sources within Skuid, you need to decide how you’ll authenticate to your AWS services using an IAM role. We recommend consulting your AWS administrator prior to making this decision.

Assume Role [[]]

This authentication method allows you to use the ARN of an assumable role instead of hard-coded AWS access keys. When a Skuid user makes a request using this authentication method, Skuid makes a secure server-side request to assume the configured role. If this request is successful, Skuid receives temporary access credentials, which Skuid caches until the credentials expire after 60 minutes.

Requests made through an assumed role are traceable via the AWS console. From there, you can see both the Skuid site and Skuid user who initiated the request, which provides AWS administrators full visibility into who requested access credentials and from what Skuid site.

Due to this method’s security and ease-of-management after initial setup, we typically recommend it over the access keys method. However, it does require some technical knowledge to configure. Be sure to read all of the instructions below.

Before you begin

Skuid can only access roles that have a defined path containing the string /skuid-assumable-role/ . This is different from the role’s name. Role paths can only be set when first created, so it isn’t possible to update an existing role with a new path.

In order to create a role with a path, you’ll need to use either the AWS IAM API or the AWS CLI.

The instructions below assume you are using AWS CLI (version 2) and that you’ve already configured an access key ID and secret access key for use with the CLI. Ensure these credentials have the iam:CreateRole permission. Command line examples are also tailored for Linux or macOS operating systems; commands for command line programs on Windows may differ.

Finally, because IAM roles require a trust relationship policy upon creation, you’ll need to save the JSON that Skuid provides when creating an authentication provider. Because of this you’ll need to navigate between your browser window, your computer’s file explorer, as well as the command line.

In Skuid

From the Skuid navigation bar, navigate to Data Sources > Authentication Providers.

Click Create.

Configure the authentication provider’s basic properties:

Name: Enter a name, such as AWSAuth.

Authentication Method: AWS: Assume Role with ARN

In the Trust Relationship Policy, click Copy to Clipboard.

This copied policy will enable your role to grant proper access to your Skuid site.
Note

If you want this role to be assumable by multiple Skuid sites—for example a staging site and a production site, make the sts:ExternalId property’s value into an array of Skuid site Ids.

For example, change this key/value:
"sts:ExternalId": "09583eba-de0e-49d3-ae42-61b3927a61b1"
Into this:
"sts:ExternalId": [
   "09583eba-de0e-49d3-ae42-61b3927a61b1",
   "e354e60f-5a80-4024-b01a-4cae13d0948c"
]
Site IDs are accessible from Settings > Site > Profile.
On your local machine

With the policy copied to your clipboard, paste the policy into a text editor.

Save the file to an easily accessible directory (for example, your desktop) on your machine with a recognizable name and the JSON file extension, like trust-policy.json

In a command line interface

With the trust policy saved to your machine, you can now use the AWS CLI to create an assumable role.
Open your command line program of choice and navigate to the directory containing your policy JSON file. For example, if you saved the file on your desktop:
1
cd ~/Desktop
Use the AWS CLI’s iam create-role command to create the role with the proper path value and trust relationship policy:
1
aws iam create-role --role-name Skuid-S3-Access --path /skuid-assumable-role/ --assume-role-policy-document file://trust-policy.json
Note

While role-name can vary from this example, the path must contain /skuid-assumable-role/

The CLI creates the role, outputting its ARN and other related metadata. Because of the configured path, your ARN should look similar to this: arn:aws:iam::1234567891011:role/skuid-assumable-role/Skuid-S3-Access.
Copy this created ARN, either from the command line output or the AWS console.
In Skuid

Return to the authentication provider window.

Paste the copied ARN into the AWS Role ARN to assume field.

Click Save.

Access Keys [[]]

This authentication method utilizes an Amazon Web Service (AWS) IAM role’s access keys to authenticate to an AWS system.

If you are not the manager of your AWS services, follow along with Amazon’s steps for creating access keys with your IT administrator. You’ll need to do the following:

In AWS

Create a new group and select the appropriate policies for that group.

These should align with your intended Skuid use. Ensure your access policies match the services (S3, DynamoDB, or SNS) you’ll be using.

Create a new user in AWS Identity and Access Management (IAM) within the newly created group.

Retrieve the access credentials generated for that user:

The secret access key is only displayed once, immediately after the access key is generated, and cannot be found later. Be sure to copy it to a safe place.

In Skuid

With your access credentials handy, you can now use them to configure a Skuid authentication provider that will work with all AWS data sources.

Navigate to Configure > Data Sources > Authentication Providers.

Click New Authentication Provider.

Configure the authentication provider’s basic properties:

Name: Enter a name, such as AWSAuth.

Authentication Method: AWS: Access Keys

Enter your IAM role’s credential information:

AWS Access Key Id: The access key ID to use when authenticating.

AWS Secret Access Key: The secret access key to use when authenticating.

Click Save.

Adding AWS access permissions¶

After choosing an authentication method for your role, you must grant the role a certain set of IAM permissions to access AWS resources. Add each of these permissions to the role used to authenticate Skuid.

s3:DeleteObject
s3:GetObject
s3:GetObjectAcl
s3:ListAllMyBuckets
s3:ListBucket
s3:PutObject

You can add these permissions using the AWS Console or the AWS CLI. You may use any managed policy that includes these permissions, or create a custom inline policy. For more information about these access options, refer to AWS documentation.

Note

When using AWS APIs, it is best practice to utilize strict and well-defined IAM policies so end users can only perform actions they have explicit access to.

Any examples within Skuid’s documentation assume the authenticated role has full access, but Skuid adheres to any policies associated with the AWS access credentials provided to it.

These permissions are important to consider in the broader context of your AWS implementation. Take care when assigning and unassigning permissions.

Creating an S3 data source¶

Navigate to Data Sources > Data Sources.
Click Create.
Select Amazon S3 for Type.
Enter an appropriate value for Name, such as AWS-ServiceName.
Click Next.
Select the previously configured AWS authentication provider.
Select the AWS Region where the resources you need within Skuid reside. See AWS documentation on regions for more information.
Click Save.

After creation, leave Use proxy disabled. The proxy does not currently support Amazon S3 data sources.

Configuring CORS¶

Cross-origin resource sharing (CORS) is an important mechanism in ensuring end-user security, only allowing specifically named external origins to access resources in the current domain. Because S3 buckets are at a differnet origin from your Skuid site, you must configure these rules on each bucket. Otherwise S3 will decline any attempts to modify the resources within a bucket.

Note

Depending on your organization’s security needs, your specific CORS configuration could vary. The CORS configuration examples below are extremely permissable for the sake of setup.

For more information on the elements that make up an S3 bucket’s CORS configuration, see AWS documentation.

You can set your CORS rules in different areas:

For general examples of these methods in a Skuid context, read below. For more information and reference to support other use cases, see the AWS docs linked above.

Note

Both examples below use https://example.skuidsite.com. Update this to match the domain of your Skuid site.
Command line examples are also tailored for Linux or macOS operating systems; commands for command line programs on Windows differ.

In the AWS console [[]]¶

Navigate to the AWS console’s S3 service page.
Click the bucket you wish to configure.
Click Permissions.
Beside the Cross-origin resource sharing (CORS) header, click Edit.

Paste the following JSON into the field:

[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "PUT",
            "POST",
            "DELETE"
        ],
        "AllowedOrigins": [
            "https://example.skuidsite.com"
        ],
        "ExposeHeaders": []
    }
]

Click Save changes.

Via the AWS CLI [[]]¶

Save the following JSON as a file on your local machine in an easily accessible location as cors.json:

{
  "CORSRules": [
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET",
            "PUT",
            "POST",
            "DELETE"
        ],
        "AllowedOrigins": [
            "https://example.skuidsite.com"
        ],
        "ExposeHeaders": []
    }
  ]
}

Note

Notice that, when using the AWS CLI, rules must be explicity listed in a CORSRules array.

Navigate to the folder containing the saved file.

# For example, if saved to the desktop...
cd ~/Desktop

Set the bucket’s CORS configuration with the AWS CLI:

aws s3api put-bucket-cors --bucket MyBucket --cors-configuration file://cors.json

Using the AWS S3 data source¶

The S3 data source type has two model entities: Bucket and Object.

Bucket¶

This model entity represents the “buckets” where data objects are stored. It only has two, self-explanatory fields:

Name
Creation Date

This entity’s data cannot be edited, and it cannot accept any conditions.

One common use for the Buckets entity would be to create a row action that activates and sets the Object entity’s Bucket model condition.

Object¶

Objects are the individual data objects within an S3 bucket. They can be standard files, but they can also contain any data value. For more information, see AWS’s Working with Amazon S3 objects.

Fields¶

A model using the Object entity will always include in these two fields:

Bucket: The S3 bucket in which the object exists.
Key: The unique identifier of an object within an S3 bucket. This key can also contain a “folder path” established by forward slashes.

There are several other fields available. Some of the most useful include:

Object Name: Generated by Skuid for convenience, this field displays the text that follows the final forward slash in an object’s key.

For example, an object with the key firstfolder/secondfolder/object1 would have an object name of object1.
Storage Class: The AWS storage class for that piece of data.
Is Folder: Returns whether or not an object is seen by S3 as a “folder,” meaning it’s a 0-byte object whose key ends with a forward slash. For more information, see Amazon’s Working with Folders.

Conditions¶

This entity has are two configurable conditions—available by default—one of which is required:

Bucket (required) : Restricts which objects are loaded based on the selected bucket.
FolderPath: Related to the S3 concept of folders, this restricts data based on the “folder path” of the key field.

For example, a condition with the value “firstfolder/secondfolder/” would return objects with the following keys:
- firstfolder/secondfolder/object1, firstfolder/secondfolder/object2, etc.

Uploading files¶

Since S3 is a data storage service, Skuid’s File Upload component can be used to upload files to an S3 bucket.

S3 Direct Upload¶

File Storage Location: S3 Direct Upload
- This default value simply means files will be placed within the bucket of the Parent Model. This value the cannot be changed.
Label: Enter a label to insert custom text for the upload button. If you leave this field blank, the upload button will be labeled “Upload.”
File Name: Use this field to rename any files uploaded through this component. Anything entered here will be set as the file’s key.
- This field is compatible with merge syntax.
- If this field is blank, files will retain their current file name as their S3 key.
- Using a dot within this field will cause the file’s extension/file type to be overwritten.
  - If used to rename “spreadsheet.xlsx” to ”budget,” the resulting filename would be “budget.xlsx.”)
  - If used to rename “spreadsheet.xlsx” to ”budget.csv,” the resulting file would be “budget.csv.” )
Snippet to Run on Completion: Enter the name of any JavaScript snippets you wish to run once a file has been uploaded to S3.
ACL of File: Determines the Access Control List for files uploaded through this component.

In some instances, you may wish to to upload files to an S3 bucket not represented by a model. In this case, set the Parent Model property to None, and several new properties will appear:

Data Source Type: Select AmazonS3.
Data Source: Select the data source connected to the appropriate S3 account.
S3 Bucket Name: Enter a specific bucket name or utilize merge syntax to access a field on a model.
- In some instances, your bucket name can be contained within a separate data source—such as a DynamoDB table. Utilize merge syntax to have your File Upload component dynamically access and utilize that data.
S3 Folder Path: Related to the S3 concept of folders, enter a folder path to be prepended to the file name.

Warning

Uploading a file with a key identical to an existing S3 object will overwrite that object.

Data source actions¶

The S3 data source type has two data source actions, both of which facilitate file downloads. This action uses Skuid’s generated Object Name field to download the correctly-titled file—meaning the key’s prefixes will not be a part of the file name.

Download Row Object: Designed to be used as a row action, this action type downloads the object represented by the row in context.
- Model of Objects: Select the model pointing to the S3 objects
Download Target Object: A more flexible downloading action, this does not need a specific model or row for context. Instead, there are two properties that allow for admins to specify which object should be downloaded by this action. The target object can be set explicitly with strings or dynamically through the use of merge syntax in both of this action’s properties.
- Key of Object
- Bucket that holds Object

Troubleshooting¶

When trying to select a model entity, I receive “An internal server error occurred.” [[]]¶

This means something went wrong when requesting resources from your AWS instance. This is typically a permissions issue or a model condition issue.

The IAM user associated with your AWS access credentials within your Skuid authentication provider probably does not have the appropriate permissions for the service you are trying to access. Return to the AWS console and ensure your IAM policy settings are correct.

If your policy settings are correct, recreate your authentication provider with a new access key.

Make sure you have set the Bucket condition for any Object models.