Machine Learning with AWS Serverless Tools

At a meetup, I gave a presentation about building an API for a machine learning task.

Here’s the post I mention about running serverless tools locally.


I strongly recommend going to local usergroups like SGF Devs. It’s an easy way to build your skills outside of work, meet people from other industries and companies, and find out about other tech events in your community.

The fact that such groups self-select for people that are interested in learning is significant: it means that you’ll be around people that are curious by nature, interested in sharing knowledge, and are passionate about the craft.

So take a look, see if there are any meetups in your area.

Q-Learning Nim with Python

In this post I’m going to explain a simple AI that can learn to win a game with no previous knowledge of the rules, goals, or decision process of its opponent.

The Game

Nim is a game where two players take turns picking up sticks from a board that contains a number of sticks. Each player can take one, two, or three sticks. In this case, the goal is to avoid taking the last stick.

This makes a nice example for tabular Q-Learning because there are a limited number of game states, there are the same available actions in each state, and the states and actions both have finite integer values. It’s also deterministic, so it is easy to create an AI that can learn the game well enough to win every time.

Q-Learning

Machine learning is a way of taking some information, and then “learning” something about it through math. For example, if you have the prices of a bunch of homes in an area, and their square footage, you could predict the value of another home without having to explicitly write out the relationship between size and value in your code.

Reinforcement learning is where an AI uses machine learning to make decisions in an environment. As an example, you could have a robot navigating a rocky landscape. It could keep track of its actions, like reaching a target location, or getting stuck, and you could assign “rewards” to those actions. After some time driving around the environment, it could figure out that driving up on a rock is bad, and driving toward the target is good. Eventually it could be trained to navigate the environment while avoiding obstacles.

The key parts of that to remember are: rewards, state, and action.

Q-Learning is a type of reinforcement learning where the AI keeps track of the “quality” of an action in a state. As the AI explores the environment, the quality values for each state and action explored are updated based on the quality value of the state reached and what rewards were given.

You can find more information about Q-Learning on page 131 of Richard S. Sutton’s “Reinforcement Learning” (link to PDF on author’s site).

Nim.py

import random
import matplotlib.pyplot as plt

# Window size for rolling average in win rate graph
GRAPH_SMOOTHING = 5

# Number of sticks
GAME_SIZE = 10

# Max number of sticks that can be removed in a single turn
MAX_STICKS_REMOVABLE = 3

ACTION_SPACE = [i for i in range(1,MAX_STICKS_REMOVABLE + 1)]

Here I import pyplot to graph the win rate of the AI. I also set some default values. The action space is an array of possible moves, which is the same for any state in this game: remove one, two, or three sticks.

class Game:
    '''
    Keeps track of the game state.
    '''
    def __init__(self):
        '''
        Sets the initial number of sticks.
        '''
        self.sticks = GAME_SIZE

    def remove(self, count):
        '''
        Remove a number of sticks from the board.
        :param count: Number of sticks to remove.
        '''
        self.sticks -= count

    def getState(self):
        '''
        Converts the number of sticks to a "state id" for the learner.
        :return: state id.
        '''
        return self.sticks - 1

    def getActionSpace(self):
        '''
        :return: Possible actions in the game.
        '''
        return ACTION_SPACE

    def isOver(self):
        '''
        :return: True if all sticks have been removed and the game is "done".
        '''
        return self.sticks <= 0

This is the code for Nim itself. It keeps track of the state of the game, and makes the possible actions available to the learner.

class QLearner:

    def __init__(self):
        '''
        Initialize the Q-Table
        '''
        self.q = []
        for i in range(GAME_SIZE):
            self.q.append([0] * len(ACTION_SPACE))

Here’s the interesting stuff: the code for the Q-Learner. The constructor initializes a 2D array, with one dimension being every possible game state, and another being every possible action.

class QLearner:

...

    def getMove(self, state):
        potential_actions = self.q[state]
        action_chosen = potential_actions.index(max(potential_actions))
        return action_chosen

This function takes a state and chooses the action for that state that will lead to the maximum reward.

In other circumstances you might want the learner to explore the environment, so it would have a choice between a random action and one with high value. Typically to explore or go for high value would be randomly chosen, weighted by a ratio, which would be reduced over time towards taking the optimal action.

In reinforcement this is called the “exploration vs. exploitation tradeoff”.

class QLearner:

...

    def learn(self, state, action, new_state, reward, is_over):
        if is_over:
            self.q[state][action] = reward
        else:
            self.q[state][action] = reward + max(self.q[new_state])

This is the critical part, the code that learns. It considers the previous state, the action taken, the new state, the reward, and whether the game is over. Remember that not every action results in a reward, so this value may be zero.

If the game is over, it knows exactly what the value of taking an action in the previous state was: the value is the reward.

If the game is not over, then it sets the expected value of the action in the previous state to be the reward, plus the average value of the actions in the new state.

The values built up by this function eventually become a sort of “map” for the AI, where it uses this information about the value of different actions to follow a “trail” of states with the highest value.

The cool thing about this is that it doesn’t have built in knowledge of the game. It doesn’t do any planning at the start of the game, it doesn’t know its opponent’s strategy, it doesn’t even know what it needs to do to win. This means that it’s general, that you could use this exact same code for something else, like tic-tac-toe.

A couple things have been left out of the learning code for simplicity. Learning rate, which would look like

self.q[state][action] = self.q[state][action] + learning_rate * (reward + max(self.q[new_state]) - self.q[state][action])

This would mean that the Q-Table would be updated in smaller increments, which is useful for things that have some randomness to them. Since Nim is a deterministic game, Nim.py uses a learning rate of 1.

Also left out is a discount factor:

self.q[state][action] = self.q[state][action] + learning_rate * (reward + discount_factor * max(self.q[new_state]) - self.q[state][action])

This is useful if you want a Q-Learner to value earlier rewards higher. For example, if you had a Q-Learner navigating a map, you could use discount factor to train it to navigate to nearest goals first.

The code section above ends up looking quite similar to a Bellman equation for Q-Learning, which describes how the Q table is updated by new information in math terms.

class RandomOpponent:
    def __init__(self):
        return

    def getMove(self, state):
        return random.choice(Game().getActionSpace())

    def learn(self, state, action, new_state, reward, is_over):
        return

class HumanOpponent:
    def __init__(self):
        return

    def getMove(self, state):
        return int(input("Sticks to remove? "))

    def learn(self, state, action, new_state, reward, is_over):
        return

These are a couple interchangeable opponents for the AI to play against. One makes random moves, the other lets you interact with the AI.

def runTrial(learner, number_of_trials, opponent_learner_type, verbose, make_graph, print_q_values):
    q_wins = []

    opponent = opponent_learner_type()
    for _ in range(number_of_trials):
        want_to_exit = False
        game = Game()
        state = game.getState()
        while True:
            reward = 0
            action = learner.getMove(state)
            move = game.getActionSpace()[action]
            game.remove(move)
            if game.isOver():
                reward = -1
                q_wins.append(0)
            else:
                opponent_move = opponent.getMove(game.getState())
                if (opponent_move == -1):
                    want_to_exit = True
                    break
                game.remove(opponent_move)
                if game.isOver():
                    reward = 1
                    q_wins.append(1)
            new_state = game.getState()
            learner.learn(state, action, new_state, reward, game.isOver())
            state = new_state
            if game.isOver():
                break
        if want_to_exit:
            break

This code runs the games. First it creates q_wins, which will have 1’s or 0’s, depending on whether the Q-learner won each game. Then it instantiates the opponent learner, which can be human or random.

Then there is a loop that runs for each game (also called a trial in the code). The game state is reset, and then another loop is started.

Inside that loop is the code that calls the Q-Learner’s decision and learning functions. The Q-Learner gets to move first, deciding how many sticks to pick up. If it picks up the final stick, the Q-Learner lost and the reward is negative, and the opponent’s move is skipped. If the Q-Learner didn’t pick up the final stick, the opponent then gets to take their turn. If the opponent picks up the last stick, the Q-Learner’s reward is positive.

After both players take their turns, the learner is given: the number of sticks it had before the turns, the action it took, the number of sticks it had after the turns, the reward, and whether the game is over.

That repeats until one of the players wins.

After the given number of games is played, a chart is generated that shows how frequently the Q-Learner won.

I’ve omitted some code that prints out what’s going on, and some code at the end of the function that generates a graph of the win rate.

The win rate is the proportion of wins in a rolling window through all the games played. Here’s an example chart showing how the Q-Learner starts out poorly, but eventually reaches a 100% win rate as it learns the game.

Win Rate Graph

The Code

The complete source code for the Q Nim Learner can be found here.


Author Links

Setting Up Continuous Integration with CircleCI

Previously I showed you how to deploy a NodeJS app in Fargate. In this post I’ll explain how to set up continuous integration with CircleCI.

Why I Wanted to do This

In my previous post I set up a NodeJS app in Fargate, but each time I wanted to make a change to the live site, I had to run commands to build and push the docker image to ECR. I also would have had to use the AWS console to make my app’s service use the new image. All of that work is boring, time consuming, and can be automated!

On top of the convenience, this sort of setup is a good practice, because you can add a stage that runs automated tests on your application’s code. For example, if I were working on an app with a login, I could create an automated test that verified that feature every time I pushed a commit.

CircleCI is a tool that will make it possible to deploy the changes when they are pushed to a github repo. I will also show how I set up main and test environments, so that I can see what my changes will look like before publishing them.

Assumptions

I assume that you:

  • Followed the previous guide on deploying a NodeJS app in Fargate.
  • Have a github account. Bitbucket might also work, but I used github to write this guide.
  • Know how to commit and push to the github.
  • You can use basic git commands.

Part Zero: Putting the App Into Github

While making the previous guide, I already had my code in a github repo. I realized I hadn’t mentioned this, so if you just created the file in a folder and followed along without using git, you’ll need this section.

If you haven’t used git yet, you’ll also want to set your user name and email.

You’ll want all of your app’s files in their own folder.

Go to this folder in the terminal, and then follow Github’s guide for adding this local folder to your github account.

Part One:

You can set up a CircleCI account with your github account by logging in at https://circleci.com/.

You’ll see a screen like this:

Once you authorize it, you’ll be taken to the home page:

Since you haven’t set up CircleCI with the project yet, you’ll want to click “Set Up New Projects”.

Find your app, and click “Set Up Project”.

The next screen will have some instructions, but first we want to select Node from the language list.

Below it will be some instructions, and a sample .yml file.

In your app’s source code folder, make a directory called .circleci , and then put the sample file from that page into config.yml . Don’t worry about what’s in it right now.

Then, commit the new file and push it to your github repo.

Finally, click “Start Building”.

After some time, you should see a failed build. This is okay. This is expected. You’ll fix it in a later step.

Part Two: Setting Up Testing Infrastructure

Open up your AWS account, and go to ECS.

Open up the default cluster, pick the Services tab.

Click Create.

Set the service name to nodeapp-test . This service will be the “test environment” for our app.

The next screen will have a lot of network settings. You’ll want to create a new load balancer, so click the link to do so in the EC2 console. You’ll want to do this in a new tab.

You’ll want to go to EC2 in a new tab.

Click Create Load Balancer. Select Application Load Balancer.

Call it nodeapp-main, change the VPC to ECS default, and add the us-east-1a and us-east-1b availability zones.

The default settings for the Security Settings step is okay.

On the Configure Security Groups setting, pick the one that has ELB Allowed Ports. Then go to the next step, Configure Routing.

You’ll want to name the new target group nodeapp-test-target.

Don’t worry about registering any targets. Finish creating the group with the default settings.

Go back to your browser tab where you were making a new service, and pick the load balancer you just created. You may need to press the refresh button to see it in the list.

Add test-container:80:80.

And configure it like this:

Add both subnets.

Then go to the next step. You’ll be asked to configure auto-scaling, ignore that. Then click Create Service.

At this point you should have two services. I had to remake one from scratch, so you may not have one called nodeapp-main. If you’re continuing from the previous post, the one you made there is what you should use instead of nodeapp-main for the rest of this tutorial.

Part Three: Building and Pushing the Docker Image with CircleCI

Open up the app in CircleCI.

Click the settings gear on the right, and go to Environment Variables.

Here we’ll add the sensitive parameters for the build/push step. It is very important that you put them here, and never in your git repo! Although it would be technically possible to hardcode them in the config.yml file for CircleCI, this would mean that your AWS credentials would be visible to anyone with access to the source code!

If you followed the previous guide, you’ll use test-node-app for AWS_RESOURCE_NAME_PREFIX.

For pushing to ECR, you’ll need these variables. You can get them from IAM, although you may have to create a new user to get a new access key. You’ll also need to check what region you’ve been working in.

Then edit the .circleci/config.yml file to look like this:

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@0.0.2
  aws-ecs: circleci/aws-ecs@0.0.10
workflows:
  build-and-deploy:
    jobs:
      - aws-ecr/build_and_push_image:
          account-url: "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
          repo: "${AWS_RESOURCE_NAME_PREFIX}"
          region: ${AWS_DEFAULT_REGION}
          tag: "${CIRCLE_SHA1}"

This just creates a step that builds and pushes the docker image. Commit and push this change.

CircleCI will automatically pick up the change.

If you go to the ECR repo, you’ll see that it was uploaded.

Part Four: Deploying to the Test Service

Now edit the .circleci/config.yml file to look like this:

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@0.0.2
  aws-ecs: circleci/aws-ecs@0.0.10
workflows:
  build-and-deploy:
    jobs:
      - aws-ecr/build_and_push_image:
          account-url: "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
          repo: "${AWS_RESOURCE_NAME_PREFIX}"
          region: ${AWS_DEFAULT_REGION}
          tag: "${CIRCLE_SHA1}"
      - aws-ecs/deploy-service-update:
          requires:
            - aws-ecr/build_and_push_image
          aws-region: ${AWS_DEFAULT_REGION}
          family: "first-run-task-definition"
          service-name: "nodeapp-test"
          cluster-name: "default"
          container-image-name-updates: "container=test-container,tag=${CIRCLE_SHA1}"

Note that there are several hardcoded values here. If you have trouble, double check the cluster name, service name, container image name, and task definition name.

Here we add the aws-ecs/deploy-service-update step. It takes the docker image we built, and deploys it to the nodeapp-test service we created earlier.

Modify your homepage.pug to look like this:

head
  title My Dockerized Node App
body
  p Hello World!
  p Hello Continuous Integration!

Commit and push both files.

When the build runs, it should deploy to your test service.

And if you open up the load balancer in the AWS console, you’ll see a DNS name that you can use to view the new page.

After some time, you’ll see the updated page.

Feel free to play around with the homepage’s contents, now. Every time you commit and push, it should trigger a build which will update the test service.

Part 5: Deploying to the Main Service

Now that you’re successfully deploying to the test service, you’ll want to add steps to deploy to the main service.

Modify your .circleci/config.yml to look like this:

version: 2.1
orbs:
  aws-ecr: circleci/aws-ecr@0.0.2
  aws-ecs: circleci/aws-ecs@0.0.10
workflows:
  build-and-deploy:
    jobs:
      - aws-ecr/build_and_push_image:
          account-url: "${AWS_ACCOUNT_ID}.dkr.ecr.${AWS_DEFAULT_REGION}.amazonaws.com"
          repo: "${AWS_RESOURCE_NAME_PREFIX}"
          region: ${AWS_DEFAULT_REGION}
          tag: "${CIRCLE_SHA1}"
      - aws-ecs/deploy-service-update:
          name: test-deploy
          requires:
            - aws-ecr/build_and_push_image
          aws-region: ${AWS_DEFAULT_REGION}
          family: "first-run-task-definition"
          service-name: "nodeapp-test"
          cluster-name: "default"
          container-image-name-updates: "container=test-container,tag=${CIRCLE_SHA1}"
      - hold:
          type: approval
          requires:
            - test-deploy
      - aws-ecs/deploy-service-update:
          name: main-deploy
          requires:
            - hold
          aws-region: ${AWS_DEFAULT_REGION}
          family: "first-run-task-definition"
          service-name: "nodeapp-main"
          cluster-name: "default"
          container-image-name-updates: "container=test-container,tag=${CIRCLE_SHA1}"

Here you can see that we’ve added two new steps. The first is a hold, to require manual approval to deploy to the main service. The second is the main deployment step. You’ll also notice that we gave the two deployment steps a name field. This is necessary for CircleCI to distinguish the two deploy steps, so the manual hold could be put between them.

Your service name for this step may be different! You’ll want to check the AWS console and see what the name of the service you created in the previous guide was (probably test-container-service).

Also notice that we’ve added a name to the old deploy-service-update step.

After you make these changes, commit and push them. Then go to CircleCI, and open the latest build.

Once it finishes successfully, open up the build-and-deploy page under the workflow label.

Click the hold step box, then click Approve.

After that, the hold and main-deploy tasks will finish.

Open up the nodeapp-main load balancer the same way you found the nodeapp-test load balancer earlier, and find its DNS name. After some time, your changes will show up on the main service.

And you’re done! Now you’ve set up continuous integration in your app with CircleCI.

Mistakes Made

After using this tutorial to start moving my personal site to AWS, I’ve noticed that having two load balancers isn’t really necessary. I could modify the routing in a single load balancer so that it routes to the different target groups. As of writing this, an ALB with low traffic will be about $17 a month, so I should probably work on reducing the ones I’ve set up into a single ALB.

Also, for this tutorial I put the test and main environments in the same account. In a production setting, you’d have them in different accounts.

I wrote this as I figured out how to do it. If there’s something dangerously wrong, or even if you just have suggestions, please reach out to me on Twitter, LinkedIn, or by email.

What Next?

I started learning how to set up HTTPS while working on moving my personal site. Doing that would add some steps to this guide, so I’ll probably write a follow up after I get this post polished up.

Although it shouldn’t be necessary for a personal site with low traffic, it would be good to document how to add autoscaling.

At some point I’d like to add instructions for adding automated testing steps to the CircleCI build. The app I started with for this guide just displays a static page, but it would still be good to show how to verify it.


Resources Used

Deploying a NodeJS App in AWS Fargate

In this post I’ll explain how to create, upload, and run a NodeJS application in a Docker container on AWS Fargate.

Why I Wanted to Do This

I’m working on a new project. I’ll be starting out by putting together a prototype of just the front end, and then adding calls to another service on the back end to add functionality.

When I was a student at Missouri State, we used NodeJS in the web applications class. I remember it being pretty fast to get a front end set up with Express and Pug, and then it being fairly easy to add some server-side logic.

I’ve used Docker in other projects. I liked that it made it easy to deploy changes to an application. I liked that I can run an application locally for quick development, but I can then run it in the cloud without having to worry too much about discrepancies between the software and configuration on my local machine versus the cloud. It also seems to be useful for making an application scalable.

What I’m Assuming You Know

I assume that you:

  • Have Docker installed, or can install it.
  • Have an AWS account.
  • Have AWS CLI tools installed.

Part 1: Creating a Dockerized NodeJS App

Setting up a simple Dockerized app doesn’t take many files. You can create them in a project folder from the code in this post, or you can clone it from my github.

You’ll only need five files:

  • package.json
  • server.js
  • homepage.pug
  • Dockerfile
  • .dockerignore

First let’s look at package.json.

{
  "name": "docker_web_app",
  "version": "1.0.0",
  "description": "Node.js on Docker",
  "author": "Andrew Rowell <________@gmail.com>",
  "main": "server.js",
  "scripts": {
    "start": "node server.js"
  },
  "dependencies": {
    "express": "^4.16.1",
    "pug": "^2.0.4"
  }
}

There’s not much going on here. In this example app you’re just using express and pug to host some static content, but it would be easy to extend it into something more interesting.

Next is server.js.

'use strict';

// This app uses express and pug.
const express = require('express');
const pug = require('pug');

// Config
const PORT = 80;
const HOST = '0.0.0.0';

// Host homepage.pug
const app = express();
app.get('/', (req, res) => {
  res.send(pug.renderFile('homepage.pug'));
});

app.listen(PORT, HOST);
console.log(`Running on http://${HOST}:${PORT}`);

All this is doing is rendering the homepage pug file when a GET request is sent to the host on port 80. It would not be too difficult to add some code to do something more interesting, like calling an external web API.

On line 14 you can see that it is rending a file called homepage.pug

head
  title My Dockerized Node App
body
  p Hello World!

This file makes the pug engine render a simple html page with a title and paragraph.

Finally, you have the Dockerfile.

# The Docker image we're creating is based on the NodeJS 10 image.
FROM node:10

# Create app directory
WORKDIR /etc/testapp

# Install app dependencies files.
COPY package*.json ./

# Install the dependencies.
RUN npm install

# Bundle app source.
COPY . .

# Copy pug files.
COPY *.pug ./

# Open port 80
EXPOSE 80

# Run Node.js
ENTRYPOINT [ "node", "server.js" ]

This tells Docker how to build the image. It starts with an existing image that has Node 10 installed, copies the app’s code to the image, installs the app’s dependencies, opens port 80, and tells it how to start Node.

You’ll want to avoid copying debug logs and local modules to the image. That’s done with .dockerignore.

node_modules
npm-debug.log

Once you have these files created, you can build the image.

docker build -t test/node-docker-app .

This should end with the message:

Successfully tagged test/node-docker-app:latest

And now you can run the docker image.

docker run -p 12345:80 -it --init test/node-docker-app

-p 12345:80 tells it to map port 12345 to port 80 on the container. -it –init makes it possible to use Ctrl+C to close the app.

Open up a browser window, go to http://localhost:12345, and you should see this page:

Now you have a working dockerized NodeJS app.

Part 2: Uploading The Docker Image to ECR

Open up ECR in the AWS console, and click Create Repository.

Create a test-node-app repository.

Now you should see it in the repository list.

It’s worth noting that the View Push Commands button will explain how to push your built image to the repository, including instructions for Windows. For completeness, we’ll still look at how to push it with Linux/Mac.

You can also find a more permanent View Push Commands button if you click on the repo’s name.

These next steps assume that the AWS CLI has been configured for us-east-1 with a user that has the necessary permissions.

You’ll need to use the log in command to authenticate the Docker client. Here we’re using the us-east-1 region.

$(aws ecr get-login --no-include-email --region us-east-1)

Then you will need to tag the image. You can find it with Docker’s images command.

$ docker images
REPOSITORY            TAG     IMAGE ID      CREATED         SIZE
test/node-docker-app  latest  104d7a97a412  41 minutes ago  920MB

Since you’ll be pushing the latest image, you can tag it for ECR with this command:

docker tag test/node-docker-app:latest <repository id>.dkr.ecr.us-east-1.amazonaws.com/test-node-app:latest

You can find the repository id in the repo URI from the console where you created the repository earlier.

Then you push the image:

docker push <repository id>.dkr.ecr.us-east-1.amazonaws.com/test-node-app:latest

Once this command finishes, you should see the image in ECR.

Part 3: Deploying the Image

To do this, you can use the ECS First Run Wizard.

You’ll want to configure a custom container.

You’ll need to set the image to the Image URI from above in ECR. For now you can set the hard memory limit to 300 MiB, and add port 80 to the port mappings.

Once you’ve set these, click Update to finish setting up your container definition. Next you’ll need to Edit your task definition. For this sample app, I recommend the lowest memory and CPU settings.

Once you’ve done that, go to the Next page.

Here you can add a load balancer. For a small static site with little traffic this might not be necessary, but I did it anyway to see what it would be like to set one up. Then go to the Next page.

Here you’ll just need to give the cluster a name.

The final page will just show a review of the settings you configured. It should look like this.

Click Create. You’ll then be taken to a screen that looks like this, except you’ll probably see first-run-task-definition:1.

Don’t worry if this seems like it’s going slow, it’ll take a few minutes.

Once it finishes, you can click on the load balancer to see its settings.

Here you’ll see the DNS name for the load balancer. If you copy that into your browser’s URL bar, you’ll open up the Node app being hosted in AWS.

Now you’ve dockerized a Node app and deployed it to ECS!

What Next?

Before I do much more with this, I’m going to read through this list of best practices for dockerized Node and apply them. I want to look into modifying this setup so I can use HTTPS, and eventually I will need to set up autoscaling. If you know of any good information or tutorials for this, feel free to send them to me on Twitter, LinkedIn, or by email.


Resources I Used: