Code demonstration of building a web application that accesses AWS API Gateway, Lambda, and Comprehend to visualize Sentiment Analysis
AWS has a variety of services in ML/AI, Robotics, and IoT that can bring life to whatever application you’re trying to build. To be able to access these services, however you need to be able to stitch together a basic full-stack template integrating AWS API Gateway and Lambda with whatever front-end you’re working with. For this demonstration, we’ll be building a Sentiment Analysis web application that accesses a REST API created on AWS API Gateway which then hits an AWS Lambda function that accesses AWS Comprehend. We’ll then process the Comprehend response to visualize the data using PlotlyJS. The purpose of the article is to help you get an idea of how to stitch a basic AWS application together, you can use the architecture as a template to build larger projects incorporating more services. Before we start, this article will be assuming some knowledge in the following topics.
- Familiarity with AWS and an AWS account (Make an account at this link)
In regards to the AWS services we will be using I’ve included a section in the beginning of the article explaining these services. If you’re already familiar with the services feel free to skip ahead to the demonstration!
NOTE: This is a long article, so feel free to skip to whatever sections you specifically need help with. If you’re looking for an introduction to just Comprehend, feel free to check out my other article on it which explains the service more in-depth. If you just want to see the project code, click here.
- AWS Services
- Project Architecture
- Front-End Setup
- Creating Lambda Comprehend Function & REST API
- Integrate Backend Lambda Function with Front-End through REST API
- Backend Functionality to work with Comprehend
- Visualizing Comprehend Results with PlotlyJS on Front-End
- Working Demo
- Entire Code & Conclusion
AWS Lambda: A serverless computing service, that allows developers to run code without managing or provisioning servers. We will be using this service to access AWS Comprehend, store user inputs and Comprehend outputs, and communicate with REST API to output results to the Front-End.
AWS S3: Amazon’s primary storage service, we will be using this service to store what the user inputs on the front-end and the Sentiment Analysis results Comprehend outputs.
Identity Access and Management (IAM): Lets you manage access of AWS services through permissions and roles. We will be creating a role for our Lambda function to be able to access AWS Comprehend, S3, and API GW.
AWS Comprehend: AWS NLP service that uses ML to perform tasks such as Sentiment Analysis, Entity Extraction, Topic Modeling, and more. For this example we will be working with its Sentiment Analysis feature.
Boto3: AWS Software Development Kit (SDK) for Python developers, we use this on our Lambda function to access other AWS services such as S3 and Comprehend.
Before tackling our problem, let us design a basic architecture of our project that will make it easy to get an understanding of the big picture.
The essence of this article is to stitch all of these different pieces together. The REST API we create with API GW will help our front and back-end communicate. Lambda is the glue of many AWS applications and is used to work with other services in your project.
Notice we have included a script tag for PlotlyJS as well, we will need this to work with Plotly to create our sentiment visualizations later. Another key piece is the onclick function “getresp()”, we will be creating this function in our script.js file later to work with our REST API and Lambda. Let’s style our page a little and we should have our front-end ready.
Now we should have a simple front-end for our application.
NOTE: This step is a little arduous and has many small details, if you would prefer a video demonstration of this portion go ahead and follow this link. However, we do work with a few extra features it does not cover so if you want to follow along exactly make sure to read the section.
Now it’s time to head over to the AWS console, once you’ve logged in head over to the IAM service. Before we can work with Lambda, we need to create a role for our Lambda function that gives it permission to work with Comprehend, S3, and API Gateway. Once at the IAM service, click Roles on the left side of the page and click Create Role. Now you choose the service the role is for, which is Lambda in our case. Click next: Permissions, and now we can look up policies in the search tab to attach to our Role. Make sure to check AmazonS3FullAccess, AmazonComprehendFullAccess, AmazonAPIGatewayInvokeFullAccess, and AmazonAPIGatewayAdministrator. Click next to check tags, and then next to review and name your role as you wish. You should arrive at a page that looks something like this.
Go ahead and click create role and we can head over to the Lambda service. Click Create Function, name your Lambda function, choose Python 3.8 as your runtime, and click Change default execution role. Now choose use an existing role and select the role that you’ve created for the Lambda service.
Go ahead and create function and we have our Lambda function ready to go. Right now the Lambda function only returns Our next step is creating a REST API with API GW and integrating it with our Lambda function. Go to the API Gateway service on the console and click create API. Choose Build REST API, name your API, and click create.
Now we need to focus on API integration with Lambda, for that we need to add the appropriate methods and headers. First let’s go to actions and click create resource. Here name your resource which defines a path for the API.
Now that we have a resource path we can create access methods (GET, POST, etc). Click on your resource path and click on actions and create method. First let’s make a GET method, when configuring make sure to select integration type as Lambda Function and choose the Lambda function that you created earlier and save the method.
Make sure to click ok when the screen asks for giving API GW permission to access your Lambda. You should have an API flow that looks something like what’s below.
Now we need to deploy the API before we can test it with the Lambda function. Click on the GET method, go to actions click deploy API and fill out the Stage name, for this case we will be doing the prod stage.
After deploying you should have a URL, this will not work yet as we need to use the correct path for the resource we created. Go to the method we created and click the URL there.
Once clicking the URL we should return whatever our Lambda is returning which is just “Hello from Lambda” at the moment.
Now that we have Lambda and API GW working together, we need to create a POST method for our front-end to be able to send data to the API we’ve created. Similar to the way we created a GET method, create a POST method and integrate it with our Lambda function.
Now there’s a few key elements we need to fix to make sure our front-end can access our API properly. Go to Actions once again on the resources page and click Enable CORS, for both the GET and POST methods. After enabling CORS, you should receive a message similar to the one below.
Now on the Lambda end we need to make sure we’re returning the appropriate headers, go to our Lambda function and adjust the return statement as following.
Now that we’ve gotten Lambda and our REST API working together, we need to make sure our front-end can hit our function through API Gateway.
We will be using the Fetch API to work with our REST API. Go to your script.js file in your project directory and make a function called getResp(). This is what we will be calling when our button is clicked and it will contain our work with the Fetch API. Copy the correct URL path and input your specific URL in the fetch call.
To test if our Lambda is returning the proper output let us submit a random word such as test into our input textarea and we should receive an alert box and log with the Lambda response body.
Great! We now have all of our architecture working together and we can get to the fun part of working with Comprehend on the backend.
For our backend work, we will need the following imports in our lambda function file.
Now that we have our input, we want to create S3 storage buckets to store input and output results. We can create buckets and store these inputs using the Boto3 API. Use create_bucket with the S3 client to create buckets and put_object to put inputs into its bucket. To differentiate between inputs in storage we are using the time library to label the text file by its time-stamp.
NOTE: S3 bucket names need to be lower-cased
Now that we have our text ready for analysis, we create a Comprehend client using boto3 so that we can access Comprehend calls such for Sentiment Analysis.
Before we can work with our text we need to understand the Comprehend calls available to us. For Sentiment Analysis there are two calls: detect_sentiment and batch_detect sentiment. Detect_sentiment can take text that is less than 5000 bytes in size, while batch_detect_sentiment is for large pieces of text and can take a list of 25 strings/documents with each string being a maximum of 5000 bytes. Thus, we need to develop some functions to process our input, determine size of the input, properly split our input if we are using the batch call, and make sure our input hits the right call.
Both calls return results in the form of percentages of four different categories: Positive, Negative, Neutral, and Mixed. Now we can implement these functions in our main function lambda_handler function and properly parse and store the output before sending it back to the Front-End. For the batch sentiment call we average sentiment across the sentences and return the overall sentiment.
Now that we have our backend properly configured we need to process what is being returned to our Front-End. We’re returning decimals rather than percentages from Comprehend so we parse the data in our getResp() JS function and prepare it for visualization with Plotly.
Here we utilize PlotlyJS to create a simple piechart with the four percentages/categories Comprehend returns. We can create another function that utilizes Plotly to return a chart.
Now we can call this function in our getResp function and pass the values we parsed.
If you’ve gotten this far (thank you if you have), let’s go ahead and see an example now that our project is finally pieced together!
To access all the code for the demonstration, go to the link posted above. The largest takeaway from this project is how crucial it is to understand your architecture. By visualizing the flow of your work you’re able to understand how each component works with each other. AWS for all its amazing services at the end of the day will not fit in seamlessly with your applications unless you understand small yet essential aspects such as IAM roles and permissions.
I hope that this article has been useful for anyone trying to work with Full-Stack Serverless Development, AWS, and NLP. Feel free to leave any feedback in the comments or connect with me on Linkedln if interested in chatting about ML & Data Science. Thank you for reading!