Video Converter - Microservices Application

A Python-based microservice application for converting MP4 videos to MP3 audio files, deployed on AWS Elastic Kubernetes Service (EKS).

Overview

This project demonstrates a microservices architecture with four main components:

• Auth Server: Handles user authentication and authorization
• Converter Module: Processes video-to-audio conversion tasks
• Database Server: PostgreSQL and MongoDB for data persistence
• Notification Server: Sends email notifications and handles two-factor authentication (2FA)

Architecture

Table of Contents

• Prerequisites
• Deployment Overview
• AWS EKS Setup
• Database Deployment
• Microservices Deployment
• Configuration
• API Documentation
• Cleanup

Prerequisites

Before you begin, ensure you have the following installed and configured:

1. AWS Account: Create an AWS account if you don't have one. Follow the AWS setup guide.

2. Helm: Kubernetes package manager. Install from the official Helm documentation.

3. Python: Required for local development. Download from the official Python website.

4. AWS CLI: Install the AWS Command Line Interface from the official installation guide.

5. kubectl: Kubernetes command-line tool. Install the latest stable version from the Kubernetes documentation.

6. Database Access: Ensure you have access to set up PostgreSQL and MongoDB instances.

Deployment Overview

The deployment process follows these high-level steps:

1. AWS EKS Cluster Setup: Create and configure the EKS cluster with appropriate IAM roles and node groups
2. Database Deployment: Deploy MongoDB and PostgreSQL using Helm charts
3. Message Queue Setup: Deploy RabbitMQ and create required queues
4. Microservices Deployment: Deploy all four microservices (auth, gateway, converter, notification)
5. Configuration: Configure email settings and secrets
6. Validation: Verify all components are running correctly

AWS EKS Setup

Step 1: Create IAM Roles

Create EKS Cluster IAM Role

1. Follow the steps in the AWS EKS service IAM role documentation using the root user.

2. After creation, your role should look similar to this:

   

3. Important: Explicitly attach the AmazonEKS_CNI_Policy if it is not attached by default.

Create Node IAM Role (AmazonEKSNodeRole)

1. Follow the steps in the Amazon EKS node IAM role documentation using the root user.

2. Note: You do NOT need to configure any VPC CNI policy mentioned after step 5.e in the documentation.

3. Attach the following policies to your role (if not attached by default):

   • AmazonEKS_CNI_Policy
   • AmazonEBSCSIDriverPolicy
   • AmazonEC2ContainerRegistryReadOnly

4. Your AmazonEKSNodeRole should look similar to this:

   

Step 2: Create EKS Cluster

1. Log in to the AWS Management Console with your AWS account credentials.
2. Navigate to the Amazon EKS service from the AWS Console dashboard.
3. Click "Create cluster."
4. Configure your cluster:
   • Choose a name for your cluster
   • Configure networking settings (VPC, subnets)
   • Select the eksCluster IAM role created in Step 1
5. Review and create the cluster.
6. Wait for the cluster to provision (this may take several minutes).
7. Once the cluster status shows as "Active," proceed to create node groups.

Step 3: Create Node Group

1. In the "Compute" section of your cluster, click "Add node group."
2. Configure the node group:
   • Choose the AMI (default recommended)
   • Select instance type (e.g., t3.medium)
   • Set the number of nodes
   • Select the AmazonEKSNodeRole created earlier
3. Click "Create node group."

Step 4: Configure Security Group

Important: Ensure all necessary ports are open in the node security group.

Step 5: Enable EBS CSI Addon

1. Enable the ebs-csi-driver addon in your EKS cluster.

2. This addon is required for enabling Persistent Volume Claims (PVCs) once the cluster is created.

   

Step 6: Configure kubectl

1. Clone this repository:

   git clone <repository-url>
   cd microservices-python-app-main

2. Set the cluster context:

   aws eks update-kubeconfig --name <cluster_name> --region <aws_region>

3. Verify the connection:

   kubectl get nodes

Database Deployment

MongoDB

1. Navigate to the MongoDB Helm chart directory:

   cd Helm_charts/MongoDB

2. Edit values.yaml and set your database username and password.

3. Install MongoDB:

   helm install mongo .

4. Wait for MongoDB to be ready:

   kubectl get pods -w

5. Connect to MongoDB:

   mongosh mongodb://<username>:<pwd>@<nodeip>:30005/mp3s?authSource=admin

PostgreSQL

1. Navigate to the PostgreSQL Helm chart directory:

   cd ../Postgres

2. Edit values.yaml and set your database username and password.

3. Install PostgreSQL:

   helm install postgres .

4. Wait for PostgreSQL to be ready:

   kubectl get pods -w

5. Connect to PostgreSQL and initialize the database:

   psql 'postgres://<username>:<pwd>@<nodeip>:30003/authdb'

6. Copy and execute all queries from the init.sql file in the PostgreSQL directory.

RabbitMQ

1. Navigate to the RabbitMQ Helm chart directory:

   cd ../RabbitMQ

2. Install RabbitMQ:

   helm install rabbitmq .

3. Wait for RabbitMQ to be ready:

   kubectl get pods -w

4. Access the RabbitMQ management interface:

   • Open your browser and navigate to: http://<nodeIp>:30004
   • Use default credentials:
     • Username: guest
     • Password: guest

5. Create Required Queues: Before deploying the converter module, create two queues in RabbitMQ:

   • mp3
   • video

Note: Ensure all necessary ports (30003, 30004, 30005) are open in the node security group.

Microservices Deployment

Deploy each microservice by applying its manifest files:

Auth Service

cd auth-service/manifest
kubectl apply -f .

Gateway Service

cd gateway-service/manifest
kubectl apply -f .

Converter Service

1. Before deploying, configure the email and password in the secret file:

   cd converter-service/manifest
   # Edit secret.yaml with your email and password

2. Apply the manifest:

   kubectl apply -f .

Notification Service

1. Configure email settings (see Notification Configuration below).

2. Apply the manifest:

   cd notification-service/manifest
   kubectl apply -f .

Configuration

Notification Configuration

To configure email notifications and two-factor authentication (2FA):

1. Go to your Gmail account and click on your profile.
2. Click on "Manage Your Google Account."
3. Navigate to the "Security" tab on the left side panel.
4. Enable "2-Step Verification."
5. Search for "App passwords" in the settings.
6. Click on "Other" and provide a name for this application.
7. Click "Generate" and copy the generated password.
8. Edit notification-service/manifest/secret.yaml and paste:
   • Your email address
   • The generated app password

Application Validation

After deploying all microservices, verify the status of all components:

kubectl get all

Check that all pods are in the Running state:

kubectl get pods

View service endpoints:

kubectl get services

API Documentation

The application exposes the following API endpoints through the gateway service (port 30002):

Login Endpoint

Authenticate and receive a JWT token.

Request:

POST http://<nodeIP>:30002/login

Example using curl:

curl -X POST http://<nodeIP>:30002/login -u <email>:<password>

Expected Response: success! (with JWT token)

Upload Endpoint

Upload an MP4 video file for conversion.

Request:

POST http://<nodeIP>:30002/upload
Authorization: Bearer <JWT Token>
Content-Type: multipart/form-data

Example using curl:

curl -X POST \
  -F 'file=@./video.mp4' \
  -H 'Authorization: Bearer <JWT Token>' \
  http://<nodeIP>:30002/upload

Expected Response: A file identifier (fid) will be sent to your email.

Download Endpoint

Download the converted MP3 file.

Request:

GET http://<nodeIP>:30002/download?fid=<Generated file identifier>
Authorization: Bearer <JWT Token>

Example using curl:

curl --output video.mp3 \
  -X GET \
  -H 'Authorization: Bearer <JWT Token>' \
  "http://<nodeIP>:30002/download?fid=<Generated fid>"

Expected Response: The MP3 file will be downloaded to your local machine.

Cleanup

To destroy the infrastructure and clean up resources:

1. Delete Microservices:

   kubectl delete -f auth-service/manifest/
   kubectl delete -f gateway-service/manifest/
   kubectl delete -f converter-service/manifest/
   kubectl delete -f notification-service/manifest/

2. Uninstall Helm Charts:

   helm uninstall mongo
   helm uninstall postgres
   helm uninstall rabbitmq

3. Delete Node Group:

   • Navigate to your EKS cluster in the AWS Console
   • Go to the "Compute" tab
   • Select the node group and click "Delete"

4. Delete EKS Cluster:

   • Once the node group is deleted, navigate to the cluster overview
   • Click "Delete" and confirm the deletion

Note: Ensure all resources are properly cleaned up to avoid unnecessary AWS charges.