SeBS Tutorial: Benchmarking Serverless Computing Platforms

This hands-on session will guide you through deploying, testing, and benchmarking serverless functions using SeBS on both local Docker and OpenWhisk platforms. Alternatively, you can deploy all functions to the cloud. For details on setting up your account, please check SeBS documentation.

Main Resources

• SeBS Repository
• SeBS Paper
• SeBS-Flow Paper

---

System Requirements

OS: Linux (Ubuntu 20.04+ recommended). We have not tested SeBS extensively on WSL and macOS, but we have reports from users that it works.

Software:

• Docker 20.10+
• Python 3.7+
• Git
• jq (JSON processor)
• curl

Cloud Accounts (optional):

• AWS, Azure, or GCP account with permissions to create serverless functions (if deploying to cloud platforms)
• When deploying to OpenWhisk, a free DockerHub account is needed to publish function images. Private Docker registries are not supported on kind.

Table of Contents

Part 1 can be executed fully locally with just Docker. For part 2, you will need a cloud account or deploy OpenWhisk on your laptop - please check OpenWhisk instructions as it takes a while to get it running.

• Part 0: Setup and Prerequisites
• Part 1: Storage & Local Deployment
• Part 2: FaaS Platforms & Experiments
• Troubleshooting
• Quick Reference

---

Troubleshooting

General Debugging Tips

1. Check logs first:

   • Local: docker logs <container-id>
   • OpenWhisk: wsk activation logs <activation-id>
   • Cloud platforms: check respective dashboards

2. Start simple:

   • Test with 110.dynamic-html (no dependencies)
   • Use test input size (smallest, fastest)
   • Deploy locally before using FaaS system.

3. Verify each step:

   • After each deployment, test with one invocation
   • Check output structure before proceeding
   • Use jq to validate JSON output

4. Clean state:

   • When running into consistency issues, clean cache: rm -rf cache or modify cache entries manually (everything is in JSON)
   • Restart deployments from scratch (particularly OpenWhisk)
   • Delete and recreate cluster

5. Ask for help:

   • Join our Slack: https://serverlessbenchmark.slack.com
   • Open an issue on GitHub : https://github.com/spcl/serverless-benchmarks/issues Please provide your configuration, full command and error output

Docker Issues

Problem: permission denied when running Docker commands

Solution:

sudo usermod -aG docker $USER
# Logout and login again
docker ps  # Should work without sudo

---

Problem: Port conflicts (port 9000, 9011 already in use)

Solution:

Check that you do not already have SeBS or storage containers running from prior experiments:

docker ps

Alternatively, find process using the port:

sudo lsof -i :9000

---

OpenWhisk Issues

Problem: Pods stuck in Pending or ContainerCreating state

Solution:

Check pod details

kubectl describe pod <pod-name> -n openwhisk

You can delete the cluster and restart the deployment:

kind delete cluster

SeBS Issues

Problem: ModuleNotFoundError or import errors

Solution: Ensure the virtual environment is activated

source python-venv/bin/activate

Then, verify the activation

which python

This should show: /path/to/serverless-benchmarks/python-venv/bin/python

---

Problem: Function returns empty or error result

Solution:

Check function execution logs. The actual solution depends on the platform.

Example for OpenWhisk:

wsk activation list
wsk activation logs <activation-id>

Alternatively, test locally first. This provides the quickest setup for debugging.

./sebs.py local start <benchmark> test output.json --config configs/local.json

---

Storage Issues

Problem: MinIO unreachable from the function, particularly deployed on OpenWhisk (connection refused).

Solution: Verify that the storage configuration uses the external address of your system.

This command should provide the address of your system.

hostname -I | awk '{print $1}'

Then, update the storage config with the correct address.

jq --arg ip "$(hostname -I | awk '{print $1}')" \
   '.object.minio.address = ($ip + ":9011")' \
   storage.json > fixed_storage.json

---

Quick Reference

SeBS Commands

# Deploy and invoke
./sebs.py benchmark invoke <benchmark> <input-size> \
    --config <config.json> \
    --deployment <local|openwhisk> \
    --verbose

# Local deployment
./sebs.py local start <benchmark> <input-size> <output.json> \
    --config <config.json> \
    --deployments 1

./sebs.py local stop <output.json>

# Storage management
./sebs.py storage start object <config.json> --output-json <output.json>
./sebs.py storage stop object <output.json>

# Experiments
./sebs.py experiment invoke perf-cost \
    --config <config.json> \
    --deployment <local|openwhisk> \
    --output-dir <results/>

OpenWhisk Commands

# List actions
wsk action list

# Invoke action
wsk action invoke <action-name> --param key value --result

# Get action details
wsk action get <action-name> --summary

# View logs
wsk activation list
wsk activation logs <activation-id>

# Delete action
wsk action delete <action-name>

kubectl Commands

# Get pods
kubectl get pods -n openwhisk

# Describe pod
kubectl describe pod <pod-name> -n openwhisk

# View logs
kubectl logs <pod-name> -n openwhisk

# Get nodes
kubectl get nodes

kind Commands

Create cluster:

kind create cluster --name openwhisk --config kind-cluster.yaml

# Delete cluster
kind delete cluster --name openwhisk

# Get clusters
kind get clusters