Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions MANIFEST.in
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include README.md LICENSE
17 changes: 17 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Create a source distribution in gztar format
sources:
python setup.py sdist --formats=gztar
cp dist/*.gz .

# Install everything from build directory
install:
python setup.py install

# Create an RPM distribution
rpm:
python setup.py bdist_rpm

# Clean up temporary files from 'build' command
clean:
python setup.py clean
rm -rf *.gz dist ec2sys_autotune.egg-info build
135 changes: 131 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,134 @@
## Autotune
Amazon Linux Kernel Autotuning:
-------------------------------

EC2 Amazon Linux Kernel Autotuning
License:
--------
The code is released under GNU GENERAL PUBLIC LICENSE 2.0. See LICENSE
for details.

## License
Description:
------------
Amazon Web Services EC2 fleet hosts a wide selection of instance types
optimized to fit different customer use cases. Instance types are carved
out for varying combinations of CPU, Memory, Storage and Networking capacity
to allow customers to pick the correct instance type for their workload.

This library is licensed under the GPL 2.0 Only.
Amazon Linux Kernel provides a stable, secure and high performance execution
environment for customer applications running on Amazon EC2. Default values
for kernel tunables is not the best and optimized value for all instance
types on EC2. To provide best customer experience for Amazon Linux Kernel
consumers on EC2, this package optimizes kernel tunable for different
instance types.

Architecture:
-------------
Following are main blocks of ec2sys-autotune:

-------------
| workload |
| specific |
| generator |
-------------
^
| Extends base generator
|
------------- -------------
| base | | Instance |
| class | <----| metadata |
| generator | | properties|
------------- -------------
|
| Auto generated config file
v
-------------
| Config |
| file |
| |
-------------
|
| Consumed by engine
v
------------- ----------
| Config | | Saved |
| engine | <----| system |
| | | state |
------------- ----------

ec2sys-autotune is controlled by /bin/autotune CLI.

Base class generator: This block identifies the EC2 instance type that it
is running on and fetches all public properties of this EC2 instance type
from instance metadata file. Based on the public property of EC2 instances
and the instance type it is running on, it generates a config file that is
appropriate for running instance. As of today, this block can generate
config file to control services controlled from systemd, sysctl tunables,
sysfs tunables and CPU settings (C-state and P-state).

Config file: This is an auto generated file from base class generator.
However an end user is free to modify this config file as deemed appropriate
for ec2sys-autotune to apply the changes on the running instance.
These config files reside in /etc/ec2sys-autotune.d directory.

Config engine: This block is transparent to the type of running EC2 instance.
Engine reads the config file and applies all settings onto the running
EC2 instance. It also saves the state of the system. Upon rolling back the
tunables, engine restores the system to default state that was present
prior to tuning the system.

Workload specific generator: Any workload specific tuning need to inherit
from base class generator and generate system tunables. This can also be
used to stack or over ride tunables in base class generator.

Building:
---------
The code can be build using the Makefile script provided in the source.

1) Create a source distribution in gztar format
make sources

2) Install everything from build directory
make install

3) Create an RPM distribution
make rpm

4) Clean up temporary files from 'build' command
make clean

Usage:
------
As of this build, there are three profiles packaged:
1) base: This is general tuning of the system and the tunables
are workload agnostic.
2) udp-server: This profile also demonstrates how the framework
can be extended to add workload load specific tuning. As of now,
this profile has few tunables that are required especially in
docker or containers workload.
3) placement-group: This profile is for instances configured
with EC2 placement group for low latecy network connectivity.

Display usage of the CLI:
/usr/bin/autotune --help

Set udp-server as profile:
/usr/bin/autotune profile udp-server

Enable the tunables:
/usr/bin/autotune apply

Disable the tunables:
/usr/bin/autotune rollback

Override a tunable in autotune profile:
autotune override sysctl:vm.swappiness:40

Exclude a tunable from autotune profile:
autotune exclude sysctl:vm.swappiness

Delete customized tunable from autotune profile:
autotune delete sysctl:vm.swappiness

NOTE:
-----
autotune is currently designed to work with EC2 instances, on-prem images support
is not currently available.
23 changes: 23 additions & 0 deletions config/ec2sys-autotune.cfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[DEFAULT]
# Configuration directory. This directory holds the configuration
# tunables file. All profiles which are auto generated and user
# customization of tunables reside in this directory.
CONFIG_DIR = /etc/ec2sys-autotune.d

# State directory. This directory holds all state information
# related to autotune including the log file which is used
# to revert the system to earlier state.
STATE_DIR = /var/lib/ec2sys-autotune

# Log file to save the system state in ${STATE_DIR}. This file
# is used to revert system tunables to earlier default system
# state when autotune service is stopped. This file is protected
# by appropriate permissions from other system users.
# This file should never be modified by any user.
LOG_FILE = system-state.json

# User customized configuration file in ${CONFIG_DIR}
USER_CONFIG = user.ini

# Current service status file
STATUS = /var/run/ec2sys-autotune
34 changes: 34 additions & 0 deletions config/user.ini
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Do not edit this config file by hand, instead customize tuning in
# user.ini as per your requirements by using /bin/autotune.

# Profile to configure the system with
[profile]
PROFILE = base

# System automatically generates list of tunables based
# on PROFILE. This auto generated list can be customized
# by end user by:
# 1) over riding auto generated tunable with a different value
# Format: <tunable> = <value>
# 2) exclude the tunable from being tweaked by autotune
# Format: <tunable> = (exclude)
# Use of "(exclude)" as tunable value makes engine
# to ignore and not tweak this tunable
# An example comment is provided in each section to
# help end user read entries in this config file.

[service]
# irqbalance = stop
# irqbalance = (exclude)

[sysctl]
# vm.swappiness = 10
# vm.swappiness = (exclude)

[sysfs]
# /sys/devices/system/clocksource/clocksource0/current_clocksource = tsc
# /sys/devices/system/clocksource/clocksource0/current_clocksource = (exclude)

[cpu]
# c-state = C1
# c-state = (exclude)
55 changes: 55 additions & 0 deletions ec2sys-autotune.spec
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
Name: ec2sys-autotune
Version: 1.0.4
Release: 1%{?dist}
Summary: AWS EC2 instance autotuning

Group: Applications/Engineering
License: GPLv2
URL: https://github.com/awslabs/autotune
Source0: %{name}-%{version}.tar.gz
BuildArch: noarch
Requires(pre): systemd
Requires: kernel-tools irqbalance procps-ng
Requires: util-linux ec2-utils grep coreutils
BuildRequires: python2 python2-devel

%description
An AWS EC2 agent that tunes guest instances automatically based on their instance type

%prep
%setup -q -n %{name}-%{version}

%build
%py2_build

%install
%py2_install

%files
%defattr(-,root,root)
%doc README.md
%license LICENSE
%{_unitdir}/autotune.service
%{_bindir}/ec2sys_autotune_start
%{_bindir}/ec2sys_autotune_stop
%{_bindir}/autotune
%config(noreplace) %{_sysconfdir}/ec2sys-autotune.cfg
%config(noreplace) %{_sysconfdir}/%{name}.d/user.ini
%dir "%{_var}/lib/%{name}"
%{python2_sitelib}/*

%clean
rm -rf $RPM_BUILD_ROOT

%post
%systemd_post autotune.service

%preun
%systemd_preun autotune.service

%postun
%systemd_postun_with_restart autotune.service

%changelog
* Mon Feb 4 2019 Vallish Vaidyeshwara <vallish@amazon.com> - 1.0.0
- Initial build
Loading