vrr/openwrt-mesh-builder
2
0
Fork 0
mirror of https://github.com/karen07/openwrt-mesh-builder.git synced 2026-10-19 07:23:32 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add CIDR-squash optimization and unify direct-list updates

Browse source 
- Add CIDR-squash package download and include it in required router packages
- Optimize direct IPv4 lists with configurable CIDR over-coverage
- Reuse the shared update-ipsets.sh implementation on both routers and exit servers
- Install CIDR-squash from source during exit server deployment
- Move direct-list paths, sources, and download settings into generated runtime environments
- Keep firewall ipset loadfile paths synchronized with runtime configuration
- Validate required runtime settings in router and server scripts
- Align generated file validation and documentation with the shared direct-list workflow
This commit is contained in:
Karen Khachatryan 2026-09-27 23:30:32 +03:00
parent 941a61de35
commit 1cbb358327
15 changed files with 396 additions and 267 deletions
Download patch file Download diff file Expand all files Collapse all files

64
README.md
View file

@ -578,28 +578,28 @@ vim config.json
./upgrade_routers.py
```
Для локального просмотра структуры, без загрузки AWG пакетов и без синхронизации `packages/`, можно запускать так:
Для локального просмотра структуры, без загрузки AWG/CIDR-squash пакетов и без синхронизации `packages/`, можно запускать так:
```sh
./generate_configs.py --skip-awg-download --skip-package-sync
./generate_configs.py --skip-awg-download --skip-cidr-squash-download --skip-package-sync
```
Такой режим удобен, если custom AWG `.apk` и per-router package repos уже не нужны для текущей проверки.
Такой режим удобен, если custom AWG/CIDR-squash `.apk` и per-router package repos уже не нужны для текущей проверки.
Hooks при этом все равно запускаются, поэтому `tools/generate.py` все еще может требовать `wg` и `openssl`, если нужно создать недостающие WireGuard/OpenVPN secrets.
Для просмотра только синхронизированной template структуры без generator hooks добавляйте `--skip-hooks`:
```sh
./generate_configs.py --skip-awg-download --skip-package-sync --skip-hooks
./generate_configs.py --skip-awg-download --skip-cidr-squash-download --skip-package-sync --skip-hooks
```
Dynamic direct-list sources из `tools/default.py` должны быть доступны, если они включены. При обычной генерации `tools/generate.py` один раз локально запускает настоящий `routers/example/files/etc/scripts/update-ipsets.sh` во временном `IPSETS_DIR`. Полученные `direct-static.txt` и `direct.txt` затем одинаково записываются во все generated router и server каталоги.
Dynamic direct-list sources из `tools/default.py` должны быть доступны, если они включены. При обычной генерации `tools/generate.py` один раз локально запускает настоящий `routers/example/files/etc/scripts/update-ipsets.sh` во временном `IPSETS_DIR`. Для этого на машине генерации должен быть доступен `cidr-squash`. Полученные `direct-static.txt` и оптимизированный `direct.txt` затем одинаково записываются во все generated router и server каталоги.
Если нужен полностью локальный smoke run без запуска `update-ipsets.sh` и без загрузки country/ASN direct-list IP sets, добавьте `--skip-direct-downloads`:
```sh
./generate_configs.py --skip-awg-download --skip-package-sync --skip-direct-downloads
./generate_configs.py --skip-awg-download --skip-cidr-squash-download --skip-package-sync --skip-direct-downloads
```
В этом режиме generated `direct.txt` будет содержать только static direct entries:
@ -815,21 +815,30 @@ Direct lists собираются из нескольких источников
- публичные `listen_ip` и `exit_ip` из `config.json`
- дополнительные CIDR prefixes из `EXIT_DIRECT_STATIC_IPSETS`
Динамическая часть задается странами и ASN. Страны берутся из IPinfo Lite через `Alice39s/ipinfo-csv-lite`: daily release содержит готовые CIDR, country code, continent и ASN. Для `DIRECT_COUNTRIES` выбираются нужные IPv4 prefixes; одиночные IPv4 записи источника нормализуются в `/32`. ASN lists по-прежнему берутся из `ipverse/as-ip-blocks`. Итоговый `direct.txt` всегда хранит IPv4 в явной CIDR-нотации.
Динамическая часть задается странами и ASN. Страны берутся из IPinfo Lite через `Alice39s/ipinfo-csv-lite`: daily release содержит готовые CIDR, country code, continent и ASN. Для `DIRECT_COUNTRIES` выбираются нужные IPv4 prefixes; ASN lists по-прежнему берутся из `ipverse/as-ip-blocks`. Перед записью итоговый список передается напрямую в `cidr-squash`: он объединяет дубликаты/пересечения, сортирует результат и сам выводит одиночные IPv4 адреса как `/32`, сохраняя явную IPv4 CIDR-нотацию.
На роутерах и exit серверах эти настройки лежат в runtime env:
Runtime-параметры direct-list, которые задаются в `tools/default.py`, генерируются в `/etc/router-autoinstall.env`. В том числе туда попадают country/ASN settings, runtime paths/names, source URLs и curl timeout/retry:
```sh
DIRECT_COUNTRIES='ru cn by'
DIRECT_ASNS='32590'
RUNTIME_IPSETS_DIR='/etc/ipsets'
RUNTIME_DIRECT_STATIC_NAME='direct-static.txt'
RUNTIME_DIRECT_OUT_NAME='direct.txt'
URL_IPINFO_LITE_CSV_GZ='https://github.com/Alice39s/ipinfo-csv-lite/releases/latest/download/ipinfo-lite.csv.gz'
URL_IPVERSE_ASN='https://raw.githubusercontent.com/ipverse/as-ip-blocks/master'
UPDATE_IPSETS_CURL_CONNECT_TIMEOUT='10'
UPDATE_IPSETS_CURL_MAX_TIME='180'
UPDATE_IPSETS_CURL_RETRY='3'
```
`update-ipsets.sh` читает `/etc/ipsets/direct-static.txt`, скачивает latest `ipinfo-lite.csv.gz` во временный файл, проверяет gzip и через `grep` + `cut` выбирает IPv4 prefixes для всех `DIRECT_COUNTRIES` (включая одиночные адреса без `/32`), затем добавляет ASN lists. Перед записью `direct.txt` все одиночные IPv4 адреса нормализуются в явный `/32`. Распакованный CSV в `/tmp` не сохраняется, а gzip удаляется сразу после фильтрации. После этого скрипт атомарно обновляет `/etc/ipsets/direct.txt` и перезагружает firewall только если итоговый список изменился. Python или IPinfo libraries на OpenWrt не нужны.
`update-ipsets.sh` читает runtime env, скачивает latest `ipinfo-lite.csv.gz` во временный файл, проверяет gzip и через `grep` + `cut` выбирает IPv4 prefixes для всех `DIRECT_COUNTRIES` (включая одиночные адреса без `/32`), затем добавляет ASN lists. Собранный список без предварительной `/32`-нормализации и сортировки передается в `cidr-squash`, который канонизирует IPv4/CIDR input и пишет отсортированный CIDR output согласно `DIRECT_CIDR_OVER_COVERAGE`. Распакованный CSV в `/tmp` не сохраняется, а gzip удаляется сразу после фильтрации. После этого скрипт атомарно обновляет итоговый direct-list и перезагружает firewall только если оптимизированный список изменился. Python или IPinfo libraries на OpenWrt не нужны.
Тот же `update-ipsets.sh` запускается один раз локально во время `generate.py` с подмененным `IPSETS_DIR` и `RELOAD_FIREWALL=0`. Поэтому initial `direct-static.txt` и `direct.txt`, которые копируются во все router/server trees, строятся той же runtime реализацией, которая затем работает на OpenWrt.
Тот же `update-ipsets.sh` запускается один раз локально во время `generate.py` с подмененными только staging-параметрами `IPSETS_DIR` и `RELOAD_FIREWALL=0`; остальные runtime-настройки берутся из тех же значений `tools/default.py`. Поэтому initial direct lists и последующие обновления используют одинаковые source URLs, timeout/retry и один `DIRECT_CIDR_OVER_COVERAGE` из `tools/default.py`.
В проекте используется IPinfo Lite через `Alice39s/ipinfo-csv-lite` (CC BY-SA 4.0). IP address data powered by [IPinfo](https://ipinfo.io).
URL country source не является произвольным runtime-параметром: `update-ipsets.sh` и `awg-server.sh` ожидают именно формат Alice39s/IPinfo Lite CSV (`cidr,country_code,...`). При смене источника на другой формат URL и parser должны меняться вместе.
На exit server генератор копирует тот же файл `routers/example/files/etc/scripts/update-ipsets.sh` как `/etc/scripts/update-ipsets.sh`. `exit-direct-guard.service` сначала запускает его с `ENV_FILE=/etc/awg-server.env` и `RELOAD_FIREWALL=0`, а затем просит `awg-server.sh` только перечитать готовый `direct.txt` в `ipset` и обновить guard rules. Поэтому deploy, роутеры и серверы используют один и тот же shell-код построения direct-list.
В проекте используется IPinfo Lite через `Alice39s/ipinfo-csv-lite` (CC BY-SA 4.0). IP address data powered by [IPinfo](https://ipinfo.io). Единственный parser dynamic direct-list теперь находится в `update-ipsets.sh` и ожидает формат Alice39s/IPinfo Lite CSV (`cidr,country_code,...`), поэтому при смене источника на другой формат нужно менять только этот общий скрипт.
Трафик к direct destination не получает mark `10000`, поэтому не уходит через exit table.
@ -852,7 +861,7 @@ router direct rule -> не отправлять direct destination на exit
server guard rule -> если direct destination все же пришел на exit, drop
```
`exit-direct-guard.timer` обновляет guard ежедневно, а `awg-server-network.service` ставит guard из уже существующего списка при network-up.
`exit-direct-guard.timer` ежедневно запускает `exit-direct-guard.service`: сначала общий `/etc/scripts/update-ipsets.sh` обновляет `direct.txt`, затем `awg-server.sh guard` перезагружает `ipset` и guard rules. `awg-server-network.service` при network-up ставит guard из уже существующего списка.
## Firewall model на OpenWrt
@ -930,6 +939,7 @@ files/etc/wireguard/<access>/clients/*.conf # при WireGuard или Amnez
servers/<exit>/
etc/awg-server.sh
etc/awg-server.env
etc/scripts/update-ipsets.sh
etc/amnezia/amneziawg/*.conf
etc/babel<exit>.conf
etc/ipsets/direct-static.txt
@ -1059,7 +1069,7 @@ Bootstrap скрипт на OpenWrt при первом запуске обра
77.88.8.8
```
Достаточно ответа хотя бы от одного адреса. Если три последовательных 10-секундных периода заканчиваются без ответа от обоих адресов, роутер перезагружается. Успешный ping сбрасывает счетчик ошибок.
Достаточно ответа хотя бы от одного адреса. Если пять последовательных 10-секундных проверок заканчиваются без ответа от обоих адресов, роутер перезагружается. Успешный ping сбрасывает счетчик ошибок.
Для ручной проверки без reboot loop:
@ -1610,7 +1620,7 @@ Profile name является безопасным ASCII identifier.
- OpenVPN defaults
- DoH/DNS failover defaults
- direct-list sources
- OpenWrt/AWG package URLs
- OpenWrt/AWG/CIDR-squash package URLs
- имена managed файлов и директорий
Именно там меняются правила, которые должны быть одинаковыми для всех конфигов.
@ -1624,7 +1634,7 @@ Profile name является безопасным ASCII identifier.
```sh
./generate_configs.py
./generate_configs.py --config prod.json
./generate_configs.py --skip-awg-download --skip-package-sync
./generate_configs.py --skip-awg-download --skip-cidr-squash-download --skip-package-sync
./generate_configs.py --skip-hooks
./generate_configs.py --force
./generate_configs.py --details
@ -1635,6 +1645,7 @@ Profile name является безопасным ASCII identifier.
1. читает и валидирует `config.json`
1. создает `routers/` из `routers/example`
1. скачивает AmneziaWG `.apk`, если не указан `--skip-awg-download`
1. скачивает CIDR-squash `.apk`, если не указан `--skip-cidr-squash-download`
1. синхронизирует per-router `packages/`, если не указан `--skip-package-sync`
1. синхронизирует шаблонные файлы из `routers/example`
1. запускает `tools/generate.py`
@ -1997,12 +2008,31 @@ openssl
apk-tools 3.x
tar с поддержкой zst
make
cmake
C compiler (gcc или clang)
```
Для `apk-tools 3.x` нужен `apk`. Также используется `apk adbdump` или `apk manifest`.
Python module `cryptography` нужен для OWMB secret/material markers.
На build/deploy машине также должен быть собран и установлен `cidr-squash`, потому что
`generate_configs.py` использует его для локальной оптимизации dynamic direct-list:
```sh
git clone https://github.com/karen07/cidr-squash.git
cd cidr-squash
cmake --preset release
cmake --build --preset release
sudo cmake --install build/release
```
После установки команда `cidr-squash` должна быть доступна через `PATH`.
На exit серверах `root/deploy.sh` также автоматически собирает и устанавливает `cidr-squash`
из `https://github.com/karen07/cidr-squash`, потому что ежедневный server-side
`update-ipsets.sh` использует тот же оптимизатор, что и роутеры.
В Debian/Ubuntu это обычно пакет:
```text
@ -2081,7 +2111,7 @@ python3 -m py_compile *.py tools/*.py
Быстрая проверка template/config flow без загрузки custom packages:
```sh
./generate_configs.py --skip-awg-download --skip-package-sync --skip-direct-downloads
./generate_configs.py --skip-awg-download --skip-cidr-squash-download --skip-package-sync --skip-direct-downloads
```
Валидация generated config:

16
generate_configs.py
View file

@ -13,6 +13,8 @@ from tools.default import (
CONFIG_PATH,
AWG_PACKAGE_NAMES,
AWG_RELEASE_BASE_URL,
CIDR_SQUASH_PACKAGE_NAMES,
CIDR_SQUASH_RELEASE_BASE_URL,
MIN_OPENWRT_VERSION_TEXT,
PACKAGE_EXTENSION,
PACKAGE_SOURCE_ROOT,
@ -208,6 +210,11 @@ def main() -> None:
action="store_true",
help="do not download AWG packages from GitHub releases",
)
ap.add_argument(
"--skip-cidr-squash-download",
action="store_true",
help="do not download CIDR-squash package from GitHub releases",
)
ap.add_argument(
"--skip-package-sync",
action="store_true",
@ -267,6 +274,15 @@ def main() -> None:
"AWG2",
)
if not args.skip_cidr_squash_download:
ensure_release_packages(
cfg_data,
routers,
CIDR_SQUASH_PACKAGE_NAMES,
CIDR_SQUASH_RELEASE_BASE_URL,
"CIDR-squash",
)
if not args.skip_package_sync:
for router in routers:
sync_router_packages(

13
routers/example/files/etc/scripts/check-doh.sh
View file

@ -6,12 +6,21 @@ if [ -r /etc/router-autoinstall.env ]; then
. /etc/router-autoinstall.env
fi
DOMAIN="${CHECK_DOH_DOMAIN:-google.com}"
DOMAIN="${CHECK_DOH_DOMAIN-}"
INTERVAL="${CHECK_DOH_INTERVAL:-5}"
RESOLV="${CHECK_DOH_RESOLV:-/tmp/resolv.conf.d/resolv.conf.auto}"
RESOLV_WAIT_MAX="${CHECK_DOH_RESOLV_WAIT_MAX:-300}"
PROVIDER_DOMAINS="${CHECK_DOH_PROVIDER_DOMAINS:-ru xn--p1ai}"
if [ "${CHECK_DOH_PROVIDER_DOMAINS+x}" != "x" ]; then
echo "CHECK_DOH_PROVIDER_DOMAINS is not set in /etc/router-autoinstall.env" >&2
exit 1
fi
PROVIDER_DOMAINS="${CHECK_DOH_PROVIDER_DOMAINS-}"
if [ -z "$DOMAIN" ]; then
echo "CHECK_DOH_DOMAIN is empty in /etc/router-autoinstall.env" >&2
exit 1
fi
get_doh_endpoints_from_uci() {
i=0

39
routers/example/files/etc/scripts/exit-route.sh
View file

@ -1,18 +1,21 @@
#!/bin/sh
if [ ! -r /etc/router-autoinstall.env ]; then
echo "runtime env not readable: /etc/router-autoinstall.env"
exit 1
if [ -r /etc/router-autoinstall.env ]; then
# Generated file, loaded once at startup. Restart to apply changes.
# shellcheck disable=SC1091
. /etc/router-autoinstall.env
fi
# Generated file, loaded once at startup. Restart to apply changes.
# shellcheck disable=SC1091
. /etc/router-autoinstall.env
TABLE="${EXIT_ROUTE_TABLE:-10000}"
TABLE="${EXIT_ROUTE_TABLE-}"
INTERVAL="${EXIT_ROUTE_INTERVAL:-5}"
if [ -z "$TABLE" ]; then
echo "EXIT_ROUTE_TABLE is empty in /etc/router-autoinstall.env"
exit 1
fi
ROUTE_SECTION="exit${TABLE}"
EXIT_ROUTE_TARGETS="${EXIT_ROUTE_TARGETS-}"
if [ -z "$EXIT_ROUTE_TARGETS" ]; then
echo "EXIT_ROUTE_TARGETS is empty in /etc/router-autoinstall.env"
exit 1
@ -39,6 +42,26 @@ target_prefix() {
eval "printf '%s\n' \"\${EXIT_ROUTE_${name}_PREFIX}\""
}
validate_targets() {
name=""
prefix=""
for name in $EXIT_ROUTE_TARGETS; do
if ! valid_target_name "$name"; then
echo "bad exit target name: $name" >&2
exit 1
fi
prefix="$(target_prefix "$name")"
if [ -z "$prefix" ]; then
echo "EXIT_ROUTE_${name}_PREFIX is empty in /etc/router-autoinstall.env" >&2
exit 1
fi
done
}
validate_targets
has_babel_target() {
name="$1"
prefix=""

2
routers/example/files/etc/scripts/ping-reboot.sh
View file

@ -9,7 +9,7 @@ fi
HOSTS="${PING_REBOOT_HOSTS:-}"
BOOT_GRACE="${PING_REBOOT_BOOT_GRACE:-300}"
INTERVAL="${PING_REBOOT_INTERVAL:-10}"
MAX_FAILURES="${PING_REBOOT_MAX_FAILURES:-3}"
MAX_FAILURES="${PING_REBOOT_MAX_FAILURES:-5}"
TIMEOUT="${PING_REBOOT_TIMEOUT:-2}"
if [ -z "$HOSTS" ]; then

108
routers/example/files/etc/scripts/update-ipsets.sh
View file

@ -10,31 +10,53 @@ fi
SCRIPT_NAME="${0##*/}"
TAG="${SCRIPT_NAME%.sh}"
# Runtime path/apply knobs may be overridden by the local generator.
IPSETS_DIR="${IPSETS_DIR:-/etc/ipsets}"
# Runtime settings come from default.py through the generated ENV_FILE.
# IPSETS_DIR and RELOAD_FIREWALL remain explicit generator/server/test overrides.
RUNTIME_IPSETS_DIR="${RUNTIME_IPSETS_DIR:-/etc/ipsets}"
IPSETS_DIR="${IPSETS_DIR:-$RUNTIME_IPSETS_DIR}"
RELOAD_FIREWALL="${RELOAD_FIREWALL:-1}"
# These defaults are tied to this script's parsers/layouts. A different
# source format requires changing the parser together with the URL.
STATIC_DIRECT_NAME="direct-static.txt"
OUT_DIRECT_NAME="direct.txt"
IPINFO_LITE_CSV_GZ_URL="https://github.com/Alice39s/ipinfo-csv-lite/""\
releases/latest/download/ipinfo-lite.csv.gz"
URL_IPVERSE_ASN="https://raw.githubusercontent.com/ipverse/as-ip-blocks/master"
RUNTIME_DIRECT_STATIC_NAME="${RUNTIME_DIRECT_STATIC_NAME:-direct-static.txt}"
RUNTIME_DIRECT_OUT_NAME="${RUNTIME_DIRECT_OUT_NAME:-direct.txt}"
STATIC_DIRECT_NAME="$RUNTIME_DIRECT_STATIC_NAME"
OUT_DIRECT_NAME="$RUNTIME_DIRECT_OUT_NAME"
URL_IPINFO_LITE_CSV_GZ="${URL_IPINFO_LITE_CSV_GZ:-https://github.com/Alice39s/"\
"ipinfo-csv-lite/releases/latest/download/ipinfo-lite.csv.gz}"
URL_IPVERSE_ASN="${URL_IPVERSE_ASN:-https://raw.githubusercontent.com/ipverse/as-ip-blocks/master}"
DIRECT_CIDR_OVER_COVERAGE="${DIRECT_CIDR_OVER_COVERAGE-}"
STATIC_DIRECT="$IPSETS_DIR/$STATIC_DIRECT_NAME"
OUT_DIRECT="$IPSETS_DIR/$OUT_DIRECT_NAME"
TMP_DIRECT="${OUT_DIRECT}.tmp"
TMP_SORTED="${TMP_DIRECT}.sorted"
TMP_OPTIMIZED="${TMP_DIRECT}.optimized"
TMP_IPINFO="/tmp/ipinfo-lite.$$.csv.gz"
TMP_COUNTRIES="/tmp/ipinfo-countries.$$.txt"
DIRECT_COUNTRIES="${DIRECT_COUNTRIES:-ru cn by}"
DIRECT_ASNS="${DIRECT_ASNS:-32590}"
if [ "${DIRECT_COUNTRIES+x}" != "x" ]; then
echo "ERROR: DIRECT_COUNTRIES is not set in $ENV_FILE" >&2
exit 1
fi
if [ "${DIRECT_ASNS+x}" != "x" ]; then
echo "ERROR: DIRECT_ASNS is not set in $ENV_FILE" >&2
exit 1
fi
DIRECT_COUNTRIES="${DIRECT_COUNTRIES-}"
DIRECT_ASNS="${DIRECT_ASNS-}"
UPDATE_IPSETS_CURL_CONNECT_TIMEOUT=10
UPDATE_IPSETS_CURL_MAX_TIME=180
UPDATE_IPSETS_CURL_RETRY=3
if [ -z "$DIRECT_CIDR_OVER_COVERAGE" ]; then
echo "ERROR: DIRECT_CIDR_OVER_COVERAGE is empty in $ENV_FILE" >&2
exit 1
fi
UPDATE_IPSETS_CURL_CONNECT_TIMEOUT="${UPDATE_IPSETS_CURL_CONNECT_TIMEOUT:-10}"
UPDATE_IPSETS_CURL_MAX_TIME="${UPDATE_IPSETS_CURL_MAX_TIME:-180}"
UPDATE_IPSETS_CURL_RETRY="${UPDATE_IPSETS_CURL_RETRY:-3}"
cleanup_tmp() {
rm -f \
"$TMP_DIRECT" "$TMP_OPTIMIZED" \
"$TMP_IPINFO" "$TMP_COUNTRIES"
}
append_static_direct() {
if [ ! -s "$STATIC_DIRECT" ]; then
@ -102,7 +124,7 @@ append_country_lists() {
--max-time "$UPDATE_IPSETS_CURL_MAX_TIME" \
--retry "$UPDATE_IPSETS_CURL_RETRY" \
-o "$TMP_IPINFO" \
"$IPINFO_LITE_CSV_GZ_URL" || {
"$URL_IPINFO_LITE_CSV_GZ" || {
logger -t "$TAG" "ERROR: failed to fetch IPinfo Lite CSV"
rm -f "$TMP_IPINFO" "$TMP_COUNTRIES"
return 1
@ -147,6 +169,32 @@ append_asn_lists() {
done
}
optimize_direct() {
if ! command -v cidr-squash >/dev/null 2>&1; then
logger -t "$TAG" "ERROR: cidr-squash is not installed"
return 1
fi
before="$(wc -l <"$TMP_DIRECT")"
cidr-squash \
-q \
-p "$DIRECT_CIDR_OVER_COVERAGE" \
"$TMP_DIRECT" >"$TMP_OPTIMIZED" || {
logger -t "$TAG" "ERROR: cidr-squash optimization failed"
return 1
}
if [ ! -s "$TMP_OPTIMIZED" ]; then
logger -t "$TAG" "ERROR: cidr-squash generated an empty direct list"
return 1
fi
after="$(wc -l <"$TMP_OPTIMIZED")"
mv -f "$TMP_OPTIMIZED" "$TMP_DIRECT" || return 1
logger -t "$TAG" \
"OK: cidr-squash optimized direct list: $before -> $after entries"
}
reload_firewall_if_needed() {
[ "$1" -eq 1 ] || return 0
[ "$RELOAD_FIREWALL" = "1" ] || return 0
@ -158,55 +206,49 @@ reload_firewall_if_needed() {
}
mkdir -p "$IPSETS_DIR" || exit 1
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
cleanup_tmp
append_static_direct || {
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
cleanup_tmp
exit 1
}
append_country_lists || {
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
cleanup_tmp
logger -t "$TAG" "ERROR: failed to build direct ipset"
exit 1
}
append_asn_lists || {
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
cleanup_tmp
logger -t "$TAG" "ERROR: failed to build direct ipset"
exit 1
}
if [ ! -s "$TMP_DIRECT" ]; then
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
cleanup_tmp
logger -t "$TAG" "ERROR: generated direct ipset is empty"
exit 1
fi
sed '/\//! s#$#/32#' "$TMP_DIRECT" \
| LC_ALL=C sort -u >"$TMP_SORTED" || {
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
logger -t "$TAG" "ERROR: failed to normalize/sort direct list"
exit 1
}
mv -f "$TMP_SORTED" "$TMP_DIRECT" || {
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
logger -t "$TAG" "ERROR: failed to replace sorted tmp direct list"
optimize_direct || {
cleanup_tmp
logger -t "$TAG" "ERROR: failed to optimize direct list"
exit 1
}
changed=0
if [ ! -f "$OUT_DIRECT" ] || ! cmp -s "$TMP_DIRECT" "$OUT_DIRECT"; then
mv -f "$TMP_DIRECT" "$OUT_DIRECT" || {
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
cleanup_tmp
logger -t "$TAG" "ERROR: failed to replace $OUT_DIRECT"
exit 1
}
changed=1
else
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
rm -f "$TMP_DIRECT"
fi
cleanup_tmp
reload_firewall_if_needed "$changed"
logger -t "$TAG" "OK: direct ipset updated, changed=$changed"

210
servers/example/etc/awg-server.sh
View file

@ -11,28 +11,26 @@ IPTABLES_BIN="${IPTABLES_BIN:-/usr/sbin/iptables}"
IPSET_BIN="${IPSET_BIN:-/usr/sbin/ipset}"
SYSCTL_BIN="${SYSCTL_BIN:-/usr/sbin/sysctl}"
MODPROBE_BIN="${MODPROBE_BIN:-/usr/sbin/modprobe}"
CURL_BIN="${CURL_BIN:-/usr/bin/curl}"
AWK_BIN="${AWK_BIN:-/usr/bin/awk}"
SED_BIN="${SED_BIN:-/usr/bin/sed}"
SYSTEMCTL_BIN="${SYSTEMCTL_BIN:-/usr/bin/systemctl}"
SERVER_NAME="${SERVER_NAME:-unknown}"
SERVER_NAME="${SERVER_NAME-}"
NODE_ADDR4="${NODE_ADDR4:-}"
NODE_IP="${NODE_ADDR4%%/*}"
NODE_IFACE="${NODE_IFACE:-awg-node}"
NODE_IFACE="${NODE_IFACE-}"
LISTEN_IP="${LISTEN_IP:-}"
EXIT_IP="${EXIT_IP:-}"
IPIP_IFACE="${IPIP_IFACE:-ipip-exit}"
IPIP_ADDR4="${IPIP_ADDR4:-10.254.0.1/31}"
IPIP_IFACE="${IPIP_IFACE-}"
IPIP_ADDR4="${IPIP_ADDR4-}"
IPIP_LOCAL="${IPIP_LOCAL:-${IPIP_ADDR4%%/*}}"
IPIP_TTL="${IPIP_TTL:-64}"
IPIP_MTU="${IPIP_MTU:-}"
EXIT_SUBNET="${EXIT_SUBNET:-}"
EXIT_SUBNETS="${EXIT_SUBNETS:-$EXIT_SUBNET}"
IPSET_NAME="${IPSET_NAME:-exit_direct}"
IPSET_NAME="${IPSET_NAME-}"
NAT_CHAIN="${NAT_CHAIN:-AWG_SERVER_NAT}"
FORWARD_CHAIN="${FORWARD_CHAIN:-AWG_SERVER_FORWARD}"
ROUTE_PROBE_IP="${ROUTE_PROBE_IP:-1.1.1.1}"
ROUTE_PROBE_IP="${ROUTE_PROBE_IP-}"
AWG_SERVICES="${AWG_SERVICES:-}"
BABELD_CONF="${BABELD_CONF:-}"
@ -41,34 +39,27 @@ BABELD_PIDFILE="/run/${BABELD_CONF_NAME%.conf}.pid"
BABELD_START_TIMEOUT="${BABELD_START_TIMEOUT:-10}"
BABELD_STOP_TIMEOUT="${BABELD_STOP_TIMEOUT:-3}"
# Runtime path may be overridden only when staging/testing the script.
IPSETS_DIR="${IPSETS_DIR:-/etc/ipsets}"
# These defaults are tied to the country CSV parser and ipverse ASN layout.
STATIC_DIRECT_NAME="direct-static.txt"
OUT_DIRECT_NAME="direct.txt"
IPINFO_LITE_CSV_GZ_URL="https://github.com/Alice39s/ipinfo-csv-lite/""\
releases/latest/download/ipinfo-lite.csv.gz"
URL_IPVERSE_ASN="https://raw.githubusercontent.com/ipverse/as-ip-blocks/master"
STATIC_DIRECT="$IPSETS_DIR/$STATIC_DIRECT_NAME"
OUT_DIRECT="$IPSETS_DIR/$OUT_DIRECT_NAME"
TMP_DIRECT="${OUT_DIRECT}.tmp"
TMP_SORTED="${TMP_DIRECT}.sorted"
TMP_IPINFO="/tmp/ipinfo-lite.$$.csv.gz"
TMP_COUNTRIES="/tmp/ipinfo-countries.$$.txt"
DIRECT_COUNTRIES="${DIRECT_COUNTRIES:-ru cn by}"
DIRECT_ASNS="${DIRECT_ASNS:-32590}"
UPDATE_IPSETS_CURL_CONNECT_TIMEOUT=10
UPDATE_IPSETS_CURL_MAX_TIME=180
UPDATE_IPSETS_CURL_RETRY=3
# Runtime direct-list path comes from default.py -> awg-server.env.
# The shared /etc/scripts/update-ipsets.sh owns downloads and optimization.
RUNTIME_IPSETS_DIR="${RUNTIME_IPSETS_DIR:-/etc/ipsets}"
IPSETS_DIR="${IPSETS_DIR:-$RUNTIME_IPSETS_DIR}"
RUNTIME_DIRECT_OUT_NAME="${RUNTIME_DIRECT_OUT_NAME:-direct.txt}"
OUT_DIRECT="$IPSETS_DIR/$RUNTIME_DIRECT_OUT_NAME"
die() {
echo "ERROR: $*" >&2
exit 1
}
validate_runtime_config() {
[ -n "$SERVER_NAME" ] || die "SERVER_NAME is not set in $ENV_FILE"
[ -n "$NODE_IFACE" ] || die "NODE_IFACE is not set in $ENV_FILE"
[ -n "$IPIP_IFACE" ] || die "IPIP_IFACE is not set in $ENV_FILE"
[ -n "$IPIP_ADDR4" ] || die "IPIP_ADDR4 is not set in $ENV_FILE"
[ -n "$IPSET_NAME" ] || die "IPSET_NAME is not set in $ENV_FILE"
[ -n "$ROUTE_PROBE_IP" ] || die "ROUTE_PROBE_IP is not set in $ENV_FILE"
}
usage() {
cat >&2 <<EOF_USAGE
Usage: $0 <mode>
@ -326,154 +317,6 @@ network_down() {
echo "OK: ${SERVER_NAME} network down"
}
append_static_direct() {
if [ ! -s "$STATIC_DIRECT" ]; then
echo "ERROR: missing or empty static direct list: $STATIC_DIRECT" >&2
return 1
fi
"$SED_BIN" '/^[[:space:]]*#/d; /^[[:space:]]*$/d' \
"$STATIC_DIRECT" >>"$TMP_DIRECT"
}
append_url_list() {
url="$1"
label="$2"
data="$("$CURL_BIN" -fsSL \
--connect-timeout "$UPDATE_IPSETS_CURL_CONNECT_TIMEOUT" \
--max-time "$UPDATE_IPSETS_CURL_MAX_TIME" \
--retry "$UPDATE_IPSETS_CURL_RETRY" \
"$url")" || {
echo "ERROR: failed to fetch $label from $url" >&2
return 1
}
lines="$(
printf '%s\n' "$data" \
| "$SED_BIN" '/^[[:space:]]*#/d; /^[[:space:]]*$/d'
)"
if [ -z "$lines" ]; then
echo "ERROR: empty $label list from $url" >&2
return 1
fi
printf '%s\n' "$lines" >>"$TMP_DIRECT"
echo "OK: fetched $label from $url"
return 0
}
append_country_lists() {
[ -n "$DIRECT_COUNTRIES" ] || return 0
country_regex=""
for country in $DIRECT_COUNTRIES; do
case "$country" in
[A-Za-z][A-Za-z]) ;;
*)
echo "ERROR: bad country code: $country" >&2
return 1
;;
esac
# shellcheck disable=SC2018 disable=SC2019
country="$(printf '%s' "$country" | tr 'a-z' 'A-Z')"
if [ -n "$country_regex" ]; then
country_regex="$country_regex|"
fi
country_regex="$country_regex$country"
done
rm -f "$TMP_IPINFO" "$TMP_COUNTRIES"
"$CURL_BIN" -fsSL \
--connect-timeout "${UPDATE_IPSETS_CURL_CONNECT_TIMEOUT}" \
--max-time "${UPDATE_IPSETS_CURL_MAX_TIME}" \
--retry "${UPDATE_IPSETS_CURL_RETRY}" \
-o "$TMP_IPINFO" \
"$IPINFO_LITE_CSV_GZ_URL" || {
echo "ERROR: failed to fetch IPinfo Lite CSV" >&2
rm -f "$TMP_IPINFO" "$TMP_COUNTRIES"
return 1
}
if [ ! -s "$TMP_IPINFO" ] || ! gzip -t "$TMP_IPINFO" 2>/dev/null; then
echo "ERROR: invalid IPinfo Lite gzip archive" >&2
rm -f "$TMP_IPINFO" "$TMP_COUNTRIES"
return 1
fi
gzip -dc "$TMP_IPINFO" \
| grep -E "^[0-9.]+(/[0-9]+)?,(${country_regex})," \
| cut -d, -f1 >"$TMP_COUNTRIES"
if [ ! -s "$TMP_COUNTRIES" ]; then
echo "ERROR: no IPinfo CIDRs for countries: $DIRECT_COUNTRIES" >&2
rm -f "$TMP_IPINFO" "$TMP_COUNTRIES"
return 1
fi
cat "$TMP_COUNTRIES" >>"$TMP_DIRECT"
rm -f "$TMP_IPINFO" "$TMP_COUNTRIES"
echo "OK: added IPinfo countries: $DIRECT_COUNTRIES"
return 0
}
append_asn_lists() {
for asn in $DIRECT_ASNS; do
case "$asn" in
'' | *[!0-9]*)
echo "ERROR: bad ASN: $asn" >&2
return 1
;;
*) ;;
esac
append_url_list \
"$URL_IPVERSE_ASN/as/$asn/ipv4-aggregated.txt" \
"as:$asn" || return 1
done
}
build_direct_list() {
mkdir -p "$IPSETS_DIR" || die "failed to create $IPSETS_DIR"
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
if ! append_static_direct; then
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
exit 1
fi
if ! append_country_lists; then
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
exit 1
fi
if ! append_asn_lists; then
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
exit 1
fi
if [ ! -s "$TMP_DIRECT" ]; then
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
die "generated direct list is empty"
fi
"$SED_BIN" '/\//! s#$#/32#' "$TMP_DIRECT" \
| LC_ALL=C sort -u >"$TMP_SORTED" || {
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
die "failed to normalize/sort $TMP_DIRECT"
}
mv -f "$TMP_SORTED" "$TMP_DIRECT" || {
rm -f "$TMP_DIRECT" "$TMP_SORTED" "$TMP_IPINFO" "$TMP_COUNTRIES"
die "failed to replace sorted tmp list"
}
mv -f "$TMP_DIRECT" "$OUT_DIRECT" \
|| die "failed to update $OUT_DIRECT"
}
restore_ipset() {
[ -s "$OUT_DIRECT" ] || die "missing or empty direct list: $OUT_DIRECT"
@ -487,13 +330,6 @@ restore_ipset() {
} | "$IPSET_BIN" restore || die "ipset restore failed"
}
ipset_update() {
build_direct_list
restore_ipset
echo "OK: refreshed $IPSET_NAME ipset from $OUT_DIRECT"
}
ensure_forward_chain() {
"$IPTABLES_BIN" -N "$FORWARD_CHAIN" 2>/dev/null || true
@ -581,11 +417,13 @@ guard_existing() {
}
guard_refresh() {
ipset_update
restore_ipset
guard_rules
echo "OK: refreshed $IPSET_NAME guard from $OUT_DIRECT"
}
mode="${1:-}"
validate_runtime_config
case "$mode" in
network-up) network_up ;;
network-down) network_down ;;

3
servers/example/etc/systemd/system/exit-direct-guard.service
View file

@ -5,5 +5,8 @@ After=awg-server-network.service
[Service]
Type=oneshot
Environment=ENV_FILE=/etc/awg-server.env
Environment=RELOAD_FIREWALL=0
ExecStart=/etc/scripts/update-ipsets.sh
ExecStart=/etc/awg-server.sh guard
TimeoutStartSec=300

37
servers/example/root/deploy.sh
View file

@ -37,6 +37,10 @@ AWG_TOOLS_REPO="${AWG_TOOLS_REPO:-amnezia-vpn/amneziawg-tools}"
AWG_KERNEL_SRC="${AWG_KERNEL_SRC:-/usr/local/src/amneziawg-linux-kernel-module}"
AWG_TOOLS_SRC="${AWG_TOOLS_SRC:-/usr/local/src/amneziawg-tools}"
CIDR_SQUASH_REPO="${CIDR_SQUASH_REPO:-https://github.com/karen07/cidr-squash.git}"
CIDR_SQUASH_REF="${CIDR_SQUASH_REF:-main}"
CIDR_SQUASH_SRC="${CIDR_SQUASH_SRC:-/usr/local/src/cidr-squash}"
fail() {
echo "ERROR: $*" >&2
exit 1
@ -443,6 +447,33 @@ install_ndpi_netfilter() {
echo "OK: installed xt_ndpi commit=$NDPI_COMMIT (DHT cache disabled)"
}
install_cidr_squash() {
echo "Installing cidr-squash from ${CIDR_SQUASH_REPO} ref=${CIDR_SQUASH_REF}..."
rm -rf "$CIDR_SQUASH_SRC"
git clone \
--depth 1 \
--branch "$CIDR_SQUASH_REF" \
"$CIDR_SQUASH_REPO" \
"$CIDR_SQUASH_SRC" \
|| fail "failed to clone cidr-squash ref=$CIDR_SQUASH_REF"
cmake \
-S "$CIDR_SQUASH_SRC" \
-B "$CIDR_SQUASH_SRC/build/release" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_INSTALL_PREFIX=/usr/local \
|| fail "failed to configure cidr-squash"
cmake --build "$CIDR_SQUASH_SRC/build/release" -j"$(nproc)" \
|| fail "failed to build cidr-squash"
cmake --install "$CIDR_SQUASH_SRC/build/release" \
|| fail "failed to install cidr-squash"
command -v cidr-squash >/dev/null 2>&1 \
|| fail "cidr-squash was not installed into PATH"
echo "OK: installed cidr-squash"
}
export DEBIAN_FRONTEND=noninteractive
export NEEDRESTART_MODE=a
@ -465,6 +496,7 @@ apt-get install -y \
jq \
iptables \
build-essential \
cmake \
dkms \
bc \
libelf-dev \
@ -490,11 +522,16 @@ purge_packaged_amneziawg
install_ndpi_netfilter
install_amneziawg
install_cidr_squash
if [ -f /etc/awg-server.sh ]; then
chmod 0755 /etc/awg-server.sh
fi
if [ -f /etc/scripts/update-ipsets.sh ]; then
chmod 0755 /etc/scripts/update-ipsets.sh
fi
systemctl daemon-reload
systemctl enable iperf3

29
tools/default.py
View file

@ -133,9 +133,19 @@ REL_OPENVPN_ROOT = Path("files/etc/openvpn")
REL_OPENVPN_UCI = Path("files/etc/config/openvpn")
REL_WIREGUARD_ROOT = Path("files/etc/wireguard")
REL_DROPBEAR_AUTHORIZED_KEYS = Path("files/etc/dropbear/authorized_keys")
REL_IPSETS_ROOT = Path("files/etc/ipsets")
REL_DIRECT_IPSET = REL_IPSETS_ROOT / "direct.txt"
REL_DIRECT_STATIC_IPSET = REL_IPSETS_ROOT / "direct-static.txt"
# Runtime ipset layout. Keep these here so changing default.py updates both
# generated file locations and the values exported to runtime shell scripts.
RUNTIME_IPSETS_DIR = "/etc/ipsets"
RUNTIME_DIRECT_STATIC_NAME = "direct-static.txt"
RUNTIME_DIRECT_OUT_NAME = "direct.txt"
_RUNTIME_IPSETS_PATH = Path(RUNTIME_IPSETS_DIR)
if not _RUNTIME_IPSETS_PATH.is_absolute():
raise ValueError("RUNTIME_IPSETS_DIR must be an absolute path")
REL_IPSETS_ROOT = Path("files") / _RUNTIME_IPSETS_PATH.relative_to("/")
REL_DIRECT_IPSET = REL_IPSETS_ROOT / RUNTIME_DIRECT_OUT_NAME
REL_DIRECT_STATIC_IPSET = REL_IPSETS_ROOT / RUNTIME_DIRECT_STATIC_NAME
REL_RUNTIME_ENV = Path("files/etc/router-autoinstall.env")
RUNTIME_ENV_REMOTE_PATH = f"/{REL_RUNTIME_ENV.relative_to('files').as_posix()}"
RUNTIME_ENV_FILENAME = REL_RUNTIME_ENV.name
@ -182,6 +192,7 @@ EXPECTED_MANAGED_ROUTER_DIRS = {
AWG_SERVER_NETWORK_SERVICE_NAME = "awg-server-network.service"
SERVER_ENV_IPSET_NAME = "exit_direct"
SERVER_ROUTE_PROBE_IP = "1.1.1.1"
IPIP_SERVER_IFACE = "ipip-exit"
NODE_SERVER_IFACE = "awg-node"
@ -345,11 +356,18 @@ AWG_PACKAGE_NAMES = [
"amneziawg-tools",
"luci-proto-amneziawg",
]
CIDR_SQUASH_RELEASE_BASE_URL = (
"https://github.com/karen07/cidr-squash-openwrt-package/releases/download"
)
CIDR_SQUASH_PACKAGE_NAMES = [
"cidr-squash",
]
# OpenWrt image packages required by generated router configs and
# router example files. User-facing config.packages is for extra packages.
ROUTER_REQUIRED_PACKAGES = [
"babeld",
"cidr-squash",
"curl",
"iperf3",
"jq-full",
@ -424,6 +442,7 @@ PING_REBOOT_INTERVAL = 10
PING_REBOOT_MAX_FAILURES = 5
PING_REBOOT_TIMEOUT = 2
ROUTING_MODE_WAN = "wan"
ROUTING_MODE_SPLIT = "split"
ROUTING_MODE_EXIT = "exit"
@ -437,9 +456,7 @@ ROUTING_WAN_MARK_TEXT = "9999"
ROUTING_FIREWALL_RULE_PREFIX = "Routing-"
EXIT_ROUTE_INTERVAL = 5
RUNTIME_IPSETS_DIR = "/etc/ipsets"
RUNTIME_DIRECT_STATIC_NAME = REL_DIRECT_STATIC_IPSET.name
RUNTIME_DIRECT_OUT_NAME = REL_DIRECT_IPSET.name
DIRECT_CIDR_OVER_COVERAGE = "0.01"
UPDATE_IPSETS_CURL_CONNECT_TIMEOUT = 10
UPDATE_IPSETS_CURL_MAX_TIME = 180
UPDATE_IPSETS_CURL_RETRY = 3

36
tools/generate_firewall.py
View file

@ -295,6 +295,40 @@ def build_firewall_blocks(
return blocks
def sync_direct_ipset_loadfile(text: str, path: Path) -> str:
desired = f"{RUNTIME_IPSETS_DIR.rstrip('/')}/{RUNTIME_DIRECT_OUT_NAME}"
lines = text.splitlines(keepends=True)
starts = [i for i, line in enumerate(lines) if line.startswith("config ")]
starts.append(len(lines))
found = False
for idx in range(len(starts) - 1):
start = starts[idx]
end = starts[idx + 1]
block = lines[start:end]
if not block or block[0].strip() != "config ipset":
continue
if not any(line.strip() == "option name 'direct'" for line in block):
continue
for line_idx in range(start, end):
stripped = lines[line_idx].strip()
if not stripped.startswith("option loadfile "):
continue
indent_len = len(lines[line_idx]) - len(lines[line_idx].lstrip())
indent = lines[line_idx][:indent_len]
newline = "\n" if lines[line_idx].endswith("\n") else ""
lines[line_idx] = f"{indent}option loadfile '{desired}'{newline}"
found = True
break
if not found:
die(f"missing direct ipset loadfile option in {path}")
return "".join(lines)
def update_firewall_part(
cfg: ConfigData,
router_name: str,
@ -304,7 +338,7 @@ def update_firewall_part(
access_groups_for_router: list[AccessGroup],
) -> None:
path = router_path(cfg, router_name, "firewall")
original = read(path)
original = sync_direct_ipset_loadfile(read(path), path)
before_marker, marker_and_tail = split_text_by_marker(original, path)
validate_firewall_shared_tail(marker_and_tail, path)

39
tools/generate_runtime.py
View file

@ -69,7 +69,19 @@ def _run_update_ipsets(cfg: ConfigData, ipsets_dir: Path) -> None:
if not script.is_file():
die(f"missing update-ipsets.sh: {script}")
need("env", "sh", "curl", "gzip", "grep", "cut", "sed", "sort", "cmp", "tr")
need(
"env",
"sh",
"curl",
"gzip",
"grep",
"cut",
"sed",
"cmp",
"tr",
"wc",
"cidr-squash",
)
env_args = [
"ENV_FILE=/dev/null",
@ -77,6 +89,15 @@ def _run_update_ipsets(cfg: ConfigData, ipsets_dir: Path) -> None:
"RELOAD_FIREWALL=0",
f"DIRECT_COUNTRIES={' '.join(cfg.exit_direct.countries)}",
f"DIRECT_ASNS={' '.join(cfg.exit_direct.asns)}",
f"RUNTIME_IPSETS_DIR={RUNTIME_IPSETS_DIR}",
f"RUNTIME_DIRECT_STATIC_NAME={RUNTIME_DIRECT_STATIC_NAME}",
f"RUNTIME_DIRECT_OUT_NAME={RUNTIME_DIRECT_OUT_NAME}",
f"URL_IPINFO_LITE_CSV_GZ={URL_IPINFO_LITE_CSV_GZ}",
f"URL_IPVERSE_ASN={URL_IPVERSE_ASN}",
f"DIRECT_CIDR_OVER_COVERAGE={DIRECT_CIDR_OVER_COVERAGE}",
f"UPDATE_IPSETS_CURL_CONNECT_TIMEOUT={UPDATE_IPSETS_CURL_CONNECT_TIMEOUT}",
f"UPDATE_IPSETS_CURL_MAX_TIME={UPDATE_IPSETS_CURL_MAX_TIME}",
f"UPDATE_IPSETS_CURL_RETRY={UPDATE_IPSETS_CURL_RETRY}",
]
run_checked(["env", *env_args, "sh", str(script)])
@ -105,7 +126,7 @@ def build_direct_ipset_lines(
direct_lines = _read_generated_ipset(direct_path, RUNTIME_DIRECT_OUT_NAME)
if generated_static != static_lines:
die("update-ipsets.sh unexpectedly changed direct-static.txt")
die(f"update-ipsets.sh unexpectedly changed {RUNTIME_DIRECT_STATIC_NAME}")
print(
"OK: update-ipsets.sh generated "
@ -136,6 +157,15 @@ def build_runtime_env(cfg: ConfigData, router_name: str | None = None) -> str:
values = {
"DIRECT_COUNTRIES": " ".join(cfg.exit_direct.countries),
"DIRECT_ASNS": " ".join(cfg.exit_direct.asns),
"RUNTIME_IPSETS_DIR": RUNTIME_IPSETS_DIR,
"RUNTIME_DIRECT_STATIC_NAME": RUNTIME_DIRECT_STATIC_NAME,
"RUNTIME_DIRECT_OUT_NAME": RUNTIME_DIRECT_OUT_NAME,
"URL_IPINFO_LITE_CSV_GZ": URL_IPINFO_LITE_CSV_GZ,
"URL_IPVERSE_ASN": URL_IPVERSE_ASN,
"DIRECT_CIDR_OVER_COVERAGE": DIRECT_CIDR_OVER_COVERAGE,
"UPDATE_IPSETS_CURL_CONNECT_TIMEOUT": str(UPDATE_IPSETS_CURL_CONNECT_TIMEOUT),
"UPDATE_IPSETS_CURL_MAX_TIME": str(UPDATE_IPSETS_CURL_MAX_TIME),
"UPDATE_IPSETS_CURL_RETRY": str(UPDATE_IPSETS_CURL_RETRY),
"CHECK_DOH_DOMAIN": CHECK_DOH_DOMAIN,
"CHECK_DOH_INTERVAL": str(CHECK_DOH_INTERVAL),
"CHECK_DOH_RESOLV": CHECK_DOH_RESOLV,
@ -209,11 +239,12 @@ def write_server_ipsets(
if static_lines is None or direct_lines is None:
static_lines, direct_lines = build_direct_ipset_lines(cfg)
server_ipsets = server_exit_dir(exit_name) / RUNTIME_IPSETS_DIR.lstrip("/")
write(
server_path(exit_name, "etc", "ipsets", "direct-static.txt"),
server_ipsets / RUNTIME_DIRECT_STATIC_NAME,
"\n".join(static_lines) + "\n",
)
write(
server_path(exit_name, "etc", "ipsets", "direct.txt"),
server_ipsets / RUNTIME_DIRECT_OUT_NAME,
"\n".join(direct_lines) + "\n",
)

21
tools/generate_server_runtime.py
View file

@ -65,15 +65,25 @@ def build_server_env(
**({"IPIP_MTU": str(ipip_mtu_value())} if ipip_mtu_value() is not None else {}),
"EXIT_SUBNETS": server_exit_subnets(cfg),
"IPSET_NAME": SERVER_ENV_IPSET_NAME,
"ROUTE_PROBE_IP": SERVER_ROUTE_PROBE_IP,
"DIRECT_COUNTRIES": " ".join(cfg.exit_direct.countries),
"DIRECT_ASNS": " ".join(cfg.exit_direct.asns),
"RUNTIME_IPSETS_DIR": RUNTIME_IPSETS_DIR,
"RUNTIME_DIRECT_STATIC_NAME": RUNTIME_DIRECT_STATIC_NAME,
"RUNTIME_DIRECT_OUT_NAME": RUNTIME_DIRECT_OUT_NAME,
"URL_IPINFO_LITE_CSV_GZ": URL_IPINFO_LITE_CSV_GZ,
"URL_IPVERSE_ASN": URL_IPVERSE_ASN,
"DIRECT_CIDR_OVER_COVERAGE": DIRECT_CIDR_OVER_COVERAGE,
"UPDATE_IPSETS_CURL_CONNECT_TIMEOUT": str(UPDATE_IPSETS_CURL_CONNECT_TIMEOUT),
"UPDATE_IPSETS_CURL_MAX_TIME": str(UPDATE_IPSETS_CURL_MAX_TIME),
"UPDATE_IPSETS_CURL_RETRY": str(UPDATE_IPSETS_CURL_RETRY),
"AWG_SERVICES": awg_services,
"BABELD_CONF": server_babeld_conf_remote_path(hub.name),
}
lines = [
"# Generated by generate.py. Do not edit on the server.",
"# Runtime settings consumed by /etc/awg-server.sh.",
"# Runtime settings consumed by awg-server.sh and update-ipsets.sh.",
]
for key, value in values.items():
lines.append(f"{key}={shell_quote(value)}")
@ -104,11 +114,18 @@ def ensure_server_layout(exit_name: str) -> None:
for p in (
server_path(exit_name, "etc"),
server_amneziawg_dir(exit_name),
server_path(exit_name, "etc", "ipsets"),
server_exit_dir(exit_name) / RUNTIME_IPSETS_DIR.lstrip("/"),
server_path(exit_name, "etc", "systemd", "system"),
):
p.mkdir(parents=True, exist_ok=True)
update_src = ROUTER_EXAMPLE_DIR / "files/etc/scripts/update-ipsets.sh"
update_dst = server_path(exit_name, "etc", "scripts", "update-ipsets.sh")
if not update_src.is_file():
die(f"missing shared update-ipsets.sh: {update_src}")
update_dst.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(update_src, update_dst)
def remove_stale_server_amneziawg_confs(exit_name: str, keep_aliases: set[str]) -> None:
conf_dir = server_amneziawg_dir(exit_name)

19
tools/generated_files.py
View file

@ -16,6 +16,9 @@ try:
REL_DIRECT_STATIC_IPSET,
REL_DROPBEAR_AUTHORIZED_KEYS,
REL_RUNTIME_ENV,
RUNTIME_DIRECT_OUT_NAME,
RUNTIME_DIRECT_STATIC_NAME,
RUNTIME_IPSETS_DIR,
SERVER_TEMPLATE_DIR,
)
from .layout import (
@ -54,6 +57,9 @@ except ImportError:
REL_DIRECT_STATIC_IPSET,
REL_DROPBEAR_AUTHORIZED_KEYS,
REL_RUNTIME_ENV,
RUNTIME_DIRECT_OUT_NAME,
RUNTIME_DIRECT_STATIC_NAME,
RUNTIME_IPSETS_DIR,
SERVER_TEMPLATE_DIR,
)
from layout import ( # type: ignore
@ -189,8 +195,9 @@ def expected_server_exact_paths(cfg: ConfigData) -> set[Path]:
expected |= {
server_babeld_conf_path(hub.name),
exit_root / "etc/awg-server.env",
exit_root / "etc/ipsets/direct-static.txt",
exit_root / "etc/ipsets/direct.txt",
exit_root / "etc/scripts/update-ipsets.sh",
exit_root / RUNTIME_IPSETS_DIR.lstrip("/") / RUNTIME_DIRECT_STATIC_NAME,
exit_root / RUNTIME_IPSETS_DIR.lstrip("/") / RUNTIME_DIRECT_OUT_NAME,
exit_root / "root/.ssh/authorized_keys",
}
@ -293,8 +300,12 @@ def validate_generated_files_exist(cfg: ConfigData) -> None:
for path in (
server_babeld_conf_path(hub.name),
server_path(hub.name, "etc", "awg-server.env"),
server_path(hub.name, "etc", "ipsets", "direct-static.txt"),
server_path(hub.name, "etc", "ipsets", "direct.txt"),
server_exit_dir(hub.name)
/ RUNTIME_IPSETS_DIR.lstrip("/")
/ RUNTIME_DIRECT_STATIC_NAME,
server_exit_dir(hub.name)
/ RUNTIME_IPSETS_DIR.lstrip("/")
/ RUNTIME_DIRECT_OUT_NAME,
server_path(
hub.name,
"etc",

27
tools/validate_runtime_files.py
View file

@ -337,8 +337,20 @@ def validate_server_env(cfg: ConfigData) -> None:
),
"EXIT_SUBNETS": server_exit_subnets(cfg),
"IPSET_NAME": SERVER_ENV_IPSET_NAME,
"ROUTE_PROBE_IP": SERVER_ROUTE_PROBE_IP,
"DIRECT_COUNTRIES": " ".join(cfg.exit_direct.countries),
"DIRECT_ASNS": " ".join(cfg.exit_direct.asns),
"RUNTIME_IPSETS_DIR": RUNTIME_IPSETS_DIR,
"RUNTIME_DIRECT_STATIC_NAME": RUNTIME_DIRECT_STATIC_NAME,
"RUNTIME_DIRECT_OUT_NAME": RUNTIME_DIRECT_OUT_NAME,
"URL_IPINFO_LITE_CSV_GZ": URL_IPINFO_LITE_CSV_GZ,
"URL_IPVERSE_ASN": URL_IPVERSE_ASN,
"DIRECT_CIDR_OVER_COVERAGE": DIRECT_CIDR_OVER_COVERAGE,
"UPDATE_IPSETS_CURL_CONNECT_TIMEOUT": str(
UPDATE_IPSETS_CURL_CONNECT_TIMEOUT
),
"UPDATE_IPSETS_CURL_MAX_TIME": str(UPDATE_IPSETS_CURL_MAX_TIME),
"UPDATE_IPSETS_CURL_RETRY": str(UPDATE_IPSETS_CURL_RETRY),
"AWG_SERVICES": awg_services,
"BABELD_CONF": server_babeld_conf_remote_path(hub.name),
}
@ -368,6 +380,15 @@ def expected_runtime_env_values(
expected = {
"DIRECT_COUNTRIES": " ".join(cfg.exit_direct.countries),
"DIRECT_ASNS": " ".join(cfg.exit_direct.asns),
"RUNTIME_IPSETS_DIR": RUNTIME_IPSETS_DIR,
"RUNTIME_DIRECT_STATIC_NAME": RUNTIME_DIRECT_STATIC_NAME,
"RUNTIME_DIRECT_OUT_NAME": RUNTIME_DIRECT_OUT_NAME,
"URL_IPINFO_LITE_CSV_GZ": URL_IPINFO_LITE_CSV_GZ,
"URL_IPVERSE_ASN": URL_IPVERSE_ASN,
"DIRECT_CIDR_OVER_COVERAGE": DIRECT_CIDR_OVER_COVERAGE,
"UPDATE_IPSETS_CURL_CONNECT_TIMEOUT": str(UPDATE_IPSETS_CURL_CONNECT_TIMEOUT),
"UPDATE_IPSETS_CURL_MAX_TIME": str(UPDATE_IPSETS_CURL_MAX_TIME),
"UPDATE_IPSETS_CURL_RETRY": str(UPDATE_IPSETS_CURL_RETRY),
"CHECK_DOH_DOMAIN": CHECK_DOH_DOMAIN,
"CHECK_DOH_INTERVAL": str(CHECK_DOH_INTERVAL),
"CHECK_DOH_RESOLV": CHECK_DOH_RESOLV,
@ -436,12 +457,12 @@ def validate_ipset_files(cfg: ConfigData) -> None:
server_paths: list[Path] = []
for hub in cfg.exit_hubs:
server_ipsets = server_path(hub.name, "etc", "ipsets")
server_ipsets = server_exit_dir(hub.name) / RUNTIME_IPSETS_DIR.lstrip("/")
validate_exact_file_set(server_ipsets, ipset_filenames)
server_paths.extend(
[
server_path(hub.name, "etc", "ipsets", "direct-static.txt"),
server_path(hub.name, "etc", "ipsets", "direct.txt"),
server_ipsets / RUNTIME_DIRECT_STATIC_NAME,
server_ipsets / RUNTIME_DIRECT_OUT_NAME,
]
)