A host whose Operating System is known is now tagged with it, with one
of five new built-in tags: Windows, macOS/iOS/iPadOS, Android, Linux and
FreeBSD (tags 31 to 27). As the other tags they are shown as badges, can
be used as filters and, on the assets, can have a TTL.
Unlike the other tags, a host (and its asset) has at most one of them:
when a different OS is detected the new tag replaces the previous one.
With a TTL, the tag is removed from the asset when the OS is not
observed again in time; the OS reported for the asset is not changed.
The assets already in the inventory are tagged, at the first start, with
the OS they are stored with.
Implementation:
* The tag is not stored in the host: it is computed from its OS, as the
tags of the services are, so there is one at most and it always
matches the OS the host reports. It is not transferred to the flows.
* When the OS of a host is set or changes, the new tag is reported to
the asset as refreshed and the previous one as removed; when the same
OS is observed again the tag is refreshed, for the TTL.
* What refreshes the tag: TCP fingerprint / nDPI hint (once per TCP
flow), OS hint from nProbe, HTTP User-Agent (when it tells the OS the
host has; when it tells another one it is ignored, as before), network
discovery. DHCP sets the OS.
* The asset side (one OS tag per asset, "replaced" as reason of the
removal, tagging of the existing assets) is in the pro repository:
without it an OS change is logged as a tag removed by the user.
View interfaces behave the same.
User documentation (tags page) and developer documentation updated.
Each tag can now have a TTL, in days: a tag is removed from an asset when
nothing has confirmed it for longer than its TTL. With a TTL of 0 (the
default) the tag never expires, as before.
Example: a host answering DNS queries gets the "DNS Server" tag; with a
TTL of 30 days the tag is removed from the asset when no DNS traffic has
been seen for 30 days. A tag is refreshed every time it is observed
(service seen again, Non-PQC flow, application/flow risk bound to the
tag) or assigned by hand. Tags coming from the configured server lists
(DNS, NTP, DHCP...) never expire for the listed addresses. The TTL applies
to all the 64 tags. Deleting a user-defined tag removes it from the
assets.
The expiration of the tags requires Enterprise L (or above): without it
the tags of an asset are the tags of its host, as before.
Implementation overview (ntopng and pro repositories):
- The asset owns its tags: for each tag it holds, an asset stores when
the tag was set, when it was last refreshed and where it comes from
(observed in the traffic, set by hand, configuration).
- A host no longer tells what its tags are, but what happened to them
since its last asset dump.
- LocalHost has three more bitmaps: tags observed (refreshed), tags set
by hand (assigned), tags unset by hand (removed). A tag or a service
is recorded as refreshed every time it is observed, also when the host
already has it.
- A host with pending refreshes of tags with a TTL is dumped to the
assets (i.e. written to the Redis queue) every
CONST_ASSETS_TAGS_REFRESH_INTERVAL (6 hours) even when nothing else
changes. New tags are dumped "immediately" (as before). When no tag
has a TTL there are no extra dumps.
- [Pro part] The existing 1-minute Lua callback drains the queue and
writes to the DB (SQLite or ClickHouse), applying the changes to the
tags of the stored asset.
- [Pro part] The same callback, every 6 hours, checks the DB for expired
tags and updates the assets. A tag expires when it has not been
refreshed for longer than its TTL plus 6 hours (the time a host can
take to tell about a refresh). The tags a host has because of the
configured server lists are skipped.
- [Pro part] A user-defined tag removed from an asset, because expired
or deleted, is removed from the host too.
- [Pro part] View interfaces: the tags of the viewed flows are refreshed
on the hosts of the view at every pass of the flows walker.
- Without Enterprise L the host dumps only its current tags, that are
copied to the asset as before, and nothing expires.
- Test mode (Redis key ntopng.prefs.tags_ttl_test_mode, read at
startup): TTL in minutes and one-minute intervals.
See doc/developers/README.asset_tags_ttl.md for the design, the test
mode, what has been verified and the known limitations.
User documentation (Tags page): new "Asset Tag Expiration (TTL)" section,
the UnsafeMail example extended with a TTL of 7 days, SSH and RDP servers
that can be set by the Network Configuration, new screenshot of the tag
edit dialog.
