Save at most MAX_NUM_ASSET_SERVER_PORTS (128) server ports per L4
protocol of a local host, keeping the lowest ports, in the server ports
bitmap saved to redis and in the asset info. A warning (once per host)
reports the number of ports and the host IP when the limit is exceeded.
A host whose Operating System is known is now tagged with it, with one
of five new built-in tags: Windows, macOS/iOS/iPadOS, Android, Linux and
FreeBSD (tags 31 to 27). As the other tags they are shown as badges, can
be used as filters and, on the assets, can have a TTL.
Unlike the other tags, a host (and its asset) has at most one of them:
when a different OS is detected the new tag replaces the previous one.
With a TTL, the tag is removed from the asset when the OS is not
observed again in time; the OS reported for the asset is not changed.
The assets already in the inventory are tagged, at the first start, with
the OS they are stored with.
Implementation:
* The tag is not stored in the host: it is computed from its OS, as the
tags of the services are, so there is one at most and it always
matches the OS the host reports. It is not transferred to the flows.
* When the OS of a host is set or changes, the new tag is reported to
the asset as refreshed and the previous one as removed; when the same
OS is observed again the tag is refreshed, for the TTL.
* What refreshes the tag: TCP fingerprint / nDPI hint (once per TCP
flow), OS hint from nProbe, HTTP User-Agent (when it tells the OS the
host has; when it tells another one it is ignored, as before), network
discovery. DHCP sets the OS.
* The asset side (one OS tag per asset, "replaced" as reason of the
removal, tagging of the existing assets) is in the pro repository:
without it an OS change is logged as a tag removed by the user.
View interfaces behave the same.
User documentation (tags page) and developer documentation updated.
With a view interface, a host could be marked with a service or a tag
that belongs to its peer. The visible case is DNS: a DNS client was
marked as a DNS server (host service "DNS", "DNS Server" tag, and from
there the matching asset) when the first packet seen on its flow is a
reply, i.e. when the query has not been captured (capture started in the
middle of the exchange, packet lost, asymmetric traffic).
When the first packet of a flow is sent by the server, the flow is
recorded "reversed" (the server looks like the client) and a swap of its
peers is requested. Until the swap is performed, the services and the
tags have to be set on the peers as they will be after the swap.
On a regular interface this is what happens, as the hosts of the flow
are taken honouring the swap request. The flows of a viewed interface
have no hosts: services and tags are stored in the IP addresses of the
flow and moved to the hosts of the view later on. There the swap request
was ignored, so they ended up on the other peer.
Fix: take the IP addresses of the flow honouring the swap request, exactly as
it is done with the hosts, when a service or a tag is set on a peer of a
flow with no hosts.
Impact:
* View interfaces only. Nothing changes on the other interfaces, where
flows have hosts.
* Only flows with a pending swap request, that is flows seen "reversed":
they are uncommon. With DNS it takes a flow starting with a reply.
* It applies to all the services and tags set on the peers of a flow,
not only to DNS, as the change is in the common code. Only the DNS
case has been tested.
* This is unrelated to the DNS replies with no records fixed in #11079:
that issue never showed up on view interfaces.
On a view interface the Operating System of a host was most of the times
not learnt from the TCP fingerprint (nor from the OS hint of nDPI/nProbe)
of its flows.
The flows of a viewed interface have no hosts. The host of the view is
bound to a flow the first time the view walks it, later on, whereas the
OS is passed to the host just once, when the protocol of the flow is
detected: if the view has not walked the flow yet (the common case, as
detection takes a few packets), there is no host and the OS is dropped.
It was working only for the flows whose detection ends after the first
pass of the view.
Fix:
* A viewed flow keeps the OS of its client (from TCP fingerprint and
from the OS hint), and the view sets it on its host when it walks the
flow, once per flow.
* A view does not walk its flows in time order: the host keeps the time
of the flow its OS hint comes from, and a hint from an older flow is
ignored. This rule is applied on all interfaces.
Test: a pcap where the same host opens a TCP connection with the fingerprint
of Windows and, later, one with the fingerprint of Linux and a Windows
User-Agent
* Without a view: Linux, learnt via TCP fingerprint (before
and after).
* With a view: Windows via User-Agent before, Linux via TCP
fingerprint now.
This commit needs the matching one in the pro repository
(the call in the walker of the view).
Issue
-----
A user-defined tag bound to an application or to a flow risk is set on
the client of a flow once, when the protocol of the flow is detected:
that is the only time the flow refreshes the tag. When the tag has a
TTL, it expires on the asset if it is not refreshed.
So, with a flow living longer than the TTL of the tag (e.g. an agent
keeping one connection open for days), the tag was removed from the
asset while the traffic it comes from was still there, and it came back
only with the next flow of that kind. The host kept the tag in the
meantime.
This happened on regular interfaces only. On view interfaces the tags of
a flow are copied to the host of the view at every pass on the flows, so
they are refreshed for as long as the flow is alive.
Fix
---
On regular interfaces, once per tags refresh interval (6 hours; 1 minute
in test mode), a flow marks as refreshed on its client its own
user-defined tags that have a TTL. This is done in the periodic update
of the flow, when the current update and the previous one are not in the
same interval, so no per-flow timestamp is needed.
Impact
------
* Only the user-defined tags set by the flows (application/risk based)
that have a TTL. Nothing is done when no tag has a TTL.
* The built-in tags are not refreshed this way: regular and view
interfaces still differ for those set before the end of a flow (Non-PQC
Compliant, DHCP Server). On all interfaces, a service set at the end of
a flow (DNS, HTTP, SSH, ...) is not refreshed before the flow ends.
* Tags assigned by hand are not involved: no flow sets them.
* View interfaces are not changed.
`Host::setnDPIOS()` was overwriting the OS of the host before calling
`setOS()`: `LocalHost::setOS()`, was then finding the new OS already
set and was doing nothing.
