Commit graph

984 commits

Author SHA1 Message Date
Alfredo Cardigliano
c4fd2fd0ad Remove unused parameters in getFlow. Always use zflow in processFlow for consistency and avoid unexpected bahavious. Fix type in ParsedFlowCore. 2026-10-02 16:21:25 +02:00
Manuel Ceroni
c01a1f953b
Improved host tag filtering (#10996) 2026-09-25 10:18:35 +02:00
Ivan Nardi
52070fcb32
Fix alert-pipeline drain race before e2e shutdown-when-done test scripts (#10929)
`Ntop::checkShutdownWhenDone()` ran the e2e runtime/post test scripts as soon as
`recipients_are_empty()`, which is only one stage of the flow/host alert
pipeline:

  checks -> `flowAlertsQueue/hostAlertsQueue` (per-interface)
         -> `flowAlertsDequeueLoop`/`hostAlertsDequeueLoop` -> recipients FIFO
         -> `recipients.process_notifications` (Lua) -> alert store

Right after a pcap file finishes, flow-end checks have just pushed a whole
batch of alerts into the per-interface queues, which the dequeue loops haven't
moved into the recipients yet - `recipients_are_empty()` is transiently true, so
the runtime script fired before most alerts were persisted.

- `SPSCQueue`: add `isEmpty()` (and constify `isNotEmpty()`).
- `NetworkInterface::alertsQueuesDrained()`: true when the per-interface flow and
  host alert queues are both empty.
- `Ntop::alerts_pipeline_drained()`: `alertsQueuesDrained()` on every interface,
  plus the internal C->Lua alerts queue, plus `recipients_are_empty()`.
- `checkShutdownWhenDone()`'s guard and `waitRecipientsQueuesDrained()` now poll
  `alerts_pipeline_drained()` instead of `recipients_are_empty()` alone, requiring
  two consecutive drained reads before running the runtime/post scripts.
2026-09-14 15:08:28 +02:00
Ivan Nardi
8ee8b740ea
First batch of make cppcheck clean up (#10878)
Rework the `cppcheck:` recipe so it is usable in practice:
  - `--enable=all` -> `--enable=warning,performance,portability`
    (drops the `style`/`information`/`unusedFunction` groups, which produced
    hundreds of opinion-level hits)
  - drop `--force` (was re-analyzing every file once per #ifdef combination;
    single files took 40+ min). Analyzes the default build config only.
  - add `-j $(shell nproc)` + `--cppcheck-build-dir=.cppcheck-build` for
    parallelism and incremental caching; `mkdir -p` the cache dir in the recipe
  - suppress checks that are noise for this codebase:
      * useInitializationList  - micro perf, POD members
      * duplInheritedMember    - intentional non-virtual name shadowing
        (operator[], swap, enqueue, incUses, lua, ...)
      * noCopyConstructor / noOperatorEq - the flagged classes
        (NetworkInterface, Ntop, Flow, ...) are non-copyable by design and
        always held by pointer; the fix cppcheck suggests (add operator=)
        is meaningless for them
  - .gitignore: ignore the new .cppcheck-build/ cache directory

FlowAlertsLoader::registerRisk (src/FlowAlertsLoader.cpp)
  Missing `else`: `alert_to_risk[alert_type.id] = risk` was executed even when
  `alert_type.id >= MAX_DEFINED_FLOW_ALERT_TYPE` -> out-of-bounds array write.
  The sibling registerAlert() already had the `else`.

Uninitialized ndpi_protocol structs (src/NetworkInterface.cpp x6,
src/LuaEngineInterface.cpp x1)
  Local `ndpi_protocol` values were only partially initialized (proto.* set,
  but breed/fpc/state/category/custom_category_userdata left indeterminate)
  and then passed by value to get_ndpi_full_proto_name() /
  get_ndpi_proto_category(). Added `memset(&x, 0, sizeof(x))` at each
  declaration, matching the existing convention elsewhere in the tree.

Uninitialized locals
  - src/ParserInterface.cpp: `src2dst_direction` was read on the
    `flow == NULL` path (getFlow() not called). Initialize to a defined value;
    the value is not otherwise observable on that path.
  - include/Bitmap.h (toHexString): if `buf_len <= 0` the first loop never
    runs and `strlen(buf)` reads uninitialized memory; also `strlen(buf) - 2`
    underflows (unsigned) when the string is shorter than 2 bytes. Null-
    terminate up front and hoist strlen into a `size_t`, iterate with
    `pos + 2 < len`.

src/NetworkInterface.cpp incNumHosts()/decNumHosts()
  `if (isShuttingDown() || (!host->isUnicastHost())) return;` - the second
  term is always false, the function already returned on `!isUnicastHost()`
  a few lines above. Dropped it.

include/NetworkInterface.h getActiveFlowsStats()
  Declaration and definition disagreed on the first two parameter *names*
  (`stats, status_stats` vs `ndpi_stats, stats`); types matched. Renamed the
  header params to match the implementation.

No behavioural change; each argument was already non-negative.

  - Interface id (`get_id()` returns int; kept int because 3 call sites test
    `== -1`) formatted with %u in redis-key macros -> cast the argument
    `(u_int)` at the call site:
      include/AutonomousSystem.h, include/Country.h,
      include/ObservationPoint.h, include/VLAN.h,
      src/BroadcastDomains.cpp (x2), src/HostPools.cpp, src/LocalHost.cpp,
      src/LuaEngineInterface.cpp (x3), src/NetworkInterface.cpp (x2),
      src/SyslogParserInterface.cpp
  - src/HTTPserver.cpp: `%d` with get_http_port()/get_https_port() (u_int)
    -> `%u`; the scanf-width builder `"%%%u[^|]..."` fed `int` macros -> `%d`
  - src/Flow.cpp: `"%u.%u.%u.%u"` with `int` octets from a `%d` sscanf -> `%d`;
    `%u` (arg 13) with get_protocol_category() enum -> cast `(u_int)`
  - src/MacManufacturers.cpp: `int _mac[3]` scanned with `%02x`
    (wants `unsigned int*`) -> declare `u_int _mac[3]`
  - src/StatsManager.cpp: `%lu` with `static_cast<long int>(key)` epoch values
    -> `%ld` (respects the explicit signed cast)
  - src/Trace.cpp: `"%s.%u"` with loop `int i` / `i + 1` -> `%d`
  - src/Prefs.cpp: `%u` in usage() help text fed CONST_DEFAULT_* /
    MAX_NUM_* / CLICKHOUSE_MAX_* int macros -> cast `(u_int)`

std::string / std::vector parameters taken by value, changed to `const &`
(declaration and definition kept in sync; none are virtual or overrides, so
callers are source-compatible):

  - AggregatedFlowsStats::setInfoKey
  - Flow::setL7JSON
  - CustomFlowLuaScriptAlert::setAlertMessage
  - CustomHostLuaScriptAlert::setAlertMessage, ctor (_msg)
  - ScanRealtimeAlert ctor (_alerts, std::vector)
  - CustomHostLuaScript::allocAlert (_msg)
  - OtherAlertableEntity::getAlertCachedValue / setAlertCacheValue /
    triggerAlert / releaseAlert (key, value)
  - HostAlert::init and HostAlert(HostCheckID, ...) ctor (_check_name)
  - Ntop::luaFlowCheckInfo / luaHostCheckInfo (check_name)
  - Ntop::incBlacklisHits (listname)
  - Ntop::setLuaCache (key, val)
2026-09-01 08:36:01 +02:00
Alfredo Cardigliano
06abe4b827 Disable Continuous Ping when processing pcap dump (fix #10863) 2026-08-31 13:15:15 +02:00
Luca Deri
fb87d35b3e Reworked ACLs 2026-08-06 16:23:16 +02:00
Manuel Ceroni
da22f6f85d
Added Location and Site filters to flows aggregated by host (#10708) 2026-08-06 15:06:07 +02:00
Luca Deri
978a940c58 Sites changes
Some checks are pending
Build / build (push) Waiting to run
CIFuzz / Fuzzing (address) (push) Waiting to run
CIFuzz / Fuzzing (undefined) (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
CodeQL / Analyze-1 (push) Waiting to run
2026-08-05 23:58:59 +02:00
Luca Deri
a8447474fe Changes due to ASN IPv4/IPv6 traffic breakdown 2026-07-29 21:24:26 +02:00
Matteo Biscosi
db55fae4f9 Added ACL scope filters 2026-07-21 14:20:57 +00:00
Matteo Biscosi
9bcf6ca1d7 Added interface index filter on hosts page 2026-07-20 15:40:04 +00:00
Luca Deri
c3cdb6bd70 Added timeseries drops
Some checks are pending
Build / build (push) Waiting to run
CIFuzz / Fuzzing (address) (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
CodeQL / Analyze-1 (push) Waiting to run
E2E Tests / e2e (push) Waiting to run
2026-07-18 07:51:04 +02:00
Matteo Biscosi
10a4c99b05 Added rest to return sites hierarchy (#10531) 2026-07-03 15:25:02 +00:00
Alfredo Cardigliano
be81b19bca Fix disaggregation names (probe ip is actually exporter ip) 2026-07-01 11:29:26 +02:00
Manuel Ceroni
f5802645f9
Added pagination to the macs list table (#10455) 2026-06-12 11:24:15 +02:00
Alfredo Cardigliano
77796d51f3 Add check on HR counters collectiong 2026-05-27 10:52:25 +02:00
Alfredo Cardigliano
600d99f333 Add SRC/DST_SITE_ID to historical flows
Some checks are pending
Build / build (push) Waiting to run
CIFuzz / Fuzzing (address) (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
CodeQL / Analyze-1 (push) Waiting to run
2026-05-20 18:58:06 +02:00
Luca Deri
7537427fb4 Extended SiteIds mappign support
Some checks are pending
Build / build (push) Waiting to run
CIFuzz / Fuzzing (address) (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
CodeQL / Analyze-1 (push) Waiting to run
2026-05-19 01:04:19 +02:00
Alfredo Cardigliano
e5acd55dd6 Split host tags into user_tags and computed_tags. Add transferrable_only flag to Host::getTags. Add and set Non-PQC tag. 2026-05-18 12:23:46 +02:00
Luca Deri
ee10df6a95 Added Internet eXchange to the traffix stats 2026-05-15 16:16:22 +02:00
Alfredo Cardigliano
fb521c1752 Rename labels to tags to avoid confusion as labels was too common. (note: tag filters are now flowfilters) 2026-05-12 11:43:07 +02:00
Luca Deri
e557b0a0fe Changes for implementing full IPv6 Support in Flow Exporters (#10325)
Some checks are pending
Build / build (push) Waiting to run
CIFuzz / Fuzzing (address) (push) Waiting to run
CodeQL / Analyze (push) Waiting to run
CodeQL / Analyze-1 (push) Waiting to run
2026-05-06 00:28:22 +02:00
Luca Deri
256d1cc5fb Experimental host code 2026-05-03 20:05:41 +02:00
Alfredo Cardigliano
55a61deff3 Implement labels filtering for hosts in C++ 2026-04-22 10:17:31 +02:00
Alfredo Cardigliano
e6cd18c61f Rework host labels: serialize per interface, keep a LabelsConfiguration per NetworkInterface 2026-04-17 17:03:04 +02:00
Matteo Biscosi
f076eef642 Reduced Lua VM memory allocated during 5min timeseries 2026-03-30 07:57:45 +00:00
Alfredo Cardigliano
c4abf4f49b Missing method declaration 2026-03-26 12:59:06 +01:00
Luca Deri
edee5e5cda Code format 2026-03-15 10:50:50 +01:00
Luca Deri
cd3e6bde4a Improved flow roles imolementation 2026-03-11 23:32:44 +01:00
Alfredo Cardigliano
a53b3ba3d8 Rework CH ts batch queue. Implement CHTimeseriesExporter similar to RRD/Influx. Add periodic ts export per interface (it was global running on system) 2026-03-09 16:02:48 +01:00
Alfredo Cardigliano
a9943c996f Add execSQLWrite for queries with no return rows 2026-03-05 09:37:48 +00:00
Alfredo Cardigliano
56e9225326 Add Allowed Host Pools for Users, configurable in the local user settings 2026-03-04 09:31:34 +00:00
Matteo Biscosi
65106ab211 Refactor of ASN stats page 2026-03-02 18:17:41 +00:00
Alfredo Cardigliano
b6e30f9511 Rework threads labeling 2026-02-24 16:21:16 +01:00
Luca Deri
8e26f4acc6 Code cleanup 2026-02-16 20:53:02 +01:00
Luca Deri
6581ba7e8e Cleanup 2026-02-16 19:57:29 +01:00
Luca Deri
9ee77a9d76 Reworked code 2026-01-30 11:31:37 +01:00
Luca Deri
2f3805ecab Implemented interface.aggregateSiteFlows() 2026-01-29 23:29:39 +01:00
Matteo Biscosi
55b5019cbf Added flow search 2026-01-14 15:38:27 +01:00
Luca Deri
0f8a09a1fe Added missing #ifdef 2026-01-12 19:10:34 +01:00
Luca Deri
e96ebc5a24 Deduplication code cleanup 2026-01-04 11:10:57 +01:00
Alfredo Cardigliano
0dd78b4958 Update (c) 2026-01-02 09:11:30 +01:00
Matteo Biscosi
e37a7ce219 Fixes assets import / export 2025-12-30 13:01:49 +01:00
Manuel Ceroni
9110160932
Add search in hosts page (#9878) 2025-12-22 09:58:38 +01:00
Alfredo Cardigliano
6a6e8f423c Add new state pcap_dump_processing_done 2025-12-12 12:25:40 +01:00
Matteo Biscosi
5f0a165f7d Added delimiter and null value to clickhouse stream download 2025-12-04 11:47:21 +01:00
Luca Deri
8966df2c2f Implemented export of nDPI-based protocols and categories 2025-11-10 18:35:18 +01:00
Alfredo Cardigliano
d307f33ed4 Fix compilation failures due to ndpi changes 2025-11-03 17:47:50 +01:00
Matteo Biscosi
4884d3f971 Added host aggregation to aggregated flows stats 2025-10-30 11:59:15 +01:00
Alfredo Cardigliano
9508c56471 Major rework of exporters stats support (second iteration) 2025-10-27 09:41:41 +01:00