Onboarding Datasources over Syslog: Best Practices
Audience: Securonix Administrators, Security Engineers, SREs, Integration Owners
Overview
When you onboard datasources over syslog, how you assign ports and write filters significantly impacts data accuracy, ingestion performance, and operational maintainability. This article describes the recommended approach (one datasource per port), explains common problems when ports are shared, and documents a fallback approach when opening additional ports is not possible.
Note: Some internal guidance informed this article. Review it with your SRE or Support team before applying changes in production.
Before you start: discovery
For each datasource confirm:
- Syslog protocol version (BSD syslog vs IETF/RFC 5424).
- Transport used (UDP or TCP).
- Whether encryption (TLS) is required.
- Whether load balancing or active-passive collectors are used.
Recommended approach: one datasource per port
The preferred pattern is to send each datasource to its own dedicated syslog port on the RIN/HUB. This simple mapping reduces ambiguity, simplifies filters, and improves reliability.
Benefits
- No data cross‑contamination: Dedicated ports avoid accidental ingestion of one datasource into another and reduce filter conflicts.
- Improved performance: Filters can be simpler and more efficient when scoped to a single datasource.
- Fewer duplicates and missing events: Isolated collection paths reduce the risk that a single misconfigured filter affects multiple datasources.
Conflicts that can arise on a shared port
When multiple datasources share a single port, the following issues commonly occur:
- Duplicate events: Overlapping filters can allow the same event to match multiple datasource definitions, producing duplicates.
- Starved datasources: Overly broad or greedy filters on one datasource can consume events before other filters are evaluated, causing other datasources to receive no events.
- Higher resource use: Complex regex-based filters force every incoming event through expensive matching logic, increasing CPU and memory usage on the RIN/HUB.
If you must share a port (for example, UDP 514)
Sharing a port is supported when opening additional ports is not possible, but it requires careful, mutually exclusive filtering and stricter operational controls.
Key requirements
- Filters must be strictly mutually exclusive: each datasource should positively match only its own messages and explicitly exclude others.
- Avoid broad, greedy matches (for example, match(".*")) that can capture messages intended for other datasources.
- Prefer source-based filtering (host() / netmask()) where possible instead of content regex.
Filter optimization recommendations
- Make filters mutually exclusive: Each datasource should positively match its logs and explicitly exclude others. For example:
{ match("") and not match(""); } - Prefer source-based filters: Use host() or netmask() where applicable — these are far more efficient than regex-based matching.
- Use message() when filtering content: message() inspects the message body only and is more efficient than match() which evaluates header + content.
- Avoid excessive use of match(): match() is the most resource-expensive operator; reserve it for cases where other selectors are not possible.
- Combine negations correctly: When using not match(), combine negated clauses with AND and group them with parentheses to avoid unexpected precedence issues.
- Filter on vendor identifiers: Where available, use structured identifiers (Windows event IDs, Cisco %ASA- prefixes) instead of deep regex scanning.
- Filter at the source where possible: Configure agents or device-side views (for example, nxlog on Windows or selective logging on network devices) to reduce noise before it reaches the RIN/HUB.
- Move heavy-filter datasources: If a datasource requires heavy content scanning on a shared port, consider assigning it its own RIN or dedicated port to protect other datasources.
Validation and operations
- Capture traffic to confirm patterns: Validate actual incoming message formats and timestamps by capturing traffic on the RIN/HUB (for example, tcpdump on the appropriate port) before finalizing filters.
- Persist filters via the UI: Save filter configurations in the Securonix UI and reschedule import jobs so changes propagate into syslog-ng configuration. Avoid manual edits to syslog-ng.conf because they can be overwritten on restart.
- Verify in Spotter: After changes, confirm that duplicates are eliminated and expected events appear as searchable records.
Additional tips for TCP listeners
- syslog-ng defaults to a small number of concurrent TCP connections; increase with max-connections(n) when expecting high concurrency (for example, max-connections(1000)).
- Use flags("threaded") on high-volume TCP listeners to improve throughput.
- Plan for higher resource usage with TCP: expect roughly 15–20% more RAM and CPU than equivalent UDP ingestion.
- Prefer TCP with IETF syslog (RFC 5424) when you require reliable delivery or TLS encryption.
Summary
| Approach | Recommendation |
|---|---|
| Dedicated port per datasource | Preferred — simplifies filters, reduces risk, and improves performance |
| Shared port (for example UDP 514) | Supported only with strictly mutually exclusive, optimized filters and strong validation |
