Cisco is the Official Safety Cloud Supplier for the Black Hat Community Operations Heart (NOC). We work with the opposite official companions to carry the {hardware}, software program and engineers to construct and safe the community, for our joint buyer: Black Hat.
- Arista: Wired and Wi-fi Community Gear
- Corelight: Open Community Detection and Response
- Palo Alto Networks: Community Safety and SOC Platform

This was our 8th yr supporting Black Hat Europe and the first mission within the NOC is community resilience. The companions additionally present built-in safety, visibility and automation, a Safety Operations Heart (SOC) contained in the NOC.
When the companions deploy to every occasion, we arrange a world class community and safety operations heart in a number of days. Our purpose stays community up time and creating higher built-in visibility and automation. Black Hat has the decide of the safety trade instruments and no firm can sponsor/purchase their manner into the NOC. It’s invitation solely, with the intention of range in companions, and an expectation of full collaboration. As a NOC staff comprised of many applied sciences and corporations, we’re constantly innovating and integrating, to supply an total cybersecurity structure resolution.

Exterior the NOC companion dashboards have been displayed for the attendees to view the quantity and safety of the community visitors.

The function of Cisco within the Black Hat NOC continues to evolve since we have been invited to companion in 2016. Black Hat has limitless entry to the Cisco Safety Cloud and its capabilities. Working with the NOC leaders (Neil “Grifter” Wyler & Bart Stump) and the chief architect (Steve Fink), we examined, deployed and built-in the next applied sciences:
ThousandEyes: Community visibility
The NOC leaders allowed Cisco (and the opposite NOC companions) to usher in extra software program to make our inside work extra environment friendly and have better visibility; nonetheless, Cisco just isn’t the official supplier for Prolonged Detection and Response (XDR), Safety Incident and Occasion Administration (SIEM), Community Detection and Response (NDR), Safety Operations and Automated Response (SOAR) or collaboration.
To raised help Black Hat, we additionally carried out:
- Cisco XDR: Menace Looking / Menace Intelligence Enrichment / Analyst dashboards / Automation with Webex
- Splunk Enterprise Safety Cloud: platform for Cisco Safety Cloud information sharing, and with ThousandEyes, Palo Alto Networks and Corelight integrations with Cisco XDR; additionally government dashboards
- Splunk Assault Analyzer: Built-in with Safe Malware Analytics
- Cisco Webex: Incident notification and staff collaboration
Introducing Cisco Duo and Identification Intelligence, by Ryan Maclennan
Cisco Duo is a brand new addition to the Black Hat NOC. We began with a Proof-of-Idea (PoC) in Black Hat Asia 2024 and turned it right into a full deployment at Black Hat Europe. With this deployment, our purpose was to create an atmosphere the place every companion would have a single sign-on (SSO) consumer to log into every product supplied by a companion. We might create teams for every consumer, which mapped to being an analyst, administrator or an approver function.
For instance, if we wished to make use of Palo Alto Networks (PANW) XSIAM product, we might log in with our consumer, however they’d solely be an analyst and couldn’t make adjustments on the platform. Nevertheless, if a PANW admin logged in, they might make adjustments as wanted. This was vice versa for them as effectively, the PANW admins could be analysts inside our Cisco merchandise, however we might make adjustments as needed on our personal merchandise, in coordination and approval of NOC Leaders.
We have been capable of combine Duo SSO into the next companion merchandise:
- PANW XSIAM
- PANW NGFW
- PANW Cortex
- PANW Panorama
- Corelight Investigator
- Arista Cloud Imaginative and prescient
Most of those integrations have been for on-prem merchandise (not publicly out there) and some have been cloud-based, displaying that we’re capable of defend an software whether or not it’s publicly out there or non-public. The Cisco merchandise already had an SSO structure with our company accounts and we are going to transition to the Black Hat SSO infrastructure for Asia 2025.
After getting all of the Duo functions setup, we have been capable of begin getting authentication requests into Duo:

Beneath, you’ll be able to see all of the functions we created to combine Duo SSO.

After the functions have been configured and the customers enrolled in Duo, we have been capable of begin utilizing the brand new Cisco Identification Intelligence, from inside Duo.
Cisco Identification Intelligence
Cisco Identification intelligence (CII) is an AI-powered resolution that bridges the hole between authentication and entry. It permits us to usher in a number of authentication supply logs right into a single entity after which analyze them to find out if a consumer is reliable. CII will give a consumer a belief rating based mostly on geographic location, login instances, Working System (OS), system varieties, variety of login makes an attempt, appropriate and incorrect logins, system belief and lots of extra criterion. CII takes all these indicators into consideration after which makes belief ranges for every consumer. You may see our belief rating unfold within the beneath screenshot:

You may see within the screenshot above that there was an untrusted consumer, three impartial, and 9 trusted customers. Most of the impartial customers have been as a result of CII didn’t have sufficient information to baseline the consumer but and was nonetheless figuring out the way it ought to classify them. The one untrusted consumer was me; as a result of the consumer I used to manage Duo and CII was the identical that I used login with to all the opposite functions.
Earlier than the London based mostly convention, I used to be administering Duo and CII in the USA. I then used a VPN a number of instances whereas in Europe, so my geography was shortly altering. These occasions contributed to my ‘Untrusted’ standing, worthy of investigation.
Beneath, we are able to see the dashboard view of CII, with the fast view of knowledge which an administrator could also be excited by seeing.

Within the above screenshot, we are able to see the month-to-month sign-ins and whether or not they have been profitable or not. Additionally, the kind of Multifactor Authentication (MFA) utilized by customers, delicate functions and the nations the place logins have been tried from.
Because the Black Hat convention international circuit continues, I’m excited to see the place we are able to take CII and use its information to higher safe our NOC companion merchandise.
Dynamic Malware Evaluation, by Ryan Maclennan
For Cisco, a core built-in operate within the Black Hat NOC/SOC is offering the platform for our companions to ship suspicious information to Safe Malware Analytics (aka Menace Grid) for dynamic malware evaluation (aka sandboxing). We’ve expanded the combination through the years, with each Corelight OpenNDR and Palo Alto Networks Firewalls submitting samples. At Black Hat Europe 2024, over 12,000 supported samples have been submitted.

The risk hunters additionally used Safe Malware Analytics to research suspicious URLs and information, with out the danger of an infection. Many of the convictions have been on URLs submitted by the NOC analysts.
At every convention, we see examples of private figuring out data despatched over the community within the clear. One which stood out was a school scholar’s transcript in clear textual content. That is what occurs while you use http on port 80 for communications (as an alternative of https). The next particulars of the scholar have been clearly out there from the contents downloaded from the self-hosted area:
- Title
- Date Of Start
- Social Safety Quantity
- Faculty attended and when
…and that’s all it’s essential to craft an id theft and/or phishing assault on the unassuming scholar. All the time confirm your connection safety!

Splunk Assault Analyzer
As a PoC at Black Hat USA, we deployed Splunk Assault Analyzer (SAA) as one other malware sandboxing software. This was a brand new integration created with the assistance of the Corelight staff and was made on the spot. This time round in Europe, we have been capable of allow all of SAA’s capabilities and despatched all information to it to match with Safe Malware Analytics. Right here is dashboard abstract of the information analyzed by SAA:

Taking a look at this we are able to see the entire quantity of information analyzed by SAA and what was convicted as malicious. Of the convictions we bought, we discovered that two have been Phish Kits.

You might have seen that Safe Malware Analytics analyzed 1000’s extra information than SAA. It is because we began to hit a price restrict, and our SAA occasion didn’t catch it in time. For the following convention, we can be working with Corelight to make the combination extra sturdy to deal with the speed limiting effectively.
In case you missed it, SAA now has Safe Malware Analytics (SMA) as an engine. This implies, while you hyperlink your SMA account to SAA, SAA will now ship information to be analyzed by SMA as effectively and use its dedication as a part of its personal scoring.
Prolonged Detection and Automation, by Ivan Berlinson and Aditya Raghavan
The Cisco XDR Command Heart dashboard tiles made it simple to see the standing of every of the linked Cisco Safe applied sciences, and the automation workflows iterations over the week.

Beneath are the Cisco XDR integrations for Black Hat Europe, empowering our risk hunters to research Indicators of Compromise (IOC) in a short time, with one search.
We respect alphaMountain.ai, Pulsedive and Recorded Future donating full licenses to Cisco, to be used within the Black Hat Europe 2024 NOC.
The view within the XDR Integrations consumer interface:


Unleashing the Energy of Cisco XDR Automate at Black Hat Europe
With the ever-evolving technological panorama, automation stands as a cornerstone in reaching XDR outcomes. It’s certainly a testomony to the prowess of Cisco XDR that it boasts a totally built-in, sturdy automation engine.
Cisco XDR Automation embodies a user-friendly, no-to-low code platform with a drag-and-drop workflow editor. This modern characteristic empowers your SOC to hurry up its investigative and response capabilities. You may faucet into this potential by importing workflows inside the XDR Automate Change from Cisco, or by flexing your artistic muscle tissues and crafting your personal.
Keep in mind from our previous blogs, we used automation for incident notifications into Webex, in addition to ‘
’ in XDR for Umbrella class blocks. Each these workflows have been spruced up and used extensively at Black Hat Europe 2024. We now see the final replace timestamp proper within the incident title itself and the Webex message, which vastly simplifies the understanding of a detection for our risk hunters.Creating an Incident

The next automation workflows have been constructed particularly for Black Hat use instances:
- Cisco SMA Malicious submission – XDR incident and notification
- Cisco SMA – Monitor Non-Malicious paperwork submission
- Palo Alto Networks Firewall – Create Cisco XDR incident – V2
- Splunk – Corelight – Create Cisco XDR incident V2
- Splunk – ThousandEyes – Create XDR create incident V2
- Incident Enrichment – Add Room and Objective
- Palo Alto Menace Logs to Splunk
In addition to #1 and #3, the remainder of these workflows have been premiered at Black Hat Europe 2024, due to the work and inspiration of Ivan.

Splunk Enterprise Safety Cloud, by Ivan Berlinson, Aditya Raghavan and Ryan Maclennan
To make our risk hunters’ lives richer with extra context from ours and our companions’ instruments, we introduced in Splunk Enterprise Safety Cloud at this Black Hat occasion to ingest detections from Cisco XDR, Safe Malware Analytics, Umbrella, ThousandEyes, Corelight and Palo Alto Networks and visualize them into practical dashboards for government reporting. The Splunk Cloud occasion was configured with the next integrations:
- Cisco XDR and Cisco Safe Malware Analytics, utilizing the Cisco Safety Cloud app
- Cisco Umbrella, utilizing the Cisco Cloud Safety App for Splunk
- ThousandEyes, utilizing the Splunk HTTP Occasion Collector (HEC)
- Corelight, utilizing Splunk HTTP Occasion Collector (HEC)
- Palo Alto Networks, utilizing the Splunk HTTP Occasion Collector (HEC)
The ingested information for every built-in platform was deposited into their respective indexes. That made information searches for our risk hunters cleaner. Looking for information is the place Splunk shines! You start by merely navigating to
and typing your search question. You do must know the Splunk Question Language (SQL) to construct your queries however that’s only a fast tutorial away. Apps > Search and Reporting
We discovered our manner via trying on the information and iterating. An instance of a easy seek for acquiring the rely of all alerts from the Suricata engine of Corelight logs is beneath.

The
tab lets you shortly convert this information into a visible format for previewing. And now, off we went to construct search queries throughout all of the datasets we ingested. These search queries have been then aggregated and visualized into an government view utilizing Splunk Dashboard Studio. Since we ended up with extra widgets than can slot in a single Government display screen, we utilized the tabbed dashboard characteristic. The next two screenshots present the ultimate dashboards together with callouts for the sources of the varied widgets. Visualization


With the constitution for us at Black Hat being a ‘SOC inside a NOC’, the manager dashboards have been reflective of bringing networking and safety reporting collectively. That is fairly highly effective and can be expanded in future Black Hat occasions, so as to add extra performance and develop its utilization as one of many main consoles for our risk hunters in addition to reporting dashboards on the massive screens within the NOC.
Menace Hunters’ Story, by Ivan Berlinson
In the course of the Black Hat occasion, the NOC opens early earlier than the occasion Registration and closes after the trainings and briefings full for the day. Which means that each risk hunter’s place should be lined by bodily, uninterrupted presence for about 11 hours per day. Even with the utmost dedication to your function, generally you want a break, and a brand new potential incident doesn’t wait till you’ve completed the earlier one.
Aditya and I shared the obligations as Menace Hunters staffing the Cisco XDR, Malware Analytics and Splunk Cloud consoles, alternating between morning and afternoon shifts. Although in actuality each of us stayed on many of the day as we had a lot enjoyable writing automation workflows and constructing dashboards, moreover finishing up our main obligations.

An instance of a few of these workflows in motion collectively helped us search out a possible case of cryptomining within the NOC itself, through the early hours of Dec 12! Due to the Corelight and PANW firewalls integrations in XDR, we had ourselves a singular correlated incident, with detections from each companions.

The workflows I constructed embody a verify to search out any open incidents with the property and/or observables in query, to append the detection below course of. If there’s none, it might create a brand new incident. As we are able to see, the detection from Corelight got here in at 09:40 GMT, adopted by the detection from PANW firewalls a couple of minutes later.


As new detections have been getting appended into the incident, I shortly up to date the automation workflows to incorporate a timestamp indicating the final seen sighting for that incident proper within the title. Whereas this won’t be what you’ll do in a manufacturing atmosphere, it vastly simplifies the power for our risk hunters to investigate all of the incidents as they arrive in.

As I investigated the incident, I uncovered one other detection that had are available in simply earlier than, this time the supply of the detection was Umbrella and virtually went below the radar as a result of it bearing a decrease precedence rating. This incident got here in via the automation workflow used previously years. This supplied the affirmation of the cryptomining exercise on the endpoint.

Subsequent query – who is that this 10.X.X.X system? And will they be cryptomining at Black Hat? Thanks to a different automation workflow, with a click on of a
motion of the Incident Response playbook we had attribution with bodily room identify and placement inside the occasion heart from a pre-defined database for the recognized asset within the incident. Response


Lo and behold – the asset was in Degree 3, Capital Suite, Room 1 linked to the NOC Wi-Fi; proper in the identical room as me! I had constructed one other automation workflow that brings in Corelight and PANW firewall risk detections into Splunk Cloud, via which we have been capable of monitor down the system within the room to a MacBook and an MAC deal with.

Time to faucet on somebody’s shoulder.
Community Visibility with ThousandEyes, by Jessica Santos, MD Foysol Ferdous, Ryan MacLennan
Black Hat Europe 2024 is the sixth consecutive occasion with a ThousandEyes (TE) deployment. We unfold that visibility throughout core switching, Registration, the Enterprise Corridor, two- and four-day coaching rooms, and Keynote areas. Beneath is A few of the {hardware} Black Hat bought for the ThousandEyes brokers.

We labored with Michael Spicer on the situation of the agent deployment to make sure consultant protection and the categories / frequency of scheduled testing.

Optimizing Community Monitoring with Thousand Eyes
We had a dashboard within the NOC, so the leaders and architect might see points in actual time, and ThousandEyes widgets within the Splunk government dashboard, as seen earlier within the weblog.

At Black Hat Europe 2024, we had an issue the place the ThousandEyes brokers have been displaying a excessive latency time to Azure. We have been receiving calls about entry to Azure being sluggish, however being the proactive NOC we’re, we went forward and investigated what’s inflicting the excessive response time.
We investigated the Azure Community path that’s recorded by ThousandEyes and located there are three locations the Azure standing portal makes use of.

Two of these locations are exterior of the UK: one in the USA and one in Japan. In the event you look within the screenshot above, you’ll be able to see a single pink hyperlink and it may be used for both the US or Japan Azure standing portal. That is the most certainly trigger for the elevated response time we have been seeing moreover the geographic distance. Seeing this, we SSH’ed into one the ThousandEyes brokers and used the HTTP’ing software to do the same check to Azure. Once we ran the check to the Azure portal standing web page, we might see some regular response instances after which many latent response instances. This matched as much as what ThousandEyes reported. This led us to the conclusion that the Azure standing portal workload balances however doesn’t do geographic load balancing.
With this information, we determined to laborious code the IP of the UK server into the ThousandEyes check to higher characterize how attendees will entry Azure.
ThousandEyes is a really highly effective software, and it is ready to decide whether or not the problem resides contained in the community or exterior the place we can not management it. Beneath is a screenshot of what number of totally different community paths can take to a single useful resource. This reveals the significance of with the ability to pinpoint precisely the place a problem is happening.

Meraki Programs Supervisor, by Paul Fidler and Connor Loughlin
Our fourth yr of deploying Meraki Programs Supervisor at Black Hat Europe, because the official Cell Gadgets Administration platform, went very easily. We launched a brand new caching operation to replace iOS units on the native community, for velocity and effectivity. Going into the occasion, we deliberate for the next variety of units and functions:
- iPhone Lead Scanning Gadgets: 68
- iPads for Registration: 9
- iPads for Session Scanning: 12
- Variety of units deliberate in whole: 89
We registered the units upfront of the occasion. Upon arrival, we turned every system on.

The Wi-Fi profile that we’d like for the Black Hat iOS units was not put in. Nevertheless, I introduced a Meraki Z3C, with mobile and Wi-Fi functionality. I’d introduced this as a result of it usually takes a few days to get Wi-Fi down in Registration, the place we arrange the units earlier than deployment. So, inside actually 15 seconds, I’d spun up a brand new SSID, prefixed it with a full cease in order that it appeared on the high of the out there Wi-Fi networks, and earlier than the primary iOS system had powered on, the Z3C was broadcasting this. So, with only a handful of seconds toil on every system, we’d bought them connecting again to Meraki Dashboard to get the proper Wi-Fi profile.

Extra ache: Location companies
Location companies is a ache level for cell system administration. Firstly, you could be sure that
is NOT skipped on the time of system supervision utilizing Apple Configurator. Which means that it’s an additional step on the time of enrollment (that means half a second of toil), fairly than having to open settings, scroll right down to Location
, then Privateness
, then flick the toggle. Sadly, this was skipped for the occasion preparation by the contractor, so we needed to allow manually this for every system. Location
Location of units is necessary for theft retrieval or if the system is misplaced. Location is enabled by opening the System Supervisor app and tapping
, then Location
, then “Allow
.” You may then faucet it once more and click on “While utilizing the applying
” which permits for location when the app is within the background. Due to (and that is the place you may find yourself in a heated dialogue) Apple’s stance on privateness, it’s such a disgrace that this will’t be managed. All the time permit
Beneath is a fast screenshot of the Z3C shopper dashboard after an hour of the units being turned on.

Attention-grabbing to see the place the system is asking out to, within the screenshot beneath.

Software Updates
Having functions up to date in the midst of an occasion can have disastrous penalties. While there isn’t an total setting or restriction to forestall this, it’s doable to do it on the software layer. And, after all, Meraki Programs Supervisor allowed us to do that for all apps on the similar time.

OS Updates
I believe this goes with out saying, however the capability to remotely replace units within the occasion of an pressing vulnerability repair is invaluable, like we had final yr in Las Vegas with Apple.

Firewall Guidelines
The safety of the Registration community is paramount as there’s Private Figuring out Data information on this community. So, we’ve some fairly strict inbound and outbound guidelines.
Managing Apple units requires that 17.0.0.0/8 be stored open for port 80 and 443 visitors. Moreover, Meraki lets you obtain a dynamically created checklist of servers that it must be open to in order that endpoints might be managed. However, as we’re utilizing Cisco Umbrella and AMP (Safe Endpoint), there’s an entire host of different endpoints that should be opened. These are listed right here.
Content material Caching
One of many largest issues affecting the iOS units at previous Black Hat occasions was the speedy must each replace the iOS system’s OS as a result of a patch to repair a zero-day vulnerability and to replace the Black Hat iOS app on the units. On the USA occasions, there are a whole bunch of units, so this was a problem for every to obtain and set up. So, I took the initiative into trying into Apple’s Content material Caching service constructed into macOS.
Now, simply to be clear, this wasn’t caching EVERYTHING… Simply Apple App retailer updates and OS updates.

That is turned on withing
and begins working instantly. System Setting
I’m not going to get into the weeds of setting this up, as a result of there’s a lot to plan for. However I’d counsel that you just begin right here. The setting I did change was:

I checked to see that we had one level of egress from Black Hat to the Web. Apple doesn’t go into an excessive amount of element as to how this all works, however I’m assuming that the caching server registers with Apple and when units verify in for App retailer / OS replace queries, they’re then advised the place to look on the community for the caching server.
Instantly after turning this on, you’ll be able to see the default settings and metrics:
% AssetCacheManagerUtil settings
Content material caching settings:
AllowPersonalCaching: true
AllowSharedCaching: true
AllowTetheredCaching: true
CacheLimit: 150 GB
DataPath: /Library/Software Help/Apple/AssetCache/Information
ListenRangesOnly: false
LocalSubnetsOnly: true
ParentSelectionPolicy: round-robin
PeerLocalSubnetsOnly: true
And after having this run for a while:
% AssetCacheManagerUtil settings
Content material caching standing
Activated: true
Lively: true
ActualCacheUsed: 528.2 MB
CacheDetails: (1)
Different: 528.2 MB
CacheFree: 149.47 GB
CacheLimit: 150 GB
CacheStatus: OK
CacheUsed: 528.2 MB
MaxCachePressureLast1Hour: 0%
Dad and mom: (none)
Friends: (none)
PersonalCacheFree: 150 GB
PersonalCacheLimit: 150 GB
PersonalCacheUsed: Zero KB
Port: 49180
PrivateAddresses: (1)
x.x.x.x
PublicAddress: x.x.x.x
RegistrationStatus: 1
RestrictedMedia: false
ServerGUID: xxxxxxxxxxxxxxxxxx
StartupStatus: OK
TetheratorStatus: 1
TotalBytesAreSince: 2023-12-01 13:35:10
TotalBytesDropped: Zero KB
TotalBytesImported: Zero KB
TotalBytesReturnedToClients: 528.2 MB
TotalBytesStoredFromOrigin: 528.2 MB
Now, helpfully, Apple additionally pop this information periodically right into a database situated at:
in a desk known as ZMETRICS. Library/Software Help/Apple/AssetCache/Metrics/Metrics.db

That is additionally out there in
. Exercise Monitor
And with a small variety of units, you’ll be able to see how shortly the server begins decreasing the impression on the WAN. Now, given the above, getting the info from the command line sometimes is painful, particularly within the format that it’s introduced.
Apple, helpfully, permit to append a –j to the tip of the standing command to current the knowledge in JSON:
{"identify":"standing","end result":{"Activated":true,"Lively":true,"ActualCacheUsed":2327774501,"CacheDetails":{"iCloud":109949295,"iOS Software program":20800617,"Mac Software program":11984379,"Different":2226505758},"CacheFree":247630759951,"CacheLimit":250000000000,"CacheStatus":"OK","CacheUsed":2369240049,"MaxCachePressureLast1Hour":0,"Dad and mom":[],"Friends":[],"PersonalCacheFree":249890050705,"PersonalCacheLimit":250000000000,"PersonalCacheUsed":109949295,"Port":49181,"PrivateAddresses":["10.10.10.10"],"PublicAddress":"X.X.X.X","RegistrationStatus":1,"RestrictedMedia":false,"ServerGUID":"FDE578EE-XXXX-XXXX-XXXX-102B60869501","StartupStatus":"OK","TetheratorStatus":0,"TotalBytesAreSince":"2024-12-12 11:58:04 +0000","TotalBytesDropped":0,"TotalBytesImported":0,"TotalBytesReturnedToChildren":0,"TotalBytesReturnedToClients":11482694,"TotalBytesReturnedToPeers":0,"TotalBytesStoredFromOrigin":1164874,"TotalBytesStoredFromParents":0,"TotalBytesStoredFromPeers":0}}
ThousandEyes Agent for the Caching Server
Provided that we’ve an Apple MacMini on the Registration community, it was a easy determination to put in the ThousandEyes macOS agent on it systematically utilizing Meraki Programs Supervisor.

This may be downloaded from
.Endpoint Brokers > Agent Settings > Add new Endpoint Agent
Nevertheless, as I discovered to my detriment, there’s not but a Common installer, so be sure you get your processor structure proper (ARM vs x86!)
In Meraki Programs Supervisor, we configure the app like this:

Now, we talked earlier about firewall settings. A System Supervisor customized app might be hosted in two methods:
- Hosted by yourself Infra or
- Meraki will host it
In the event you’re selecting the latter, simply be aware that Meraki really hosts it on AWS. Particulars right here.
So, just be sure you have the precise AWS occasion open in your firewalls, or host packages your self.
Checking with the PANW firewall staff, we decided the caching server saved 5% of the visitors for the week, releasing up bandwidth for coaching and demos. For Black Hat Asia 2025, we plan to discover learn how to host Home windows Updates, a big shopper of bandwidth, on the primary day of coaching and briefings.
Maintaining with Encrypted DNS, by Christian Clasen and Justin Murphy
For the previous couple of years, we’ve been using the PANW edge firewalls to redirect outbound DNS queries in the direction of our inside resolvers. This closed a spot in coverage and visibility that existed for attendees on the Black Hat occasion. As evidenced by the DNS Statistics charts later within the weblog, the technique paid off with a noticeable bounce in noticed queries. However because it so typically goes in know-how (and safety in particular), these kinds of methods can result in an arms race of types.
In those self same years, browser and working system builders have expanded the deployment of encrypted DNS protocols. Along with wrapping DNS in uncooked TLS and HTTPS, extra unique applied sciences at the moment are within the combine. Chief amongst them is Apple’s implementation of Oblivious DNS over HTTPS (ODoT). The aim of ODoT is to forestall the snooping of DNS queries –not simply on the native LAN, but additionally by the DNS suppliers themselves. I supplied an outline of this know-how in my “Historical past of DNS Safety” Cisco Stay speak.
The gist of the ODoH is as follows:
- The “first hop” recursive DNS resolver accepts receives the shopper lookup, however the shopper has completed one thing sneaky to forestall this preliminary resolver from realizing what the area identify is that the shopper is searching for. It has wrapped the unique question in an encrypted blob and added a bogus identify to the “outer” message.
- When the recursive resolver sends the question upstream to the authoritative identify server for the “bogus” area, the message might be decrypted as a result of that identify server is an ODoH-aware server that expects this encrypted message!
- The server sees the question data and may recurse the DNS for the reply as regular however isn’t made conscious of the unique shopper IP…it is just servicing the primary recursive resolver as its shopper.
This separation of duties ensures that, within the absence of collusion between the primary and second DNS suppliers, a shopper and its queries can by no means be correlated, and helpful monitoring is rendered not possible.

Apple implements this structure in its Non-public Relay characteristic. Along with all of the privateness options detailed above, Non-public Relay makes use of QUIC to move packets to Apple making the communication much more opaque to community operators like us within the Black Hat NOC. The broad deployment of Non-public Relay has led to a drop in DNS queries evaluated and logged by Umbrella.
We introduced our observations and suggestion to the NOC Leaders, who determined it might be finest to try to block these protocols (DoT, DoH and Non-public Relay) for higher visibility. The morning of final day, we added the coverage to Umbrella.

We instantly noticed blocks for domains related to Apple’s MASQUE proxies within the exercise search, in addition to these utilized by Android telephones for DoT. The tip consumer expertise was not impacted.

Over 129k of those blocks occurred from 11:05am to the shutdown at 6pm on the final day, 12 December.

We’ll proceed this coverage going ahead at different Black Hat occasions and monitor the statistics as traditional.
DNS Statistics, by Christian Clasen and Justin Murphy
We will see the bounce in queries as a result of compelled DNS redirection on the edge, and the drop because of the enlargement of Apple Non-public Relay (see earlier weblog part for detailed evaluation).

The highest classes for 2024 (and 2023) are beneath.

Umbrella tracks the distinctive apps connecting to community. We noticed a marked improve in GenAI. If wanted, we are able to block apps that exhibit a risk to the convention.

2021: 2,162 apps
2022: 4,159 apps
2023: 4,340 apps
2024: 4,902 apps

All in all, we’re very pleased with the collaborative efforts made right here at Black Hat Europe by each the Cisco staff and our companions within the NOC. Nice work everyone!

Black Hat Asia can be in April 2025, on the Marina Bay Sands, Singapore…hope to see you there!
Acknowledgments
Thanks to the Cisco NOC staff:
- Cisco Safety: Ivan Berlinson, Aditya Raghavan, Christian Clasen, Justin Murphy and Ryan Maclennan
- Meraki Programs Supervisor: Paul Fidler and Connor Loughlin
- ThousandEyes: MD Foysol Ferdous and Jessica Santos
- Further Help and Experience: Tony Iacobelli and Abhishek Sha

Additionally, to our NOC companions Palo Alto Networks (particularly James Holland and Jason Reverri), Corelight (particularly Dustin Lee and Mark Overholser), Arista Networks (particularly Jonathan Smith), and your entire Black Hat / Informa Tech workers (particularly Grifter ‘Neil Wyler’, Bart Stump, Steve Fink, James Pope, Michael Spicer, Jess Stafford and Steve Oldenbourg).

About Black Hat
Black Hat is the cybersecurity trade’s most established and in-depth safety occasion sequence. Based in 1997, these annual, multi-day occasions present attendees with the most recent in cybersecurity analysis, growth, and developments. Pushed by the wants of the neighborhood, Black Hat occasions showcase content material immediately from the neighborhood via Briefings displays, Trainings programs, Summits, and extra. Because the occasion sequence the place all profession ranges and tutorial disciplines convene to collaborate, community, and focus on the cybersecurity matters that matter most to them, attendees can discover Black Hat occasions in the USA, Canada, Europe, Center East and Africa and Asia at: Black Hat.com. Black Hat is dropped at you by Informa Tech.
We’d love to listen to what you suppose. Ask a Query, Remark Beneath, and Keep Related with Cisco Safe on social!
Cisco Safety Social Channels
Share: