Category Archives: Training

Cuckoo’s Egg – Week 4 Notes

Session Recording: (Available 12/8-1/4)

Next Sessions Registration:

This week, we reviewed chapters 15-23.

Cliff discovers the attacker attempting to find a pathway into the CIA system by querying the Milnet NIC. He doesn’t find any computers, but he does find the names of four people. Cliff calls these people and finally gets in touch with someone to let him know that the attacker was searching for a CIA computer. The CIA take interest and send someone out the following Monday.

Cliff presents his findings to the CIA, including an agent named Teejay. He learns that DOCKMASTER isn’t a Navy shipyard, but actually an unclassified NSA system. The CIA lets Cliff know they can’t do much and it’s up to the FBI to pursue it. Teejay tells Cliff to keep monitoring and keep him informed regardless. He also shares a story about the zero trust model used at the CIA and a time when an insider intercepted agent data. He was caught when a secretary noticed the last login time on her terminal was something unexpected.

Most Security Practitioners are Choice Architects

The story Teejary shared about the CIA is interesting because of how they caught it. A secretary who was on vacation came back and logged in to her terminal. When a user there logs in they see the output of the last successful login they made. The secretary noticed her last login occurred while she was on vacation and she notified someone, which began the investigation that caught the inside attacker. The last login message is a trigger for a choice, and the people who implemented it are choice architects. All security people are, to some degree, choice architects.

The concept of libertarian paternalism (note: the term libertarian has nothing to do with politics) poses that it is possible and legitimate for someone to affect behavior while also respecting the freedom of choice. We have the ability to allow users to make their own choices while also “nudging” them towards choices that are in their best security interest. This is why default options exist, for example.

In class, we went through several examples of choice architecture that are less than desirable including Facebook’s implementation of “Last Login”, how Word/Excel notify users about macros, and Outlook’s user experience for opening attachment.

More Reading:

Dig Deeper Exercises:

  • Level 1
    • Observe your daily work and note opportunities for security-based choice architecture.
  • Level 2
    • Choose one of the examples you found, or one I presented in class and come up with a way to better nudge users towards a more secure state.
    • Optional: E-mail/DM your idea to me for feedback (

The attacker logs back in and finds a password to the Livermore lab network. This lab does secret research and those computers are supposed to be isolated. They have unclassified computers connected to the network, however. Cliff discovers this when he observes the attacker log into the LBL lab from Livermore. He wasn’t aware that was even possible, but as attackers often do, a new pathway was discovered.

That attacker breaks into the MIT network from LBL. Cliff calls the network operator and discovers this was likely possible because a scientist who accessed Livermore’s computers also accessed MIT computers, and probably left his password laying around.

Network Architecture, Zero-Trust Networks, Beyond Corp, and Air Gaps

A network should be built with defensibility in mind. This means building a network assuming you will be attacked, and assuming at least some of those attacks will be successful. I discussed the components of a defensible network as defined by Richard Bejtlich. A defensible network must be: monitored, inventories, controlled, claimed, minimized, assessed, and current. 

Traditional networks are perimeter focused. Many call this the M&M model with a crunch external shell and a soft interior. Things inside the network are trusted, things outside are not. However, the perimeter has shifted over time thanks to the heavy usage of cloud apps for critical services, the needs of remote or WFH employees, and bring your own device (BYOD).

Many people are now looking to Zero Trust Network models like Google’s BeyondCorp. When you plug into a ZT network, you aren’t automatically afforded any trust. You have to gain trust through multiple factors. Your system has to authenticate via a certificate, the user has to authenticate in two ways, the user has to be enrolled in the proper job classification, and more. All assets are available over the Internet. There’s no VPN to access things anymore or single points of trust assessment, it a combination of multiple rules and trust evaluations going on all the time. This is an oversimplification, but it changes how you might think of a traditional perimeter network.

Air-gapped networks are those that are theoretically physically disconnected from public Internet-touching networks. I say theoretically because in practice many of them aren’t. Someone once said that an air-gapped network is really just a high latency network.


More Reading:

Dig Deeper Exercises:

  • Level 1
    • Research BeyondCorp and examples of real-world deployments outside Google. What were the challenges faced?


Cliff discusses the attack with friends and draws a link between some of the attacker activity. The passwords he’s chosen…jaeger and hunter are german. Benson and hedges are also German — a specific brand of cigarettes.

The attacker breaks into an ELXSI super computer at LBL by guessing a password to a default SYSTEM level account. Cliff discovers this and writes a program to slow the computer down to a crawl when the attacker dials into it. This is to not give away that the attacker has been discovered.

Cliff strengthens his monitoring system by purchasing a pager to notify him when a compromised account logs in. This keeps him from sleeping at the office.

Cliff calls the DOE about the Livermore break in. They tell him to keep it quiet, but to call the National Computer Security Center, which operates out of the NSA. The NCSC is receptive, but can’t do anything about it.

Cliff does some legal research and discovers a warrant isn’t legally required to do a phone trace (USCA SS 3121). He looks over his notes and realizes he wrote down all the numbers the VA telco operator said during the trace. There are only a few available permutations, so he social engineers the operator and has her check the registered owner of all of them, claiming he was erroneously charged for calls to these numbers. Only one is active, and it points to MITRE, a defense contractor in McClean, VA.

He calls the VA Telco and asks them if they could confirm the number he found on his own. They aren’t supposed to do that, but they do it anyway. This is essentially a form of social engineering by getting someone to confirm a piece of information rather than just asking them for it.

Social Engineering

Cliff used social engineering to extract information that he needed to further his investigation. Social engineering in security is an act that influences a person to take an action that may or may not be in their best interest. It usually takes the form of phishing (e-mail), vishing (phone), or impersonation (e-mail, phone, or in person). The human plays a significant role in many breaches. The success rate of external pen tests with humans out of scope is often fairly low (<20%). With humans in scope, it is usually near or at 100%. 

In class we examined a few different SE scenarios and debated which types of scenarios would be most effective. We discussed Maslow’s Hierarchy of Needs and how attackers will leverage primary and secondary needs to illicit action, supress action, reveal information, or change information.


More Reading:

Dig Deeper Exercises:

  • Level 1
  • Level 3
    • Experiment with BeEf to get a sense of what control an attacker has simply by getting you to visit a link.

He speaks to a network operator at MITRE who says that it is impossible his network is hacked. He agrees to put a trace on the line and wait for Cliff to call him the next time the attacker logs in. This would validate the connection. 


Questions to Consider

Are Zero Trust Networks inevitable for all modern networks?

  • Why or why not?
  • What current challenges exist for specific types of networks (see below) to move towards a ZT/BeyondCorp model?
    • Small networks
    • ICS network
    • International networks


Next Session

January 4th 7:30PM ET

Read Chapters 24-30

Register/Attend Here:


Cuckoo’s Egg – Week 3 Notes

Next Week’s Registration:

This week, we reviewed chapters 9-14.

Cliff observes the attacker logging in again via the Sventek account. Sventek uses Kermit to copy a file over. The file is an application that solicits users to enter their password before redirecting them back to a legitimate application. The purpose of the tool is clearly to steal user passwords, but the attacker fails at deploying it successfully and it never executes. 

Realities of Password Theft

We use this opportunity to talk about password theft and the dramatic impact it can have. I posed the question to the group, which of these is worse?

  • An attacker having root privileges on a single system without a clear text user password?
  • An attacker having user privileges no a single system with a clear text user password?

Of course, the answer is “it depends.” The nightmare scenario for prevention and detection is an attacker with clear text credentials for a user with great power.

I highlighted four realities of password theft:

  1. If I can authenticate to a machine as you, the machine gives me the privileges assigned to you.
  2. An attacker doesn’t have to attack vulnerabilities in software if they have legitimate credentials.
  3. An attacker who can access a network with legitimate credentials will almost always do so.
  4. Many long-term attacks involve the use of legitimate credentials.

It’s also important to keep in mind that a user account is not equivalent to a user, it only represents them. An attacker can authenticate as a user, but can never be that user. It is that distinction that we must leverage to detect and prevent attackers who would seek to impersonate.

Clear Text Password Theft

Clear text passwords primarily exist in three places: the user’s head, in transit on the network, in limited places on the operating system. There are techniques attackers can use to steal passwords from all three locations. I performed a demo of each one of these attacks.

Harvesting from the Human: We used the Social Engineering Toolkit to replicate legitimate sites. These are delivered to the victim via some form of social engineering (like a phishing e-mail). The attacker inputs their password, which is covertly sent to the attacker.

Harvesting from the Network: Some protocols perform submission of credentials over clear text. Anyone with a packet sniffer in the right location can intercept these credentials. I demonstrated extracting web application credentials that were transmitted over HTTP.

Harvesting from the OS: While passwords most often exist as file hashes on the local system, there are methods that can be used to extract their clear text representation. One of the most common techniques on Windows systems is the use Mimikatz to take advantage of the LSASS process. I demonstrated the execution of Mimikatz on a Windows 7 system.


More Reading:

Dig Deeper Exercises:

  • Level 1: Download the Social Engineering Toolkit and use the credential collection feature that will clone an existing website. Consider how you might compose a phishing e-mail that tricks a victim to inputting their credentials (don’t actually send it)
  • Level 2: Perform a packet capture while browsing to applications you authenticate to on a regular basis. Assess whether your credentials are submitted in the clear, or over an encrypted channel.


Sandy, a colleague of Cliff finds a computer lab in the library setup to auto-dial Tymnet when students login. It seems logical that an internal attacker (like a student) might be using these terminals to attack the network. Cliff and Sandy work with local law enforcement to post someone in the lab. Cliff monitors for the next time the attacker logs in and calls the lab. Unfortunately, nobody is logged into any of the terminals. The theory that the attacker was coming from the lab is debunked. 

Insider vs. Outsider Threat

We briefly discussed the source of threats. The insider threat has potential to be much more damaging and hard to detect. However, the hype surrounding insider threat is dramatically overblown. Insider threat accounts for an incredibly small percentage of actual breaches.


Cliff begins going through his attacker logs in more depth. He eventually discovers more compromised accounts. A portion of the attacker’s tradecraft is revealed. The attacker will search for old, unused accounts and edit the password file to reactivate them. The attacker would also clear their password so it could be reset, making the accounts perfectly suitable for use again. This was all made possible by the same emacs bug. 

Password Hash Theft

In most places, passwords are stored as hashes rather than in clear text. A hash is a one-way cryptographic function that creates a representation of a password. This is used by the operating system for authentication and storage because it’s more secure than keeping the plaintext password in multiple places. While a password hash is less valuable than a clear text password, it can still be leveraged by attackers to gain access.

I discussed two techniques relating to password hashes.

Password Cracking: An attacker who desires the clear text password associated with a user can attempt to crack the password. I used John the Ripper to demonstrate this process.

Pass the Hash: Sometimes, all you need is the hash. I discussed the Pass the Hash toolkit and how an attacker could use this to gain access as the user whose password hash they’ve stolen.


More Reading:

Dig Deeper Exercises:

  • Level 1: Create a user account on a Windows system. Extract the hash and use John to attempt to crack the password.
  • Level 2: Increase the complexity of the password minimally, and perform the same task again. Keep increasing the complexity and take note of how much longer it takes to crack the password.


Cliff observes the attacker using the LBL connection to connect to White Sands Missile Range (WSMR). The attacker fails to get in. Cliff notifies the FBI of what he’s seen, but they don’t care enough to investigate it. He also notifies the AF OSI. They start looking into it but don’t provide any immediate significant response. 

The next time the attacker dials in, Cliff initiates another trace. The local phone company traces it to a telco in Virginia who is able to trace it to the next hop. Unfortunately, they can’t share the results with Cliff. The telco works with the police, not individuals. Furthermore, that would require a warrant in Virginia and Cliff’s warrant is only good for California. For now, Cliff’s stuck. 


Critical Question(s)

Should this crime have warranted closer inspection by the FBI?

  • Why or why not?
  • How do you determine the threshold for a crime worthy of investigation? Think about this from a macro (FBI) and micro (your company) scale. What is worth the expenditure of resources to pursue?


Next Session

December 7th 7:30PM ET

Read Chapters 15-23

Register/Attend Here:


Cuckoo’s Egg – Week 2 Notes

Next Week’s Registration:

This week, we reviewed chapters 4-8.

After setting up his monitoring system, Cliff observes another suspicious login using the Sventek account. Now, Cliff is better equipped to figure out what is going on and monitor every command issued by the potential attacker. He reads the logs showing the activities taken by the attacker over a three hour span. First of all, he notices that the attacker is logged in from Tymnet. This means the attacker could be coming from anywhere as Tymnet was a global network. Cliff digs into the meat of the log and is alarmed the find that Sventek is operating as a superuser. That shouldn’t be possible!

The Principle of Least Privilege

In security, we operate under the principle of least privilege. The means that users on a system should only be allowed to do the bare minimum required to complete their task and for only as long as they are required to complete it. This means users should not have administrative privileges to change the system, and applications should be installed with limited user accounts. This speaks to the concept of the attack surface. Any access an attacker has on a system provides points of interaction. Generally speaking, the more points of interaction the attacker has, the more opportunities they have to force applications or systems to respond in ways that might allow compromise. This characterizes the attack surface. The fewer interaction points we give users, the smaller the attack surface is when an attacker compromises a user’s account.

In class, we completed an exercise where we mapped the privileges of various roles in a bank setting and analyzed the attack surface of roles controlled by an attacker. I also discussed mechanisms meant to ease the burden of PLP, just as Windows User Account Control (UAC) and Linux sudo.  We are all sysadmins on our own mobile devices, and I discuss how these vendors have taken the approach of forcing us to consent applications into specific access based on their feature needs. Facebook was given as an example along with screenshots showing all the things it needs to access in order to provide its full array of features.

More Reading:

Dig Deeper Exercises:

  • Level 1:  Look at the permissions granted to a social media app on your phone. Try to map each permission to a specific function.
  • Level 2: On a Windows system, use a limited user account and attempt to perform an administrative action. Find an event log that indicates you took this action.
  • Level 3: On a Linux system, create a limited user account. Give it sudo privileges to perform administrative actions. Test this out, and find an event log indicating you took this action.


Cliff continues to dig around and discovers how the attacker became a superuser. It turns out that the attacker found a bug in the gnu-emacs application related to how it assigns ownership of files mailed between users on the system. The application simply changes the ownership of a file without respect to privilege assignments. Since the application was installed as an admin user, this allowed the attacker to trick it into copying a file such that it became owned by a privileged account. The attacker used this big to replace a systems ATRUN file, which executes every 5 minutes to conduct system administration tasks. When the attacker’s version of this file ran, it added the Sventek user to an administrative users group.

As a superuser, the attacker begins pillaging the network. They read sensitive files and e-mails, and look for passwords. The attacker also appears to be very paranoid and constantly looking over their shoulders. They enumerate who the system operators are and constantly look to see who is logged into the system and what processes are running.

Cliff presents these findings to his boss, who tells him to keep watching the attacker but not to take action yet. The monitoring system is refined to alert only when known compromised accounts login.

Process Monitoring

We use this opportunity to talk about process monitoring. The concept of a process is easy to understand, but we should give attention to the tools we can use to monitor running processes and their limitations. Active process monitoring is something done by defenders to look for suspicious or malicious processes. It is also something done by attackers to look for processes that might detect them or prevent them from accomplishing a goal.

In class, I demonstrated the PS command on Linux and discussed different ways to use it to look for running processes. I also demonstrated Process Explorer on a Windows system, as well as TASKLIST for command-line and remote process monitoring.

More Reading:

Dig Deeper Exercises:

  • Level 1: Choose three processes you use on a daily basis. Use the tools here to locate them and trace their parent processes as far as you can. Research what each parent process does.
  • Level 2: Compare your system to a clean system with no software installed (you may need to set up a virtual machine in a lab to do this). Identify what processes are unique to your system and those that are standard with the operating system.


In discussion with one of Cliff’s colleagues, the attacker’s use of the PS command is brought up. The attacker uses a flag (-g) that is supported on AT&T Unix, but not Berkeley Unix that is run at LBL. Someone wagers that the hacker is not from the west coast or he would be well versed in their flavor of Linux.

The attacker dials back in but doesn’t stay on LBL systems for long. Instead, they use LBL as a hop point to connect to a DoD server as the user Hunter. Cliff queries the NIC and finds that this IP belongs to Anniston Army Depot in Alabama. He calls the system operator and explains what he observed. The operator is aware of the attacker having been there, but thought he had been eradicated. It turns out that the attacker was using the same emacs bug but is coming in through LBL and not the front door anymore. Cliff ponders the thought that the attacker might be southern, as it would explain the familiarity with AT&T Unix which was run on the Anniston systems.


Next Session

November 30th 7:30PM ET

Read Chapters 9-14

Register/Attend Here:


Cuckoo’s Egg – Week 1 Notes

Next Week’s Registration:

This week, we reviewed chapters 1-3.

We met Cliff Stoll, an astronomer turned computer wizard at Lawrence Berkeley Lab in California. Once of Cliff’s first tasks is to figure out an accounting error that amounted to $0.75 of CPU time. He investigates and finds the error is tied to a mysterious user account named Hunter. He can’t find the source of the account so he deletes it.

Locard’s Exchange Principle

We use this opportunity to discuss Locard’s Exchange Principle. Edmund Locard is considered by many to be the father of modern forensic science. His principle states, “The perpetrator of a crime will bring something into the crime scene and leave with something from it.” This is the basis for all forensic investigations. Locard has a particularly nice quote on the subject:

“Wherever he steps, whatever he touches, whatever he leaves, even unconsciously, will serve as a silent witness against him. Not only his fingerprints or his footprints, but his hair, the fibers from his clothes, the glass he breaks, the tool mark he leaves, the paint he scratches, the blood or semen he deposits or collects. All of these and more, bear mute witness against him. This is evidence that does not forget. It is not confused by the excitement of the moment. It is not absent because human witnesses are. It is factual evidence. Physical evidence cannot be wrong, it cannot perjure itself, it cannot be wholly absent. Only human failure to find it, study and understand it, can diminish its value.”

In class, we discussed the principle as the basis for computer forensic investigations as well. We did a couple activities to exercise our minds and think about things taken and left behind. One in relation to a physical theft, another in relation to the 2017 Equifax breach.

More Reading:

Dig Deeper Exercises:

  • Level 1: Pick a crime in your local newspaper and break down what could have been left behind.
  • Level 2: Pick an attack time from this list: Consider your network or make up a fictional network. Research the attack and determine what might be left behind and how you might gain visibility into it.


Cliff gets a weird e-mail from a system called DOCKMASTER. The system owner claims that someone from LBL tried to break into his computer. Eventually, Cliff figures out this system belongs to a Naval Shipyard. He correlates timestamps provided by DOCKMASTER and finds the user Sventek was active at this time. He also discovers two logging systems reporting different timestamps for the activity. While odd at first, this turns out to be related to time drift between two system clocks.



The investigation work Cliff is conducting is contingent on logs that contain timestamps, which he uses to perform time-based correlation. It’s easy to think of timestamps as a trivial thing, but they are far from it. Most investigations require examination of multiple data sources to build a clear picture of what events have transpired. To properly query data and sequence events, we need accurate timestamps.

There are multiple challenges between an investigator and reliable, consistent timestamps. We discussed syncing timestamps, time sources, network time protocol (NTP), W32Time, and how Windows domain members sync time. We also discussed the challenges associated with timezones and daylight savings time with plenty of confusing examples (Seriously, Samoa?)

I showed multiple examples of timestamps, and also showed a log collection pipeline and Logstash configuration files used to adjust timing and define timestamps. Finally, I listed a few best practices for dealing with timestamps that include: syncing all systems to the same source, utilizing UTC time in your investigation tools, and using ISO 8601 compliant timestamps.

More Reading:

Dig Deeper Exercises:

  • Level 1: Determine where your system is syncing time from and change it to another source.
  • Level 2: Setup your own NTP server and configure your system to sync from it.
  • Level 3: Capture network traffic while syncing with your own NTP server. Examine each field, and try to determine the function of each one.


Cliff eventually learns that the Sventek user is not on campus and is unlikely to be using his account. Considering the anomalies encountered with the Hunter and Sventek accounts and the report from DOCKMASTER, Cliff begins to suspect someone has broken into his network. He takes matters into his own hands and builds a monitoring system. He writes a program to log keystrokes on his systems and connects them in between the system and the external modems. He connects physical printers to these systems to print out commands as people are entering them while dialed in remotely. He sleeps in his office all weekend to monitoring these connections and awakens one night to find something very interesting…


Next Week

November 16th 7:30PM ET

Read Chapters 4-8

Register/Attend Here:


The AND Student Charitable Profit Sharing Program

When I decided to launch Applied Network Defense, I did so with the intention of using it as a platform for making positive change in the world. We do that through our primary business model of educating people so that they can further their careers and secure their works. We also do that by donating a portion of every course purchase to charity. Thus far, those donations have helped introduce tens of thousands of kids to computer science education and save lives by outfitting entire villages in Africa with mosquito nets.

It’s time for the next evolution of our mission, and that is a student charitable profit sharing program. Now, AND students will have a say in where a portion of their course proceeds go. Periodically, AND students will have the opportunity to submit a charity to receive a donation. After nominations have been received, the collective group will vote and we will select 2-5 winners to receive a donation.

If you are a current or former student of mine, I want you to know that my life is enriched by having been able to interact with you. Now, I’m thrilled to be able to help you contribute to causes that matter to you as well. It’s very important that people who purchases training from AND know that they aren’t just educating themselves by doing so, they are enriching the lives of others too. This is another way for us to do that, together.