My Testimony

blurry-sky-crossWe didn’t have much growing up in rural western Kentucky. Not many people who lived in Mayfield did, really. Those who did have means, the few doctors we had in town or the occasional lawyer or pharmacist who had managed to do well for themselves, still lived their lives with modest sensibilities. It was a slower way of life than I suspect most are familiar with.

 

Looking back on the first fifteen years of my life, I can say that if there were a parent lottery, I was undoubtedly the winner. Although we didn’t have much, I never remember doing without. I always had presents under the tree at Christmas and I was never without supper on the table, although dessert after supper was a foreign concept to me until college. Most importantly, although my family was small, I always felt loved.

 

My Dad was a trucker who was known to take odd jobs around town. I can’t count the number of fence posts I helped him set or the number of pole barns I helped erect. He was a tough man, but he was also the kind of person who could start a conversation with anybody and make them feel like they had known him for years.

 

My Mom was no stranger to a hard days work either. She spent most of my young life working as a machine operator in a few different textile factories around town. Miss Judy, as my friends typically grew to call her, was one of the strongest willed women you would ever meet. Most people would call her fiery or opinionated, and some who got on her bad side might have some other creative ways of expressing the same sentiment. Regardless, nobody could ever accuse Mom of being anything less than one of the most caring people you would ever meet. As a proper southern woman, it was typical for her to greet a visitor to our home by asking them if they’d eaten yet, and subsequently forcing a few bites of something down their throat regardless of their answer.

 

She was fiercely protective of those she loved, especially me. It was something that frustrated me to no end. My dim and narrow view on the world frequently led me to believe that she was holding me back, or that her protection was driven by something other than her simply wanting what was best for me. She had gone through a falling out with my much older sister Kim that had led Kim to leave town and become a very fleeting part of our lives. I suspected that Mom was simply trying to make sure that she didn’t repeat those same mistakes with me. I generally preferred to spend my time indoors reading or tinkering with anything that I could take apart and rebuild. I specifically remember how she would come into my room on weekends and demand that I go outside and find something constructive to do in the fresh air. “Go Play!” she would say as she shooed me out the back door, making sure I was wearing the appropriate shoes. I despised being forced to do it.

 

I figured out at an early age that my Mom wasn’t in the best of health. She smoked a couple of packs of cigarettes a day and had a regiment of pills prescribed for everything from high blood pressure to diabetes. Mom was prone to get sick enough to be stricken to the couch on a regular basis. While most people tend to relegate themselves to their bed when struck by illness, Mom preferred the couch. I think it was because she felt more accessible in the living room. She particularly liked it when I would sit with her when she wasn’t feeling well, even if it were just to watch TV.

 

It wasn’t until I approached my teens that things started getting more serious than I knew they were. Around this time Mom’s bouts with illness went from periodic stays on the couch to periodic stays in the hospital. These trips were never too long or too serious; an overnight stint for a really bad flu here, a three-day stay for pneumonia there.  I started to become more familiar with the local hospitals than I realized.

 

It’s important to explain at this point that religion in my family was a bit like that weird cousin everybody has, or an odd haircut: it was always there, but was not spoken about often. It was just something that we accepted. Dad was a traditional southern man who didn’t tend to talk a lot about things like religion or his emotions.  Mom, who had been a Sunday school teacher in her younger years, took her religion very seriously. Like Dad though, she didn’t talk about it too much. Both of my parents explained the bible to me when I was younger and made sure I had the resources to educate myself on topics of faith and religion, but they both lived their lives in such a way as to show faith rather than speak about it.

 

Although I believe my parents were well intentioned, it’s hard for a young person to read between the lines and derive faith by example. Because of the nature of my parent’s work, and the later state of Mom’s health, we didn’t attend church regularly. As a result, I lacked a lot of the core knowledge that helps in the understanding of faith and religion. Don’t get me wrong, there was a healthy amount of God fearing in our household, but I suppose faith was something that I was just expected to learn about on my own, rather than being taught.

 


 

This story really picks up when I was fifteen. This was in 2001, just after my birthday, which unfortunately, was the same day as the infamous 9/11 attacks. At this point in my life I was a typical high school sophomore who tended to get lost in the crowd. I didn’t really have any particular hobbies other than fiddling with the computer on occasion, and I wasn’t anything more than a B- student. I was a good kid, but simply put, I wasn’t living for anything. I was more or less floating through life.

 

Not long after my birthday, Mom was admitted to our local hospital for a bout of pneumonia. This wasn’t anything out of the ordinary. I remember visiting her on a Sunday night with my Dad. Although Mom wasn’t feeling great by any means, she was still herself, and we weren’t expecting anything different than the normal two to three day visit.

 

The Mayfield hospital was right across the highway from my high school. The spot where I would catch the bus home in the afternoon faced the west side of the hospital, which just happened to be where Mom’s room was this time. She had noticed this and made me promise to wave to her when I was getting on the bus the next afternoon. The distance between the school and the hospital was great enough that I couldn’t see whether or not she saw me waving, but I did just as I had promised her I would. There was no doubt that this action probably looked a bit strange to any of my peers who happened to witness it.

 

Dad had spent part of that Monday with Mom, but I didn’t go visit her that day. Even though the hospital was only a few miles away from my house, the routine nature of these hospital stays and the low severity of the pneumonia she had acquired warranted that Dad would only shuttle me to and from the hospital every other day or so as to not disrupt my routine too much. Nothing in particular stood out about that Monday, until later that night.

 

“Chris, wake up.”  Dad said, as he grabbed my arm and gently shook me.

 

I tossed and turned, not showing too many signs of life.

 

“Chris, get up.” Dad’s grasp tightened, his shaking intensified. This was very unusual.

 

“Dad?” The only word I could muster in my only partially awaken state.

 

“Get your shoes on.” Dad said. “Something happened with Mom, we’ve got to go to the hospital.”

 

I don’t honestly remember a whole lot about the ride to the hospital. The combination of being awoken from my deep sleep at 3AM and the general shock of the situation left me in a state where I wasn’t sure if what was happening was real, or only a dream.

 

I do remember that Dad didn’t say a word on the short trip. By the time we had arrived at the hospital I knew nothing more than what he had told me when he woke me up a few short minutes earlier.

 

We were met in the ICU by our family practitioner, Dr. Jones. Being a small town, Dr. Jones had been our family doctor for my entire life. He was a friend to the family, and we had close ties to other members of his family. His son was one of the few town veterinarians, and cared for the variety of animals we kept over the years. His oldest daughter was a banker and helped provide the financing for my first home many years later. I went to high school with his youngest daughter, although I didn’t know her too well at this point in my life.

 

Dr. Jones explained to us that Mom had become unresponsive at some point during the night, and her vital stats had dropped dangerously low. They had no idea what happened, and at this point they were still working to stabilize her. My memory of this night is still very fuzzy. We remained at the hospital for several hours until they were eventually able to stabilize her condition, although she remained unconscious. Nobody was able to provide any insight into why she had crashed. Dad wouldn’t let me see her that night, no doubt attempting to shield me from seeing her in that state.

 

At the urging of Dr. Jones, Dad and I returned home to get a few hours of sleep. We got home just as the sun was rising. I don’t know if Dad actually went to sleep at this point. I’d guess not. I think that the only reason he agreed to go home was for my sake. I was tired and confused, and he knew that I needed to be well rested in order to be able to process the events that had occurred. I woke up a few hours later, realizing that I wouldn’t be going to school that day. We headed back to the hospital.

 

I distinctly remember the moment I walked into the ICU and laid eyes on Mom. I had seen her in the hospital dozens of times, but I had never seen her like this. She had several lines and tubes entering her body. A variety of machinery lined the walls of the very carefully designed room. The thing I remember most was the heart monitor. I had seen these on TV, but had never seen one in real life. It beeped with regularity just like I had seen on “ER”, a show Mom and I watched every Thursday night.

 

Beep…..Beep…..Beep…..Beep…..

 

The beeps corresponded to the spikes on the machines display, coming at regular intervals. I didn’t know anything about medicine, but I knew that this machine measured the frequency of her heartbeat, and that it was of critical importance.

 

Mom remained in this state for several days, which were mostly uneventful. Dad thought that it was important that I not miss too much school, so I alternated my days between school and the hospital.  Although I was incredibly worried, I surprised myself in my ability to continue through the school day as though everything was normal. It was a strength whose source I didn’t know.

 

Towards the end of the first week, no news quickly turned into devastating news. Although Dad had tried to shield me from a lot of things throughout my life, when Dr. Jones came in with updates, Dad wanted me to hear it at the same time he did. Looking back, I think it helped him as much as it did me. I don’t have children, but I can’t imagine it’s easy to deliver bad news to them.

 

Dr. Jones had come to tell us that Mom’s organs were starting to shut down. Her lungs weren’t in the best shape as it was, and now her liver was showing signs of trouble and her kidneys were starting to fail to produce urine. It was clear that this discussion wasn’t easy for Dr. Jones. He was a life long friend of the family, and he was trying to tell a Father and his fifteen year-old son that their wife and mother was dying. I remember the steady sound of the heart monitor drowning out the doctors words.

 

Beep…..Beep…..Beep…..Beep…..

 

A few more days passed and Mom’s condition worsened. It wasn’t long until word got out that things had gotten very serious, and visitors started to arrive. Although our family was small, those that existed showed up to see Mom and give Dad and I their support. This included my sister Kim, and my Aunt Sandy, who was also Mom’s best friend. Word had also gotten to my school as my absence became noticeable, and several of my teachers showed up to show their support as well. One of the best things about rural communities like Mayfield is how they rally around people in their time of need.

 

The number of people around us became a bit smothering. Within a couple of days however, Dr. Jones brought more news that made me forget that anyone else was around. The doctors treating Mom now believed that an infection in the fluid that surrounds her heart was to blame for her current condition. Her already frail state and the years of abuse her body had already taken continued to take a toll, and now her kidneys were barely functioning at all. She wasn’t able to breathe on her own, and she now required a feeding tube.

 

Dr. Jones chose his next words very carefully. It was clear he had given this speech before, and despite his best efforts to give it in the caring, understanding tone you would expect from a friend, his words came across as somewhat robotic. I can’t say I blame him. I’m not sure a normal human could get through it without detaching themselves from the words they were speaking to some extent.

 

He explained to us that he and the other treating doctors believed that Mom was dying, and that the machines that she was connected to were now the only things sustaining her life. Very carefully, but very clearly, I listened as Dr. Jones explained to my Dad that he had to make a decision on whether or not measures should be taken to prolong my Mom’s life. It was the very same speech I had seen doctors give to fictitious patients on “ER”. He insisted that Dad shouldn’t make the decision immediately, and that he should take some time to talk to the family about it.

 

The rest of that day I witnessed a range of emotions I had never seen from my Dad. A man that wasn’t used to showing emotion whatsoever was attempting to deal with what I can only imagine was sadness, confusion, and anger all at once.

 

At one point, I remember Dad speaking to me angrily as we sat next to Mom’s hospital bed. He wasn’t mad at me, but mad at the situation he found himself in. “Son” he said, “I don’t wish this on anybody. Not my worst enemy. If they ever try to hook me up to any of these machines, I want you to let me die right then and there. This isn’t right.”

 

It was hard to hear Dad talk like this. He had never been so candid with me about his own mortality. I wasn’t prepared to handle it. I had tried for so long to stay strong for Dad, but this was too much. I broke down crying as Dad tried to comfort me. The only thing louder than my sobbing was the rhythmic beeping of the heart monitor.

 

Beep…..Beep…..Beep…..Beep…..

 

Several hours later, something truly remarkable happened. Dad, Kim, Sandy and I were staying in a nearby empty room the hospital staff had provided for us. Dr. Jones came into the room hastily without knocking. Mom was awake.

 

We wanted to rush to her side, but Dr. Jones stopped us before we could exit the room. He spoke quickly but concisely. This was another speech he had given before. He explained to us that although Mom was lucid and communicating, that this didn’t mean that she was getting better. In fact, these moments of lucidity were often an indicator that things were about to get much worse.

 

As Dad, Kim, Sandy and I went into her room, my eyes made contact with hers. Her eyes grew wider and the rhythm of the heart monitor increased. It was the first time its rhythm had changed since this whole event had started. She tried to talk, but the breathing tube in her throat prevented it. Not more than a moment later, a nurse came into the room with a special card that was used for situations like these. The card had all of the letters of the alphabet on it, as well as numbers, a few common conversational phrases, and thumbs up and down symbols.

 

Mom attempted to point at letters on the card, but to no avail. She didn’t have her glasses, and we weren’t able to find them immediately. She improvised quickly, and pulled my Dad close to her. She started writing letters with her finger on his chest.

 

Her first question was simple.

 

“What happened?”

 

Dad explained the gravity of the situation and the events that had taken place as best he could. Eventually, Dad began to explain to her the seriousness of her condition, and that the doctors believed she was dying. Mom reacted to Dad’s words as he delivered the news to her. She winced on a few occasions, and at one point tears began to roll from her eyes and down the side of her face.

 

Once Dad was finished, he asked her if she understood how serious the situation was.

 

She nodded.

 

Next, Dad explained to her that her new state of consciousness could be a sign that she was about to get much worse.

 

She nodded.

 

Finally, Dad told her that the doctors caring for her didn’t believe she was going to survive her current state. He explained, almost word for word, what Dr. Jones had told him regarding the use of extraordinary measures. Then, without wasting any words, my Dad asked his wife of thirty years if she wanted the life support machines to be turned off, and if she was ready to die.

 

Her gaze went around the room as she made eye contact with each one of us independently. She paused for a few seconds as she looked at each of us before moving on to the next. First Dad, then me, then Kim, then Sandy, and back at Dad again. It was though she was trying to gauge our reactions. That she was trying to comfort us in some way, knowing that she couldn’t speak to us or reach out and hold us.

 

One more time, she nodded.

 

Seeing her awake helped me to hold back any tears that might have otherwise poured out of me. I felt like I needed to be strong for her and my Dad. As much pain as I was feeling, I knew Dads had to be exponentially worse.

 

A few minutes later, as the gravity of the situation set in further, Mom motioned for me to come closer to her and take her hand. She looked at me and then to my Dad, and began writing letters on his chest with her finger.

 

She wrote a capital G, followed by a small O. She paused briefly, and then wrote a capital P, and a straight line indicating a lowercase L. She paused once again, noticeably wincing. I wasn’t sure if the wince was a result of the physical pain she must have been in or an emotional response to the current situation. She began writing again, making the lowercase letters A and Y.

 

“Go Play.”

 

Mom didn’t want me to see her like this. It wasn’t fair that I had to see her die; that I had to watch her slowly dying for the majority of my childhood.  Just as her and Dad had been so protective of me in all situations of life, she wanted to protect me from having to experience the pain that was growing inside me. She was literally dying in front of me, and her only concern was protecting me from that pain.

 

It was too much.

 

I left the room before my tears had the chance to overtake me and collapsed into a chair positioned just outside of the ICU room. I stayed in that chair for several minutes as the others continued to communicate with Mom. Although I could audibly hear their voices, I didn’t retain a single word of their conversation. I couldn’t take it. I had to escape. Once again I found myself focusing on the beeping heart monitor.

 

Beep…..Beep…..Beep…..Beep…..

 

It wasn’t long after that before Mom drifted back into unconsciousness. Two more days passed and she continued to worsen. Honoring Moms wishes, Dad signed the paperwork to end life support on a Tuesday morning. It just so happened that day was their 30th wedding anniversary.

 

The doctors explained to us that once the life support was removed, it might take quite some time before anything happens. We remained vigil throughout most of that day. Towards late afternoon, Mom became conscious once again.

 

In the hour or so that followed, we all spent some time with her individually. She was able to talk, although it was very laboring for her, so her words were kept to a minimum.

 

My dad went first. I’ve thought about what might have been said between them many times, but I’ve never asked, and I never will. I can’t imagine what words a husband tells his dying wife, or how a dying mother ensures her husband knows that he is capable of raising their son by himself. I truly hope that when my time comes, it happens quickly, and that I go before my wife. It must have been agony for both of them.

 

I honestly didn’t believe I would ever be able to talk to Mom again, so I wasn’t prepared when it became my turn to say goodbye. I won’t go into the details of everything that we discussed, but there were a couple of really important things. First of all, I felt that it was important that she knew that I would take care of Dad as much as he would take care of me. I told her that we would never be whole without her, but that we would make it because of her impact on our lives.

 

Mom lay there and patiently waited for me to say everything that I felt I needed to say. Then, she spoke to me. She made sure I knew how proud she was of me and that I knew how much she loved me. She told me how important education was, her herself not having graduated from high school. She obviously regretted that. She also told me that I should pursue my interest in computers, and that I was bound for great things if I did. These things didn’t catch me too much by surprise as they were recurring themes she continually reinforced to me when she had been in better health.

 

Then, she brought up something that caught me entirely off guard. Something that we had never really talked about in great detail, and that was her faith. Speaking with more clarity than I could have imagined was possible in her debilitated state, she spoke about God, religion, heaven, and hell. She wanted me to know that I knew where she was going when she died; that her soul transcended her physical being because of her relationship with Christ, and that there was no relationship more important than a person’s relationship with God.

 

She continued, and told me that she greatly regretted never discussing this with me, before now. She told me that she wasn’t afraid of dying, because she was going to walk with God in all of his glory, and that there would be no more pain where she was going. She spoke about heaven in the same way a young child speaks about Christmas or Disney World, with childlike wonder, and amazement. She made sure that I knew that I would see her again some day. She told me that God needed her more than I did now, and that this wasn’t goodbye, it was just merely “See you later.”

 

Finally, she told me that there was nothing more important to her than my establishing a relationship with Christ. She pronounced that if I was to put faith in Him that all things would be possible, and that alone I am nothing but flesh and bones, but through Him I would gain eternal life. Mom and I prayed together, and it was in that moment that I accepted Jesus Christ as my savior.

 

Not much longer after that, Mom drifted back to sleep. I was lying down in the extra room we were staying in several hours later. It was just past one in the morning and although my mind had been racing, I managed to fall asleep from pure exhaustion. Then, I was abruptly awoken by my sister’s voice.

 

“Chris, wake up. It’s happening.”

 

I rushed into the Mom’s ICU room. Dad was at her side rubbing her forehead. I looked to the heart monitor. The beeping had slowed and the numbers displayed on the screen were dropping lower.

 

Beep……..Beep……..Beep…

 

Sandy placed herself at the foot of the bed. Kim moved to the far side of the bed. I positioned myself next to Dad. He took my hand and placed it into my Mom’s left hand at her side. I grasped it tightly.

 

Beep……………Beep……………..Beep…

 

The numbers on the heart monitor continued to drop. Both Kim and Dad’s gaze were locked on Mom. Her last moments we growing nearer.

 

Beep……………………Beep………………………

 

As the numbers inched lower, Mom’s eyes opened slightly. She looked intently into Dad’s eyes while he looked back at her. Then her eyes slowly moved to the right and made contact with mine.

 

“Beep…………………………………Beep…………………………..”

 

Her hand squeezed mine ever so slightly. Her mouth moved very faintly. As it started to move we all leaned in closer as she said two words to me…“Go Play.” I managed to hold my tears back. I had to remain strong for everyone else.

 

Beep……………………………………………………….Beep…………….

 

She looked back at Dad. He spoke to her, “It’s happening.” He said. Then he asked her, “Are you ready?”

 

She nodded, and then she faintly said to Dad, “I love you”.

 

We all remained silent. Dad stared intently into Mom’s eyes as her eyelids closed for the last time. Kim held her right hand, as I grasped her left. Dad gently stroked her hair back from her forehead as she took her last breath.

 

Beep……………………………………………………………………………..…………..

 

The numbers on the heart monitor were no longer visible. The nurse who had been in the room turned and switched the machine off. We lingered for a few more moments, and then we left the room together.

 

Mom had gone to be with the Lord, and she was no longer in pain.

 


 

Losing my Mom was without a doubt the hardest thing I’ve ever been through in my life. It was difficult to see at the time, but those last few days with her changed my life. I entered that experience as an empty shell; a person who was living for nothing. I had no future in life or in death. I was simply going through the motions.

 

After Mom’s passing, something very unexpected happened. My family started looking to me for the same strength they used to rely upon her for. This was especially evident a couple of years later when we lost my Aunt Sandy and my sister Kim.

 

Romans 1:19-21 tells us that God makes himself apparent to everyone at some point. Further, Romans 10 tells that we learn that everyone has the chance to be saved by the grace of God; they must only be prepared to accept it.

 

Romans 10:9 says:

 

“Because, if you confess with your mouth that Jesus is Lord and believe in your heart that God raised him from the dead, you will be saved.”

 

Sitting next to my Mom, as I felt the life leaving her body, I had been saved. I had been transformed, and as time went on my life quickly gained clarity and focus. I began to find immense joy in the things and people around me, I turned my computer hobby into a career, and I began to study His word and live through it. I started loving others more than myself. I gained patience, kindness, and strength. These things had been inside me all along, but God had now provided me with the ability to recognize it. I had been reborn in Christ’s love. I was living for something bigger than myself. I was living for Him.

 

When tragedy occurs in our lives, it is very hard to see that it is all a part of His plan for us. There is no verse that states it more beautifully than Jeremiah 29:11-13:

 

“’For I know the plans I have for you,’ declares the Lord, ‘plans to prosper you and not to harm you, plans to give you hope and a future. Then you will call on me and come and pray to me, and I will listen to you.  You will seek me and find me when you seek me with all your heart.’”

 

Although she is no longer with me in body, I feel her spirit with me every day. Everything God does has a purpose. Everything He does is part of the plan he has for me. He took my Mom on that cold October day because He needed her more than I did.  He used my Mom to bring me into the body of Christ and to radically change my life. I don’t think Mom would have had it any other way. I can’t wait to talk to her about it some day, and I can’t wait to see what He has planned for me next.

 

Applied Network Security Monitoring, the book!

I’m thrilled to announce my newest project, Applied Network Security Monitoring, the book, along with my co-authors Liam Randall and Jason Smith.

Better yet, I’m excited to say that 100% of the royalties from this book will be going to support some great charities, including the Rural Technology Fund, Hackers for Charity, Hope for the Warriors, and Lighthouse Youth Services.

You can read more about the book, including a full table of contents at its companion site, here: http://www.appliednsm.com/.

Information Security Incident Morbidity and Mortality (M&M)

It may be a bit cliché, but encouraging the team dynamic within an information security group ensures mutual success over individual success. There are a lot of ways to do this, including items I’ve discussed before such as fostering the development of infosec superstars or encouraging servant leadership. Beyond these things, there is no better way to ensure team success within your group than to create a culture of learning. Creating this type of culture goes well beyond sending analysts to formalized courses or paying for certifications. It relies upon adopting the mindset that in every action an analyst takes, they should either be teaching or learning, with no exceptions. Once every analyst begins seeing every part of their daily job as an opportunity to learn something new or teach something new to their peers, then a culture of learning is flourishing.

A part of this type of organizational culture is learning from both successes and failures. The practice of Network Security Monitoring (NSM) and Incident Response (IR) are ones that are centered on technical investigations and cases, and when something bad eventually happens, incidents. This is not unlike medicine, which is also focused on medical investigations and patient cases, and when something bad eventually happens, death.

Medical M&M

When death occurs in medicine, it can usually be classified as something that was either avoidable or inevitable from both a patient standpoint and also as it related to the medical care that was provided. Whenever a death is seen as something that may have been prevented or delayed with modifications to the medical care that was provided, the treating physician will often be asked to participate in something called a Morbidity and Mortality Conference, or M&M as they are often referred to casually. In an M&M, the treating physician will present the case from the initial visit, including the presenting symptoms and the patients initial history and physical assessment. This presentation will continue through the diagnostic and treatment steps that were taken all the way through the patient’s eventual death.

The M&M presentation is given to an audience of peers, to include any other physicians who may have participated in the care of the patient in question, as well as physicians who had nothing to do with the patient. The general premise is that these peers will question the treatment process in order to uncover any mistakes that may have been made or processes that could be improved upon.

The ultimate goal of the medical M&M as a team is to learn from any complications or errors, to modify behavior and judgment based upon experiences gained, and to prevent repetition of errors leading to complications. This is something that has occurred within medicine for over one hundred years and has proven to be wildly successful.

Information Security M&M

I’ve written about how information security can learn from the medical field on multiple occasions, including recently discussing the use of Differential Diagnosis for Network Security Monitoring. The concept of M&M is also something that I think transitions very well to information security.

As information security professionals, it is very easy to miss things. I’m a firm believer that prevention eventually fails, and as a result, we can’t be expected to live in a world free from compromise. Rather, we must be positioned so that when an incident does occur, it can be detected and responded to quickly. Once that is done, we can learn from whatever mistakes occurred that allowed the intrusion, and be better prepared the next time.

When an incident occurs we want it to be because of something out of our hands, such as a very sophisticated attacker or an attacker who is using an unknown zero day. The truth of the matter is that not all incidents are that complex and often times there are ways in which detection, analysis, and response could occur faster. The information security M&M is a way to collect that information and put it to work. In order to understand how we can improve from mistakes, we have to understand why they are made. Uzi Arad summarizes this very well in the book, “Managing Strategic Surprise”, a must read for information security professionals. In this book, he cites three problems that lead to failures in intelligence management, which also apply to information security:

  • The problem of misperception of the material, which stems from the difficulty of understanding the objective reality, or the reality as it is perceived by the opponent.
  • The problems stemming form the prevalence of pre-existing mindsets among the analysts that do not allow an objective professional interpretation of the reality that emerges from the intelligence material.
  • Group pressures, groupthink, or social-political considerations that bias professional assessment and analysis.

The information security M&M aims to provide a forum for overcoming these problems through strategic questioning of incidents that have occurred.

When to Convene an M&M

In an Information Security M&M, the conference should be initiated after an incident has occurred and been remediated. Selecting which incidents are appropriate for M&M is a task that is usually handled by a team lead or member of management who has the ability to recognize when an investigation could have been handled better. This should occur reasonably soon after the incident so important details are fresh on the minds of those involved, but far enough out from the incident that those involved have time to analyze the incident as a whole, post-mortem. An acceptable time frame can usually be about a week after the incident has occurred.

M&M Presenter(s)

The presentation of the investigation will often involve multiple individuals. In medicine, this may include an initial treating emergency room physician, an operating surgeon, and a primary care physician. In information security, this could include an NSM analyst who detected the incident, the incident responder who contained and remediated the incident, the forensic investigator who performed an analysis of a compromised machine, or the malware analyst who reverse engineered the malware associated with the incident.

M&M Peers

The peers involved with the M&M should include at least one counterpart from each particular specialty, at minimum. This means that for every NSM analyst directly involved with the case, there should be at least one other NSM analyst who had nothing to do with it. This aims to get fresh outside views that aren’t tainted by feeling the need to support any actions that were taken in relation to the specific investigation. In larger organizations and more ideal situations, it is nice to have at least two counterparts from each specialty, with one being of lesser experience than the presenting individual and one being of more experience.

The Presentation

The presenting individual or group of individuals should be given at least a few days notice before their presentation. Although the M&M isn’t considered a formal affair, a reasonable presentation is expected to include a timeline overview of the incident, along with any supporting data. The presenter should go through the detection, investigation, and remediation of the incident chronologically and present new findings only as they were discovered during this progression. Once this chronological presentation is given, the incident can then be examined holistically.

During the presentation, participating peers should be expected to ask questions as they arise. Of course, this should be done respectfully by raising your hand as the presenter is speaking, but questions should NOT be saved for after the presentation. This is in order to frame the questions to the presenter as a peer would arrive at them during the investigation process.

Strategic Questioning

Questions should be asked to presenters in such a way as to determine why something was handled in a particular manner, or why it wasn’t handled in an alternative manner. As you may expect, it is very easy to offend someone when providing these types of questions, therefore, it is critical that participants enter the M&M with an open mind and both presenters and peers ask and respond to questions in a professional manner and with due respect.

Initially, it may be difficult for peers to develop questions that are entirely constructive and helpful in overcoming the three problems identified earlier. There are several methods that can be used to stimulate the appropriate type of questioning.

Devils Advocate

One method that Uzi Arad mentions in his contribution to “Managing Strategic Surprise” is the Devils Advocate method. In this method, peers attempt to oppose most every analytical conclusion made by the presenter.  This is done by first determining which conclusions can be challenged, then collecting information from the incident that supports the alternative assertion. It is then up to the presenter to support their own conclusions and debunk competing thoughts.

Alternative Analysis (AA)

R.J. Heuer presents a several of these methods in his paper, “The Limits of Intelligence Analysis”. These methods are part of a set of analytic tools called Alternative Analysis (AA).

Group A / Group B

This analysis involves two groups of experts analyzing the incident separately, based upon the same information. This requires that the presenters (Group A) provide supporting data related to the incident prior to the M&M so that the peers (Group B) can work collaboratively to come up with their own analysis to be compared and contrasted during the M&M. The goal is to establish to individual centers of thought. Whenever points arise where the two groups reach a different conclusion, additional discussion is required to find out why the conclusions differ.

Red Cell Analysis

This method focuses on the adversarial viewpoint, in which peers assume the role of the adversary involved with the particular incident. In doing this, they will question the presenter as to how their investigative steps were completed in reaction to the attackers actions. For instance, a typical defender may solely be focused on finding out how to stop malware from communicating back to the attacker, but the attacker may be more concerned with whether or not the attacker was able to decipher the communication that was occurring. This could lead to a very positive line of questioning that results in new analytic methods that help to better assess the impact of the attacker to benefit containment.

What If Analysis

This method is focused on the potential causes and effects of events that may not have actually occurred. During detection, a peer may ask a question related to how the attack might have been detected if the mechanism that did detect it didn’t do so. In the response to the event, a peer might question what the presenter would have done had the attacker been caught during the data exfiltration process rather than after it had already occurred. These questions don’t always relate directly to the incident at hand, but provide incredibly valuable thought provoking discussion that will better prepare your team for future incidents.

Analysis of Competing Hypothesis

This method is similar to what occurs during a differential diagnosis, where peers crate an exhaustive list of alternative assessments of symptoms that may have been presented. This is most effectively done by utilizing a whiteboard to list every potential diagnosis and then ruling those out based upon testing and review of additional data. You can review my article on differential diagnosis of NSM events here for a more thorough discussion of this type of questioning.

Key Assumptions Check

Most all sciences tend to make assumptions based upon generally accepted facts. This method of questioning is designed to challenge key assumptions and how they affect the investigation of a scenario. This most often pairs with the What If analysis method. As an example, in the spread of malware, it’s been the assumption that when operating within a virtual machine, the malware doesn’t have the ability to escape to the host or other virtual machines residing on it. Given an incident being presented where a virtual machine has been infected with malware, a peer might pose the question of what action might be taken if this malware did indeed escape the virtual environment and infect other virtual machines on the host, or the host itself.

Outcome

During the M&M, all participants should actively take notes. Once the M&M is completed, the presenting individuals should take their notes and combine them into a final report that accompanies their presentation materials and supporting data. This reporting should include a listing of any points which could have been handled differently, and any improvements that could be made to the organization as a whole, either technically or procedurally. This report should be attached the case file associated with the investigation of the incident.

Additional Tips

Having organized and participated in several of these conferences and reviews of similar scope, I have a few other pointers that help in ensuring they provide value.

  • M&M conferences should be held only sporadically, with no more than one per week and no more than three per month.
  • It should be stressed that the purpose of the M&M isn’t to grade or judge an individual, but rather, to encourage the culture of learning.
  • M&M conferences should be moderated by someone at a team lead or lower management level to ensure that the conversation doesn’t get too heated and to steer questions in the right direction.
  • If you make the decision to institute M&M conferences, it should be a requirement that everybody participates at some point, either as a presenter or a peer.
  • The final report that is generated from the M&M should be shared with all technical staff, as well as management.
  • Information security professionals, not unlike doctors, tend to have big egos. The first several conferences might introduce some contention and heated debates. This is to be expected initially, but will work itself out over time with proper direction and moderation.
  • The M&M should be seen as a casual event. It is a great opportunity to provide food and coordinate other activities before and after the conference to take the edge off.
  • Be wary of inviting upper management into these conferences. Their presence will often inhibit open questioning and response and they often don’t have the appropriate technical mindset to gain or provide value to the presentation.

It is absolutely critical that when initiating these conferences, it is done with care. The medical M&M was actually started in the early 1900s by a surgeon named Dr. Ernest Codman at Massachusetts General Hospital in Boston. MGH was so appalled that Dr. Codman suggested that the competence of surgeons should be evaluated that he eventually lost his staff privileges. Now, M&M is a mainstay in modern medicine and something that is done in some of the best hospitals in the world. I’ve seen instances where similar types of shunning occur in information security when these types of peer review opportunities are suggested. As information security practitioners it is crucial that we are accepting of this type of peer review and that we encourage group learning and the refinement of our skills.

References:

  • Campbell, W. (1988). “Surgical morbidity and mortality meetings“. Annals of the Royal College of Surgeons of England 70 (6): 363–365. PMC 2498614.PMID 3207327.
  • Arad, Uzi (2008). Intelligence Management as Risk Management. Paul Bracken, Ian Bremmer, David Gordon (Eds.), Managing Strategic Surprise (43-77). Cambridge: Cambridge University Press.
  • Heuer, Richards J., Jr. “Limits of Intelligence Analysis.” Orbis 49, no. 1 (2005)

4 Ideas for Operationalizing Honeypots

I’ve always thought that the concept of a honeypot was one of the most fascinating things in information security. If you aren’t familiar with honeypots, they are basically traps used to detect or deter attackers on a network. They typically come in two forms; low interaction and high interaction. A low interaction honeypot is software that emulates a set number of services that may run on a computer. When an attacker connects to a low interaction honeypot, he/she will be able to interact with that service on a limited basis, and that interaction will be logged. A high interaction honeypot is more robust and emulates all aspects of an operating system. This is most often a deployed operating system running a number of legitimate services with an extensively level of logging enabled. The thing both of these implementation methods have in common is that the honeypot doesn’t actually contain real data. Should an attacker compromise either type of honeypot, there is no real direct risk of critical data being exposed when deployed properly.

Almost every single honeypot implementation I’ve seen deployed is for research purposes. There isn’t anything wrong with a research honeypot, after all, I run a couple myself (at home) and have learned a lot from it. However, I think there is a lot of operational value that can be gained from deploying honeypots in production environments. I wanted to discuss, at a high level, a few of these strategies and the benefit that can be gained from them.

Honeypots for Prevention

There has been a fair amount of talk recently about security mechanisms designed to drive up the cost of exploiting a network by increasing the time it takes to do so. As a matter of fact, Adobe’s Senior Directory of Product Security and Privacy, Brad Arkin, even recently said that “My goal isn’t to find and fix every security bug. It’s to drive up the cost of writing exploits. We invest a lot of time in building up mitigations that increase the cost and complexity of writing exploits that will become reliable.” Of course, Arkin was referring to the exploitation of software, but the concept still applies to the network side of the house. I’m still a firm believer that your detection capability is still the most important because prevention eventually fails, but if you can drive up the cost of exploiting a network this has the potential to deter some attackers. At a minimum deterring attacks of opportunity can be achieved if you can increase the time cost of exploiting a network, and this may even work to deter attacks of choice as well.

Honeypots can do this by adding to the frustration factor. I see a couple of ways this can be done. The first of which is to utilize a  large number of low interaction honeypots with varying configurations. The important thing here is to vary their configurations as much as possible in order to prevent an attacker from characterizing them and automating them out of their window of visibility. For instance, if you deploy twenty honeypots and they all have ports 22, 80, and 3306 open and all provide the same responses to banner grabs, an attacker is going to be able to correlate this pretty quickly and will simply scan and exclude those hosts from his list of potential targets. The other method for preventive use is to deploy a significant number of high interaction honeypots. This requires a significant time investment, but the right configuration can cause an attacker to waste a significant amount of time in the right places. Again, this strategy isn’t going to prevent aggressive adversaries from reaching their goal, but it will drive up the time cost of lesser determined foes.

Honeypots for Attack Sense and Warning

This is the sacrificial lamb approach to honeypot deployment. In this scenario, honeypots are deployed based upon trust zones within your network. There are different strategies for outlining trust barriers but on a simple network you might define a low trust zone within a wireless or user space network segment, a medium trust zone in a DMZ, and a high trust zone within a server farm network segment. In that sort of topology, all three zones would contain honeypots configured with security comparable to the next step lower. The idea here is that the honeypot should be slightly more vulnerable to attack than everything else in the zone that it is currently in. This configuration provides value in a couple of ways. First, if a honeypot gets compromised, it will likely serve as a warning that other assets within that trust zone may be compromised soon as well, if they aren’t already. Taking this one step further, it is often logical to assume that if a lower trust zone honeypot becomes compromised, the next highest trust zone may be the next target. Depending on how the network is setup, if a higher trust zone includes a honeypot that gets compromised, it could mean that all of the trust zones below it could also have fallen victim to the adversary. This whole model relies on a lot of assumption, but that is the space AS&W operates in.

Honeypots for Detection Related to Critical Assets

I’m a big fan of target-based IDS deployment where instead of deploying a single IDS to your network perimeter, you user more focused IDS’s with finer tuned rule sets and place them closer to organizationally critical assets. This allows for better use of resources across the board as it usually requires less beefy hardware and ensures your analysts won’t see nearly as many false positives. For instance, if your critical data is housed in SQL servers on a single network segment, then deploy an additional IDS to that segment and only utilize SQL focused signatures there rather than on the perimeter IDS. This also allows you to prioritize IDS sensors so that alerts generated by sensors in high priority areas are given priority when it comes to investigations.

I think the same concept of target-based deployment can be tied to Honeypot deployments in the protection of critical assets. If your organization has prioritized their assets (and they should have), then the general idea behind target based honeypot deployment for the purpose of detection would be to configure and deploy honeypots that are virtually identical to the critical servers. This means that they should be running the same services,  talking to the same hosts, and vulnerable to the same types of attacks. The thought here is that if the critical server gets compromised, then so should the honeypot, and vice verse. This is valuable because it isn’t always feasible to log everything on a production server based upon its volume of traffic. This applies to both host based and network based logging. Utilizing an identically configured honeypot that doesn’t see the same amount of utilization allows you to use more aggressive logging, which may allow you to gain more visibility into an attackers movements. This can provide value in helping you determine exactly how an attacker has compromised a system, what they are utilizing the same for, if there is particular data they may be after, and if they have compromised any other systems on the network.

Reverse Honeypots for Intelligence Collection

Although the concept of a reverse honeypot is a bit radical, it really appeals to me considering the industry I work in. The concept of a traditional honeypot is that in which you fill a pot with honey and hope the attacker gets attracted to the honey and sticks his hand in the pot. A reverse honeypot is where you throw some honey in the direction of a target in such a manner as to leave a trail back to the source. The idea being that the target will notice, follow the trail, see the pot, and stick his hand in. In more practical terms, this means that you would attempt to attack a target elsewhere on the Internet. This attack doesn’t necessarily have to be successful and it may just constitute something as simple as a port scan or something as overt as a DoS attempt. During these attacks, no masking of your source IP address should occur and no third party hop points would be used, thus meaning that the target would see your true IP address when reviewing logs of your attack on his network. Given the nature of your target, this may result in his curiosity being peaking and him reciprocating your attack back at you in another form. Of course, within your network you have several vulnerable honeypots of varying interaction levels waiting for the target.

This type of honeypot is solely for the purpose of target based intelligence gathering, but has the potential to be very effective. First and foremost, should the target scan or attack your network you should be able to capture some of the tools, techniques, and procedures (TTPs) that he is using. This type of intelligence can help in recognizing, characterizing, or attributing other computer network exploitation activity to this attacker and may also lend to better detection techniques in the future. One more added value which is incredibly attractive in the modern threat landscape is the identification of hops points. Although you very purposefully did not mask your true source IP address, the attacker may choose to do so. It’s incredibly common for attackers to compromise other hosts elsewhere on the Internet to launch their attacks from, but it’s also common that they will reuse these same hop points for an extended amount of time. If you can identify these hop points then you can use that information to attribute the attacks to a particular operator or group. This is extremely valuable. Of course, this type of activity should be done from non-production networks, because it’s very possible that you might lure an attacker into launching a large scale DDoS attack on your network 10,000 bots strong.

Conclusion

I think there is a lot of room for operationalizing honeypots in production environments. The major factors prohibiting this are a lack of research in this area and a lack of production-grade tools for implementing these techniques. Unfortunately, we are still in a time that IDS is having trouble gaining traction because of the cost it entails, so a future where honeypots can be deployed for the purpose of enhancing network security seems far off. Don’t be surprised however, if you seen a job posting five years down the road for “Honeypot Administrator”. I know I’d have one if I could.

NSM Collection vs. Detection

I was going back through some old bookmarks when I stumbled upon on a post by Richard Bejtlich from 2007 entitled “NSM and Intrusion Detection Differences“. In this article, Richard discussed the concept of ‘immaculate collection’ versus ‘immaculate detection’. Richard’s article references IDS developers desiring immaculate detection while NSM practitioners typically vie for immaculate collection. Given this, I posed the following question to several of my colleagues: Which is more important, collection or detection?

 

The question itself is open to a bit of interpretation, but my group was split about 60/40 favoring collection over detection. I tend to agree with that majority, although the minority had some valid points as well.

 

Those favoring detection argued that a mountain of data, no matter how eloquently collected, is useless without some level of detection capability. Additionally, most in this camp agreed that your detection capability shapes how you perform collection. A few even made the point that they considered collection to be a function of network operations, and not NSM. I can’t disagree with the first of these arguments, but I’m opposed to the other two. I’ll address the argument of whether or not detection shapes collection here.

 

When I think about NSM, I typically think of it in three phases: collection, detection, and analysis. Collection is the gathering and parsing of relevant network security data, and it often performed by a combination of hardware and software. Detection is the process of finding anomalies in collected data that may represent a potential intrusion. Detection is most often done by software, but can be done by humans to a lesser extent. Analysis is the review and investigation of alert data generated during detection. Analysis is typically (and most effectively) done by humans.

 

 

Phases of Network Security Monitoring

 

 

The key takeaway from these three phases is that they form a cycle rather than a beginning to end process. Collected data feeds the detection capability, and the alert data generated from detection feeds the analysis process. What makes this process cyclical is that the investigation and research performed during the analysis process is used to define and shape what data you are collecting.

 

That said, I argue that collection is the most important phase of network security monitoring for a couple of reasons:

 

Detection Depends on Collection

Abraham Lincoln was quoted in saying that if you were to give him six hours to chop down a tree, he would spend the first four hours sharpening his ax. This analogy fits perfect here, because no matter how much thought you put into your detection tools, they are utterly useless if they aren’t digesting the right data. That nice beefy Snort sensor might just be wasting cycles if you’ve placed it on the wrong side of your firewall. Detection fails if collection isn’t done well.

 

Analysis Also Depends on Collection

I hate using the needle in the haystack analogy, but if the hay is covered in manure then you sure aren’t going to want to  spend all of that time digging through it. A human analyst interprets alert data provided by a detection mechanism and then goes out and collects more data in an effort to support his/her investigation. If this data isn’t being collected in an easily retrievable and digestible format then analysis fails. An IDS signature might tell me that a potential attacker is attempting SQL injection on my public facing web server but if I’m not collecting PCAP data and my web server/database logs aren’t accessible then I’m going to have a really hard time finding out if the attack is actually successful.

 

Analysis Feeds Collection Moreso than Detection

I’ve served in the role where I’m the guy creating the detection tools and also in the role where I’m the guy analyzing the alerts generated by the detection tools. It is absolutely true that in some cases collection software/hardware is designed and configured in such a way that it provides data in the appropriate format to a detection tool. This might lead someone to the conclusion that it is detection shaping the collection, but that argument is only made seeing a narrow view of the entire thought process. It is actually the analysis of previous alert data that typically has identified the need for the detection tool that is being created. Remember that detection is most often a task performed by software and it is analysis that is performed by individuals. Software doesn’t identify needs, people do.

 

 

Again, I think this is one of those questions that may or may not have a right answer, but for my two cents, if you gave me six hours to find the bad guys, I’d spend the first four making sure I collected the right data.