By Brandon Stultz.
Every day, new vulnerabilities are discovered in the software critical to the function of the modern world. Security analysts take apart these new vulnerabilities, isolate what is necessary to trigger them and write signatures to block any exploits targeting them. For Snort, these signatures are called Snort rules — and they’re extremely versatile. They can access specific network service fields, locate a vulnerable parameter and scan that parameter for the presence of an exploit. They can also leverage numerous rule options to traverse protocols and file formats. Written well, these rules can have high efficacy and performance with few or no false positives. This approach to defense is very good at protecting networks from known threats, but what if the threat is unknown? What if a vulnerability is discovered, an exploit for it is written, and the security community has no knowledge of it? We need another approach to defense that doesn’t require prior knowledge of the attack to function. Over the past year at Cisco, we have been prototyping and building this new approach into a new detection engine for Snort. Today, I am proud to announce we are open-sourcing this engine to the community in the latest Snort 3 release (version 3.1.82.0). This new detection engine is called “SnortML.” SnortML is a machine learning-based detection engine for the Snort intrusion prevention system. At a high level, there are two components to this new detection engine. The first component is the snort_ml_engine itself, which loads pre-trained machine learning models, instantiates classifiers based on these models and then makes the classifiers available for detection. The second is the snort_ml inspector, which subscribes to data provided by Snort service inspectors, passes the data to classifiers, and then acts on the output of the classifiers. Currently, the snort_ml_engine module only has one model type, namely the http_param_model, but we plan on building other models in the future. This http_param_model is used for classifying HTTP parameters as malicious or normal. Once the snort_ml_engine loads the http_param_model, it can be used in the snort_ml inspector to detect exploits. The inspector subscribes to the HTTP request data provided by the HTTP inspector through the publish/subscribe interface. It then passes this data (HTTP URI query and optionally HTTP POST body) to a binary classifier based on the http_param_model. This classifier then returns the probability that it saw an exploit. Based on this probability, SnortML can generate an alert, similar to a Snort rule alert, which can be configured to block malicious traffic. Now that you know how the machine learning engine works, let’s get into how the models work. SnortML models are designed to be extremely flexible, much like their Snort rule counterparts. To that end, we based our models and our inference engine on TensorFlow. The TensorFlow project is a free and open-source library for machine learning and artificial intelligence. Any TensorFlow model can be a SnortML binary classifier model so long as it satisfies three conditions, namely, the model must have a single input tensor and a single output tensor, the input and output tensor types must be 32-bit floating point, and finally, the output tensor must have only a single element. We plan on adding other model types in the future (including multiclass classifiers), but right now, this is the only model type currently supported. The SnortML engine uses TensorFlow through a support library we call LibML. The LibML library handles loading, configuring and running machine learning models for Snort. It also includes the XNNPACK accelerator needed to run CPU-bound models at line rate. The easiest way to build a SnortML model is to use the TensorFlow Keras API. If you are new to machine learning, don’t worry, Keras is a simple but powerful deep-learning framework that allows you to build neural networks and train them in a few lines of Python. To get started, import the following:
import os
import numpy as np
import tensorflow as tf
from tensorflow.keras import layers
from urllib.parse import unquote_to_bytes
We are going to train our example model on just two samples, but a real production model would use far more:
# Example data
data = [
{ 'str':'foo=1', 'attack':0 },
{ 'str':'foo=1%27%20OR%201=1%2D%2D', 'attack':1 }
]
The next thing we need to do is prepare our data. SnortML models expect input data to be zero-padded which is what we are going to do here:
# Prepare Data
maxlen = 1024
X = []
Y = []
def decode_query(str):
return unquote_to_bytes(str.replace('+',' '))
for item in data:
arr = decode_query(item['str'])[:maxlen]
arrlen = len(arr)
seq = [0] * maxlen
for i in range(arrlen):
seq[maxlen - arrlen + i] = arr[i]
X.append(seq)
Y.append(item['attack'])
Now, we need to construct a neural network that can classify our data. This example uses a simple LSTM (Long Short-Term Memory) network, but other combinations of layers available in Keras work here as well. LSTM is a type of neural network that is keenly suited to identify patterns in sequences of data, such as the sequences of bytes in HTTP parameters. To translate the bytes on the wire to tensors that the LSTM can accept, we can place an embedding layer in front of it. Embedding layers are a kind of association layer, they can learn relationships between input data (bytes in our case) and output those relationships as tensors that the LSTM neurons can accept. Finally, we will converge the output of our LSTM neurons to a single output neuron with a Dense layer. This will serve as the output of the neural network.
#
# Build Model (Simple LSTM)
#
model = tf.keras.Sequential([
layers.Embedding(256, 32, input_length=maxlen, batch_size=1),
layers.LSTM(16),
layers.Dense(1, activation='sigmoid')])
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
model.summary()
Now for the fun part — let’s train this neural network: # # Train Model # model.fit(np.asarray(X).astype(np.float32), np.asarray(Y).astype(np.float32), epochs=100, batch_size=1) Training output: Model: "sequential" ----------------------------------------------------------------- Layer (type) Output Shape Param # ================================================================= embedding (Embedding) (1, 1024, 32) 8192 lstm (LSTM) (1, 16) 3136 dense (Dense) (1, 1) 17 ================================================================= Total params: 11,345 Trainable params: 11,345 Non-trainable params: 0 ----------------------------------------------------------------- Epoch 1/100 2/2 [==============================] - 1s 129ms/step - loss: 0.6910 - accuracy: 0.5000 ... Epoch 100/100 2/2 [==============================] - 0s 134ms/step - loss: 0.0208 - accuracy: 1.0000
As you can see above, the accuracy of our network increased, and the loss dropped. These metrics show that the neural network learned to differentiate attack from normal in our example dataset. Now, let’s save this model to a file so we can load it in Snort:
#
# Save Model
#
converter = tf.lite.TFLiteConverter.from_keras_model(model)
snort_model = converter.convert()
with open('snort.model', 'wb') as f:
f.write(snort_model)
Now that we have a model file, we can run it against PCAPs with Snort 3:
$ snort -q --talos \
--lua 'snort_ml_engine = { http_param_model = "snort.model" };' \
--lua 'snort_ml = {};' \
-r test.pcap
##### test.pcap #####
[411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1)
#####
If you have Snort 3 built with debug messages enabled, you can even trace the ML engine input and output.
$ snort -q --talos \
--lua 'trace = { modules = { snort_ml = { all = 1 } } };' \
--lua 'snort_ml_engine = { http_param_model = "snort.model" };' \
--lua 'snort_ml = {};' \
-r test.pcap
P0:snort_ml:classifier:1: input (query): foo=1' OR 2=2-- P0:snort_ml:classifier:1: output: 0.971977 P0:snort_ml:classifier:1: <ALERT> ##### test.pcap ##### [411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1) #####
P0:snort_ml:classifier:1: input (query): foo=1' OR 2=2--
P0:snort_ml:classifier:1: output: 0.971977
P0:snort_ml:classifier:1: <ALERT>
##### test.pcap ##### [411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1) #####
##### test.pcap #####
[411:1:0] (snort_ml) potential threat found in HTTP parameters via Neural Network Based Exploit Detection (alerts: 1)
#####
Notice that even with variations in the SQL injection attack above, we still detected it. For years, we had dreamed about tackling the zero-day problem, providing coverage for attacks that were like those we had seen before, but targeting different applications or parameters. Now, with SnortML, this dream is becoming a reality. You can find the SnortML and LibML code here. Feel free to join the conversation on our Discord or on the Snort users mailing list if you have any questions or feedback.
from Snort Blog https://ift.tt/gW60LlU
via IFTTT
No comments:
Post a Comment