Deploying Intel-Owl on GKE
Scaling the scalable
‘Intel Owl’ is a one-stop destination for all your threat intelligence needs. This application, itself was designed on the idea of scalability and provides docker configurations for the same.
What if, one needs to deploy it for their organisation, which actively performs threat intelligence, and need high performance application for the same?
Well, Intel Owl deployed over GKE (Google Kubernetes Engine) might be able to perform well for that!!
NOTE: This is for versions prior to v2 of Intel Owl and specific to Google Kubernetes Engine.
For latest and more generic implementation, see https://gitlab.com/mostwanted002/intel-owl-gke
- A GCP (Google Cloud Platform) account.
- Kubernetes Engine, Service Account Creation, and Cloud SQL API, Cloud Repository accesses, and Cloud SQL Admin API enabled.
dockerinstalled on a host.
- Internet, some coffee and common sense 🙂
Phase 1: Preparing Infrastructure
- Login to your GCP Console and head over Kubernetes Engine → Clusters, and create a cluster according to your/company’s needs.
- While the cluster is being prepared and provisioned, head over to Storage→SQL and create a PostgresQL Instance.
- And while both the above steps are being processed, clone IntelOwl and
to your host with
kubectlinstalled. Extract the latter files into
- After all GCP Resources are provisioned, head over to APIs & Services→Library and search for Cloud SQL Admin API, and see if it’s enabled. If not, enable the API.
- Go to your Cloud SQL instance, and add a database user and password which will be used to access database by IntelOwl. After that, create a database, that’ll be used by IntelOwl. Keep a note of these, as they’ll be mentioned in Secrets file
- Head over to IAM & Admin→Service Accounts and create a service account, with Role of Cloud SQL Client. We’ll be using this service account Cloud SQL Proxy to make our web app’s connection secure to database.
Phase 2: Preparing deployment files and configuration
Copy all the files from
gcloudapplication, and set your project. (Quick Guide)
dockerto be able to push images to Cloud Repository by running the following command:
gcloud auth configure-docker
Fetch Cluster credentials and configure
gcloud container clusters get-credentials <CLUSTER_NAME> --zone/region <ZONE/REGION>
Change directory into
IntelOwlfolder, and run the following command to configure
sed -i -e 's/server uwsgi/server backend-webapp-service.default.svc.cluster.local/g' ./configuration/intel_owl_nginx_http
To set your API Keys, make the desired changes in
kustomization.yaml. Set DB_HOST to localhost. Also change the DB_NAME, DB_USER and DB_PASSWORD accordingly. (Similar to Deployment Preparation ).
Phase 3: Deployment
Run the following script to build, tag and push container images to Cloud Repository:
By default, images will be pushed to
gcr.io. To change it, modify
[build-tag-push.sh](http://build-tag-push.sh)accordingly. And use the same modifications in further steps.
Run the next command, to set image location in
.yamldeployment files. Make sure to change
<PLACE_YOUR_PROJECT_ID_HERE>with your Project ID.
sed -i -e 's/<project-id>/<PLACE_PROJECT_ID_HERE>/g' *.yaml
First, we’ll create Secrets, (ENV Variables, and Cloud Proxy Credentials). Run
kubectl apply -k .to create ENV Variable secrets. After Secrets has been created, run
kubectl get secretsand copy the name of secret that begins with
env-app-secrets-. Run the following command to place this value in deployment. Replace the
env-app-secrets-xxxxxxwith the value you copied.
sed -i -e 's/<PLACE_SECRETS>/env-app-secrets-xxxxxx/g' backend-webapp.yaml
Run the script
secrets-for-cloudproxy.shto create Secrets, to be used by Cloud SQL Proxy Container, as following:
./secrets-for-cloudproxy.sh <SERVICE_ACCOUNT_EMAIL> <SECRETS_NAME>
<SECRETS_NAME>with a suitable name. Use this name in next commands to configure it in
sed -i -e 's/<YOUR-SA-SECRET>/<SECRETS_NAME>/g' backend-webapp.yaml sed -i -e 's/<YOUR-SA-SECRET-VOLUME>/<SECRET_NAME>-volume/g' backend-webapp.yaml
Head over to SQL Instance and copy the Instance Connection name, and use it in this command, in place of <SQL_INSTANCE> to configure Cloud SQL Instance connection details in
sed -i -e 's/<INSTANCE_CONNECTION_NAME>/<SQL_INSTANCE>/g' backend-webapp.yaml
Also, use the port 5432 (default for PostgresQL instances) in place of <SQL_PORT> in this command:
sed -i -e 's/<DB_PORT>/<SQL_PORT>/g' backend-webapp.yaml
Almost there now!
Deploy the broker,
kubectl apply -f ./rabbitmq-daemon.yaml
Deploy the services:
kubectl apply -f ./services.yaml
Deploy the backend:
kubectl apply -f ./backend-webapp.yaml
Wait for couple of minutes, before it deploys. After the pods are running, proceed.
Deploy the frontend:
kubectl apply -f ./frontend-deployment.yaml
Voila!! To get the IP Address for reaching the app, run
kubectl get svc and Public IP will be available for Load Balancer Service.
To create admin user, first get the list of pods with
kubectl get pods -l app=backend-webapp. Then run
kubectl exec -it <ANY_POD_NAME_FROM_PREVIOUS_COMMAND> -c uwsgi python manage.py createsuperuser , and follow the instructions to create an admin. Login and Enjoy!