DevSecOps

Definitions

1. A holistic secure by design approach to the DevOps pipeline aligned with People, Process, Tools.

2. Bring 3 teams together under a common goal of providing secure, consistent, faster path to production. Thus enhancing the organization's security posture¹ (Secure software supply chains).

3. Add Secure development practices from the beginning into the DevOps and Agile practices². Ensuring you have a secure delivery of objectives from beginning to end.

Practices: SAST, DAST, RASP, IAST: static, dynamic, runtime, interactive application security testing
Who is? A professional with knowledge of the development lifecycle, operations & systems engineering with a security focus – Bridging between teams to make sure both ends work with security best practices.
Task Securing Infrastructure, DevOps profesionals become inherentely part of the Blue team.
Skills CI/CD (Jenkins, CircleCI, Github), Bash, Configuration Management tools (Ansible, Puppet, Salt, Chef, Packer), Python

Key ideas

Continuous Delivery: Goal clarification and mechanism (pipelines) to deliver on those goals
Optimise to make software is ready release, create a release. Security is part of the requirement.
Deployment pipeline must have complete traceability (end to end story)
Pipeline should be fast, from commit to release in under an hour.
Making secure systems releasable: regulations & compliance, auditable & traceable, provenance (control dependencies, trusted repos, IaaC, only automated access to production), testing (pentesting, secure behaviours, static analysis, common vulnerabilities), culture (preferred behavior, security is everyone's responsibility, train others to adopt security mindset)

Landscape

Exploding cloud infra services
Increased DevOps velocity and application delivery
Many teams spinning infrastructure and deciding on security
Stricter compliance requirements
Large number of security tools to choose from
Nover attack vectors

Cloud Native Security

The cloud native DevOps Pipeline

Developers ⏩  Code Repo ⏩  Build Automation ⏩  ☁️📦
	commit          build		push wrk: containers, serverless
			⏬				
              Container Registry

Avoid slow down of DevOps via security, Automation in CI/CD pipeline, Plethora of tools, Linux heavy
Vulnerabilities: 1. in FOSS & own code, 2. Vulnerable Container Images, 3. 3rd party app, SQL injection (lambda), malicious files (s3?)

Need to protect from exploitation of vulnerabilities, data exfiltration: SQL inj, remote command exec, illegal file access, malicious payloads, URL redirects – while keeping minimal impact in performance and dev streams with visibility and multi-platform support.

The DevOps pipeline

Cloud applications, powered by an automated CI/CD pipeline and driven by a DevOps organization, accelerate the speed at which your business delivers new applications and features. Unfortunately, that also means your DevOps pipeline can inadvertently and rapidly introduce security vulnerabilities without proper consideration and management. In a shared responsibility model, you are responsible for securing your code and the tools you use to deliver applications to the cloud. The servers and serverless assets that make up your DevOps toolchain must be protected, including code repositories, Docker image registries, Jenkins orchestration tools, etc. Beyond securing your CI/CD pipeline, you can—and should—leverage CI/CD automation processes to shift security left, by integrating security into the code and making it part of the build. This idea of “shifting left” means automated testing against clearly defined security requirements, early and often in the development process, so that new vulnerabilities are caught and remediated before being merged into the larger code tree or introduced into a production service. –

Shared Responsibility Model Explained

Securing your build pipeline

Key part of your agile lifecycle (automated build, integration,testing, deployment process) as it is an attractive target of attack, more so if CD as changes are often put through production.

Attack surface increases by auytomating build and deployment as it includes build environment and toolchain (repositories, build servers, configuration management systems):

By gettting read access: Data, source code, secrets, passwords, API keys might be exfiltrated
By getting write access: Backdooring apps, malware injection, redirect/intercept traffic or destruction of production.
Test system is compromised: May produce an attack path into the pipeline
Pipeline compromised: Loose capability to respond to attacks (push patch or fix)
Insider attacks: Changes must be fully authorized, transparent, and traceable end to end. No bypassing controls without detection.

Define a threath model for a build pipeline: Weaknesses setup controls, gaps in auditing and logs.

Configuration management & build pipelines

Harden run-time config management, build & test envs
The systems that host your source code, artifacts, any server used for integration and delivery, and servers that contain configuration management, deployment and release tools. These are sensitive production systems too. Apply Firewall and network configuration so as not to expose to public. Isolate further using containers and VMs.
Take inventory of steps taken in the cloud
Understand what part of the infra is in the cloud vs. on-premise. Cloud use could potentially expand attack surface. Code repositories are key targets if used in larger organizations that could compromise secrets too. Github is a target due to poor usage and in cases the service provider's fault. Secure via strong (multi-factor) authentication, verify private repos, monitor your Github, and code review for sensitive information. Be aware of potential, personal dotfile leakage, and hosted CI configuration (TravisCI, Codeship) – Update access for changing/leaving staff and consider a Single-sign-on investment.
Harden build, CI/CD tool chains
Most tools aren't secure by default. Understant the tool you are using and tighten access, setup trust releationships. Isolate pipelines for different teams to prevent escalation. Keep tools updated and patched but test them first (dependency issues).
Secure access to config management tools
Any person with access to your Ansible, Puppet or Chef tools can add accouts, change permissions, auditing policies, install sw, change firewall rules thus configured them safely. Access should be controlled and audited.
Keys and secrets
Make sure that secrets used to provision, run, deploy, start up aren't hardcoded in scripts or plaintext in code.
Lock source and binary repos
Audit access to repositories, prevent unidentified/anon users or shared credentials user access. Impleement rules, read access first and limit write access to avoid malicious code being checked in. Cached third party libraries are kept in binary repos that could be used to inject malware. Verify signatures during download and when bundling and sign the binaries/artifacts after.
Chat platform security (ChatOps)
Slack, Mattermost, hubot could open security risks. Chatbots can track ops status and report back to message channlels or be used to perform operations related to build, tests, release and deployment via simple commands. Verify access to these channels, commands, rooms, and what info is available to third parties. Also understand how the bots are set up, where they run and their scripts. Chat ops scripts are like operations code: review and check it in repo. If using cloud providers, verify their controls and privacy policies. – Also make sure sensitive information is not shared in message rooms. Add MFA to access chat platforms for users and set permissions for bots.
Scheduled log review for config management, build , & test envs
Part of same ops workflow as production. Review logs and tools so information is complete and trace from end to end, check-in to deployment. Logs must be immutable, rotated, and backed up.
Ephemeral servers, for building and testing (Phoenix servers)
Using configuration management tools, Docker, Vagrant, Terraform to stand, set up, patch, tear down "build slaves" and test servers as needed. Build and test boxes are disposable and ephemeral, reducing attack surface, testing config management workflow, and increase confidence before hardened production.
Avoid test data from production
Don't use test data derived from production data, handle snapshots carefully and reliable so sensitive info like PIIE aren't used in test. Audit and review to demostrate compliance.
Build pipelines should be monitored in a similar way that production
Ops security must be also in the tools, infra, includes vulnerabiltity scanning, IDS/IPS, run-time monitoring. File integrity checks can be used to review unauthorized changes to config and data.
Keeping secrets secure but accessible
As they are needed by members of the team and passing them around is a recipe for disaster. Also maintaining these in platforms such as Github can easily led to leaking Slack API tokens (eavesdropping) etc. There are scanners such as Gitrob, TruffleHog to check for sensitive info in repos. Other tools can encrypt secrets as they are being checked in (see git-secrt, git-crypt, Blackbox) – take advantage of precommit hooks to add checks for passwords and keys in code using projects such as talisman, git hound, and awslabs/git-secrets. For config tools accessing secrets review Ansible Vault, Chef vault, hiera-eyaml. A general purpose secret manager across all tools and apps
1. store and encrypt sensitive info,
2. restrict access and audit, enforce authentication and fine-grained AC rules.
3. Secure access via API,
4. Handle failover (external), check square/keywhiz, pinterest/knox, lyft/confidant, fugue/credstash, hashicorp/vault

Key questions

Observability: Do we know what's happening across the pipeline
Traceability: What is being deployed? Can we prove it
Confidence: Guarantee that what's being delivered is what it was conceived
Compliance: Important in many industries, e.g., Healthcare, public, banking

  ─────────────────────────────────────────────────────────────────►
  IDEA   >>   CODE   >>   BUILD   >>  DEPLOY   >>   MANAGE   >>   LEARN

  User		 TDD	  Linting       Notary        Prevent
  Stories	 Pair	  Scanning      Immutable     mutation
  	         progr                  images

Use cases:

Lack of visibility: How apps move, what, who and when is responsible
Troubled audits: Prove with data what you are producing
Unified governance: Use same pipeline across the organization
Risk mitigated: Mitigate risk of leakages and reputational damage