How to find and eliminate blind spots in the cloud


Visibility in the cloud is an important but difficult problem to solve. Poor visibility can lead to all kinds of security risks, from data loss to credential abuse to cloud misconfigurations. This is one of the biggest challenges facing CSOs today as they seek to adopt cloud technologies. In an April 2020 survey from Enterprise Strategy Group, researchers found that 33% of respondents believed that a lack of visibility into the activity of the infrastructure hosting their cloud-native applications was the biggest challenge with securing these applications.

The depth of visibility differs between cloud providers, and each has its own positives and negatives. This article covers some of the logging and visibility options offered by Amazon Web Services (AWS) and Google Cloud Platform (GCP), and highlights their blind spots and how to eliminate them.

Know and understand your visibility options

Cloud providers usually offer some sort of default logging or monitoring at no additional cost, but that’s never enough to get enough visibility into what’s going on in your organization. They also provide additional paid services that allow you to gain more visibility into your environments for a variety of use cases. Since each cloud provider does things slightly differently, the blind spots and lack of visibility differ from provider to provider.

The essential control plan

The “control plane” offered by a cloud provider is essentially what handles “cloud operations” or API calls. The control plane should be accessible from anywhere on the internet and this is what allows users to interact with the cloud and the resources within it. For example, API calls made through the AWS Command Line Interface (CLI) are routed through the AWS control plane, allowing you to perform actions such as launching a new Amazon Elastic Compute Cloud instance (EC2). In GCP, the control plane handles things like calls made through the “gcloud” CLI.

Visibility into API calls made in your environment is extremely important, which is why many free default logging services are provided. These include CloudTrail Event History in AWS (a 90-day history of API calls made in your account) or something like Activity Logging in GCP, which provides a broad overview of activity in a project. These default offerings have some shortcomings, however – without additional configuration, you’ll miss out on critical items.

To monitor the cloud control plane, you should use the built-in services provided by your cloud provider, but don’t stop there. These logs should then be exported to an external Security Information and Event Management (SIEM) solution, such as Splunk, for further analysis and alerting. These services include things like configuring AWS CloudTrail for each region and for each type of event in your organization or applying GCP audit logs to all supported services at the organization level. This will help give you the additional information needed to make more informed security decisions.

But what about the network and the hosts?

When it comes to activity within your cloud network or hosts within that network, there’s usually no default free tier to gain visibility. There may be default logging on your hosts, like bash history, but nothing aggregates those logs and gives you access to all your hosts through a unified interface.

There are many integrated and third-party offerings available to gain visibility into activity on your network and on your hosts. Some cloud workload protection platforms offer host-based monitoring and visibility that lets you detect and prevent threats in real time. Other offerings, such as AWS Virtual Private Cloud (VPC) traffic mirroring or GCP packet mirroring, can help you capture all packets within your cloud network. VPC flow logs are another useful tool for network visibility, and offerings like AWS GuardDuty can monitor these flow logs, along with DNS logs and CloudTrail logs, to detect threats and unusual activity in your environment.

Pain points and blind spots

Regardless of the monitoring and visibility options available, even if you use all of your cloud provider’s integrated services (and even some third-party services), there are probably pieces of the puzzle missing.

The ins and outs of data-level events

Data-level events differ from control plane or management plane events because they are actions performed on specific data rather than a cloud resource. For example, AWS Simple Storage Service (S3) data events record object activity in an S3 bucket, providing information about who is interacting with which objects. Without additional configuration (and therefore additional cost), these types of events are not logged. Some services offer data-level event logging, such as S3 bucket access logs, but others, such as the AWS EBS Direct APIs, do not.

To ensure that you don’t lose information about data-level events, you should enable them whenever possible. Where this is not possible, we recommend that you deny all users access to these APIs at the data level.

Undocumented APIs

Each cloud provider publishes documentation for their supported APIs, allowing you to identify what is possible in their specific cloud. The problem is that there are often undocumented APIs supported by each cloud provider that can be exposed when they shouldn’t be. Often these undocumented APIs are used to improve the user experience, such as providing good UX through the web console provided by the vendor. Since they are undocumented and generally not intended for direct access, they are not logged where other documented APIs are logged. This lack of logging could allow an attacker to perform certain actions in your environment without you knowing they have done so.

To prevent undocumented APIs from being used maliciously in your environment, it’s important to grant permissions at a granular level. That is, don’t grant permissions using wildcards (like using a “*” in AWS), don’t use “managed” permission sets (as they are often too permissive) and regularly review the permissions you grant to your users. ensure they are properly limited. By allowing list permissions at a granular level, you can avoid much of the risk that undocumented APIs expose.

Pre-GA Services

Pre-GA (general availability) services are another potential cloud blind spot. This may include alpha, beta, gamma, pre-production or similar services. These services are often blocked to the public and only allow access to listed users, but they are often made available to the public while still in pre-GA status.

An example of how to access pre-GA services are the commands available through the “beta” and “alpha” groups of the “gcloud” CLI, accessible with “gcloud beta ” and “gcloud alpha ,” respectively. Just because they’re pre-GA doesn’t mean they’re unlogged, but that’s where you may come across services that don’t yet support logging and have potential You can prevent access to many of these beta and alpha services in GCP by denying users access to enable new APIs in their projects and only giving them access to a list of trust.

The principle of least privilege

Permissions should be set at a granular level so that they only grant access to necessary services and APIs. These risks often arise when wildcards or managed permission sets are applied in an environment where you cannot be 100% sure of every service and action you have granted to your user. When you set permissions granularly, you can ensure that nothing unexpected will be accessible to your users.

Without taking advantage of the logging and visibility offerings provided by cloud providers, you can quickly lose insight into what is really happening in your environment. Visibility gaps can still exist even with the appropriate services enabled, as we discussed with Data Level Events, Undocumented APIs, and Pre-GA Services, so it is necessary to follow the principle of least privilege to Avoid granting access to services and APIs that are beyond your oversight capabilities.

You have a role to play

It may seem like the cloud provider’s responsibility not to release undocumented pre-GA APIs or services, but until they do, it’s your responsibility to delegate permissions and access so that these APIs do not expose your environment to undue risk.

To learn more, visit us here.

Copyright © 2022 IDG Communications, Inc.


Comments are closed.