Compute

Compute Engine zonal

  • Fast-booting Virtual Machines (VMs) you can rent, on demand
  • Infrastructure as a Service (IaaS)
  • Pick set machines type — standard, highmem, highcpu — or custom CPU/RAM
  • Pay by the second (60 second min.) for CPUs, RAM
  • Automatically cheaper if you keep running it ("sustained use discount")
  • Even cheaper for "preemptible" or long-term use commitment in a region
  • Can add GPUs and paid OSes for extra cost
  • Live Migration: Google seamlessly moves instances across hosts, as needed

Kubernetes Engine (GKE) regional

  • Managed Kubernetes cluster for running Docker containers (with autoscaling)
  • Used to be called "Google Cotnainer Engine" (but still GKE) until Nov, 2017
  • Kubernetes DNS on by default for service discovery
  • No IAM integration (unline AWS's ECS)
  • Integrates with Persistent Disk for storage
  • Pay for underlying GCE instances
    • Production cluster should have 3+ nodes
  • No GKE management fee, no matter how many nodes in cluster, as of Nov 2017

App Engine (GAE) regional

  • Similar to Elastic Beanstalk, Heroku
  • Platform as a Service (PaaS) that takes your code and runs it
  • Much more than just compute — Integrates storage, queues, NoSQL, etc.
  • Flex mode ("App Engine Flex") can run any container & access VCP
  • Auto-scales based on load
    • Standard (non-Flex) mode can turn off last instance when no traffic
  • Effectively pay for underlying GCE instances and other services

Check out my wife's FREE UI designs at uidesigndaily.com

Cloud Functions (GCF) regional

  • Runs code in response to an event — Node.js, Python, Java, Go
  • Functions as a Service (FaaS), "Serverless"
  • Pay for CPU and RAM assigned to function, per 100ms (min. 100ms)
  • Each function automatically gets an HTTP endpoint
  • Can be triggered by GCS objects, Pub/Sub messages, etc.
  • Massively scalable (horizontally) — Runs many copies when needed
  • Often used for chatbots, message processors, IoT, automation, etc.

Storage

Local SSD zonal

  • Very fast 375GB solid state drives physically attached to the server
  • Can stripe across eight of them (3TB) for even better performance
  • DATA WILL BE LOST whenever the instance shuts down
    • But can survive a Live Migration
  • Like all data at rest, always encrypted
  • Pay by GB-month provisioned

Persistent Disk (PD) zonal

  • Flexible, block-based network-attached storage; boot disk for every GCE instance
  • Performance scales with volume size; max way below Local SSD, but still plenty fast
  • Persistent disks persist, and are replicated (zone or region) for durability
  • Can resize while in use (up to 64TB), but will need file system update within VM
  • Snapshots (and machine images) add even more capability and flexibility
    • Pay for incremental ($ and time), but use/delete like full backups
  • Not file-based NAS, but can mount to multiple instances if all are read-only
  • Pay for GB/mo provisioned depending on perf. class; plus snapshot GB/mo used

Cloud Filestore zonal

  • Fully-managed file-based storage
  • "Predictably fast performance for your file-based workloads"
  • Accessible to GCE and GKE through your VPC, via NFSv3 protocol
  • Primary use case is application migration to cloud ("lift and shift")
  • Fully manages file serving, but not backups.
  • Pay for privisioned TBs in "Standard" (slow) or "Premium" (fast) mode
  • Minimum privisioned capacity of 1TB (Standard) or 2.5TB (Premium)

Cloud Storage (GCS) regional multi-regional

  • Infinitely scalable, fully-managed, versioned, and highly-durable object storage
  • Designed for 99.999999999% ("eleven nines") durability
  • Strongly consistent (even for overwrite PUTs and DELETEs)
  • Integrated site hosting and CDN functionality
  • Lifecycle transitions across classes: Multi-Regional, Regional, Nearline, Coldline
    • Diffs in cost & availability (99.95%, 99.9%, 99%, 99%), not latency (no thaw delay)
  • All classes have same API, so can use gsutil and gcsfuse (but beware)
  • Pay for data operations & GB-months stored by class
  • Nearline/Coldline: Also pay for GBs retrieved - plus early deletion fee, if < 30/90 days

Databases

Cloud SQL regional

  • Fully-managed and reliable MySQL and PostgreSQL databases
  • Supports automatic replication, backup, failover, etc.
  • Scaling is manual (both vertically and horizontally)
  • Effectively pay for underlying GCE instances and PDs
    • Plus some baked-in service fees

Cloud Spanner regional multi-regional global

  • "The first horizontally scalable, strongly consistent, relational database service"
    • From 1 to hundreds or thousands of nodes
    • A minimum of 3 nodes is recommended for production environments
  • Chooses Consitency and Partition-Tolerance (CP of CAP theorem)
  • But still high Availability: SLA has 99.999% SLO (five nines) for multi-region
    • Nothing is actually 100%
    • Not based on fail-over
  • Pay for provisioned node time (by region/multi-region) plus used storage-time

BigQuery (BQ) multi-regional

  • Serverless column-store data warehouse for analytics using SQL
  • Scales internally, so it "can scan TB in seconds and PB in minutes"
  • Pay for GBs actually considered (scanned) during queries
    • Attempts to reuse cached results, which are free
  • Pay for data stored (GB-months)
    • Relatively inexpensive
    • Even cheaper when table not modified for 90 days (reading still fine)
  • Pay for GBs added via streaming inserts

Cloud Bigtable zonal

  • Low latency & high throughput NoSQL DB for large operational & analytical apps
  • Supports open-source HBase API
  • Integrates with Hadoop, Dataflow, Dataproc
  • Scales seamlessly and unlimitedly
    • Storage autoscales
    • Processing nodes must be scaled manually
  • Pay for processing node hours
  • Pay for GB-hours used for storage (cheap HDD or fast SSD)

Cloud Datastore regional multi-regional

  • Managed & autoscaled NoSQL DB with indexes, queries, and ACID trans. support
  • NoSQL, so queries can get complicated
    • No joins or aggregates and must line up with indexes
    • NOT, OR, and NOT EQUALS operators not natively supported
  • Automatic "built-in" indexes for simple filtering and sorting (ASC, DESC)
  • Manual "composite" indexes for more complicated, but beware them "exploding"
  • Pay for GB-month of storage used (including indexes)
  • Pay for IO operations (deletes, reads, writes) performed (i.e. pre-provisioning)

Firebase Realtime DB zonal & Cloud Firestore multi-regional

  • NoSQL document stores with ~real-time client updates via managed websockets
  • Firebase DB is single (potentially huge) JSON doc, located only in central US
  • Cloud Firestore has collections, documents, and contained data
  • Free tier (Spark), flat tier (Flame), or usage-based pricing (Blaze)
    • Realtime DB: Pay more for GB/month stored and GB downloaded
    • Firestore: Pay for operations and much less for storage and transfer

Data Transfer

Data Transfer Appliance

  • Rackable, high-capacity storage server to physically ship data to GCS
  • Ingest only; not a way to avoid egress charges
  • 100TB or 480TB versions
  • 480TB/week is faster than a saturated 6Gbps link

Storage Transfer Service global

  • Copies objects for you, so you don't need to set up a machine to do it
  • Destination is always GCS bucket
  • Source can be S3, HTTP/HTTPS endpoint, or another GCS bucket
  • One-time or scheduled recurring transfers
  • Free to use, but you pay for its actions

External Networking

Google Domains global

  • Google's registrar for domain names
  • Private Whois records
  • Built-in DNS or custom nameservers
  • Supports DNSSEC
  • Email forwarding with automatic setup of SPF and DKIM (for built-in DNS)

Cloud DNS global

  • Scalable, reliable, & managed authoritative Domain Name System (DNS) service
  • 100% uptime guarantee
  • Public and private managed zones
  • Low latency globally
  • Supports DNSSEC
  • Manage via UI, CLI, or API
  • Pay fixed fee per managed zone to store and distribute DNS records
  • Pay for DNS lookups (i.e. usage)

Static IP regional global

  • Reserve static IP addresses in projects and assign them to resources
  • Regional IPs used for GCE instances & Network Load Balancers
  • Global IPs used for global load balancers:
    • HTTP(S), SSL proxy, and TCP proxy
    • "Anycast IP" simplifies DNS
  • Pay for reserved IPs that are not in use, to discourage wasting them

Load Balancing regional global

  • High-perf, scalable traffic distribution integrated with autoscaling & Cloud CDN
  • SDN naturally handles spikes without any prewarming; no instances or devices
  • Regional Network Load Balancer: health checks, round robin, session affinity
    • Forwarding rules based on IP, protocol (e.g. TCP, UDP), and (optionally) port
  • Global load balancers w/ multi-region failover for HTTP(S), SSL proxy, & TCP proxy
    • Prioritize low-latency connection to region near user, then gently fail over in bits
    • Reacts quickly (unlike DNS) to changes in users, traffic, network, health, etc.
  • Pay by making ingress traffic billable (cheaper than egress) plus hourly per rule

Cloud CDN global

  • Low-latency content delivery based on HTTP(S) CLB & integrated w/ GCE & GCS
  • Supports HTTP/2 and HTTPS, but no custom origins (GCP only)
  • Simple checkbox on HTTP(S) Load Balancer config turns this on
  • On cache miss, pay origin -> POP "cache fill" egress charges (cheaper for in-region)
  • Always pay POP -> client egress charges, depending on location
  • Pay for HTTP(S) request volume
  • Pay per cache invalidation request (not per resource invalidated)
  • Origin costs (e.g. CLB, GCS) can be much lower because cache hits reduce load

Internal Networking

Virtual Private Cloud (VPC) regional global

  • Global IPv4 unicast Software-Defined Network (SDN) for GCP resources
  • Automatic mode is easy; custom mode gives control
  • Configure subnets (each with a private IP range), routes, firewalls, VPNs, BGP, etc.
  • VPC is global and subnets are regional (not zonal!)
  • Can be shared across multiple projects in same org and peered with other VPCs
  • Can enable private (internal IP) access to some GCP services (e.g. BQ, GCS)
  • Free to configure VPC (container)
  • Pay to use certain services (e.g. VPN) and for network egress

Cloud Interconnect regional multi-regional

  • Options for connecting external networks to Google's network
  • Private connections to VPC via Cloud VPN or Dedicated/Partner Interconnect
  • Public Google services (incl. GCP) accessible via External Peering (no SLAs)
    • Direct Peering for high volume
    • Carrier Peering via a partner for lower volume
  • Significantly lower egress fees
    • Except Cloud VPN, which remains unchanged

Cloud Virtual Private Network (VPN) regional

  • IPsec VPN to connect to VPC via public internet for low-volume data connections
  • For persistent, static connections between gateways (i.e. not for a dynamic client)
    • Peer VPN gateway must have static (unchanging) IP
  • Encrypted link to VPC (as opposed to Dedicated Interconnect), into one subnet
  • Supports both static and dynamic routing
  • 99.9% availability SLA
  • Pay per tunnel-hour

Dedicated Interconnect regional multi-regional

  • Direct physical link between VPC and on-prem for high-volume data connections
  • VLAN attachment is private connection to VPC in one region; no public GCP APIs
    • Region chosen from those supported by particular Interconnect Location
  • Links are private but not encrypted; can layer your own encryption
  • Redundant connections in different locations recommended for critical apps
    • Redindancy achieves 99.99% availability; otherwise 99.9% SLA
  • Pay fee per 10 Gbps link, plus (relatively small) fee per VLAN attachment
  • Pay reduced egress rates from VPC through Dedicated Interconnect

Cloud Router regional

  • Dynamic routing (BGP) for hybrid networks linking GCP VPCs to external networks
  • Works with Cloud VPN and Dedicated Interconnect
  • Automatically learns subnets in VPC and announces them to on-prem network
  • Without Cloud Router you must manage static routes for VPN
    • Changing the IP addresses on either side of VPN requires recreating it
  • Free to set up
  • Pay for usual VPC egress

CDN Interconnect regional multi-regional

  • Direct, low-latency connectivity to certain CDN providers, with cheaper egress
  • For external CDNs, not Google's Cloud CDN service
    • Supports Akamai, Cloudflare, Fastly, and more
  • Works for both pull and push cache fills
    • Because it's for all traffic with that CDN
  • Contact CDN provider to set up for GCP project and which regions
  • Free to enable, then pay less for the egress you configured

Machine Learning / AI

Cloud Machine Learning (ML) Engine regional

  • Massively scalable managed service for training ML models & making predictions
  • Enables apps/devs to use TensorFlow on datasets of any size; endless use cases
  • Integrates: GCS/BQ (storage), Cloud Datalab (dev), Cloud Dataflow (preprocessing)
  • Supports online & batch predictions, prioritizing latency (online) & job time (batch)
  • Or download models & make predictions anywhere: desktop, mobile, own servers
  • HyperTune automatically tunes model hyperparameters to avoid manual tweaking
  • Training: Pay per hour depending on chosen cluster capabilities (ML training units)
  • Prediction: Pay per privisioned node-hour plus by prediction request volume made

Cloud Vision API global

  • Classifies images into categories, detects objects/faces, & finds/reads printed text
  • Pre-trained ML model to analyze images and discover their contents
  • Classifies into thousands of categories (e.g. "sailboat", "lion", "Eiffel Tower")
  • Upload images or point to ones stored in GCS
  • Pay per image, based on detection features requested
    • Higher price of OCR of full documents and finding similar images on the web
    • Sime features are priced together: Labels + SafeSearchm ImgProps + Cropping
    • Other features priced individually: Text, Faces, Landmarks, Logos

Cloud Speech API global

  • Automatic Speech Recognition (ASR) to turn spoken word audio files into text
  • Pre-trained ML model for recognizing speech in 110+ languages/variants
  • Accepts pre-recorded or real-time audio, & can stream results back in real-time
  • Enables voice command-and-control and transcribing user microphone dictations
  • Handles noisy source audio
  • Optionally filter inappropriate content in some languages
  • Accepts contextual hints: words and names that will likely be spoken
  • Pay per 15 seconds of audio processed

Cloud Natural Language API global

  • Analyzes text for sentiment, intent, & content classification, and extracts info
  • Pre-trained ML model for understanding what text means, so you can act on it
  • Excellent with Speech API (audio), Vision API (OCR), & Translation API (or built-ins)
  • Syntax analysis extracts tokens/sentences, parts of speech & dependency trees
  • Entity analysis finds people, places, things, etc. labels them, & links to Wikipedia
  • Analysis for sentiment (overall) and entity sentiment detect +/- feelings & strength
  • Content classification puts each document into one of 700+ predefined categories
  • Charged per request of 1000 characters, depending on analysis types requested

Cloud Translation API global

  • Translate text among 100+ languages; optionally auto-detects source language
  • Pre-trained ML model for recognizing and translating semantics, not just syntax
  • Can let people support multi-regional clients in non-native languages, 2-way
  • Combine with Speech, Vision, & Natural Language APIs for powerful workflows
  • Send plain text or HTML and receive translation in kind
  • Pay per character processed for translation
  • Also pay per character for language auto-detection

Dialogflow global

  • Build conversational interfaces for websites, mobile apps, messaging, IoT devices
  • Pre-trained ML model and service for accepting, parsing, lexing input & responding
  • Enables useful chatbots and other natural user interactions with your custom code
  • Train it to identify custom entity types by providing a small dataset of examples
  • Or choose from 30+ pre-built agents (e.g. car, currency, dates) as starting template
  • Supports many different languages and platforms/devices
  • Free plan has unlimited text interactions and capped voice interactions
  • Paid plan is unlimited but charges per request: more for voice, less for text

Cloud Video Intelligence API regional global

  • Annotates videos in GCS (or directly uploaded) with info about what they contain
  • Pre-trained ML model for video scene analysis and subject identification
  • Enables you to search a video catalog the same way you search text documents
  • "Specify a region where processing will take place (for regulatory compliance)"
  • Label Detection: Detect entities within the video, such as "dog", "flower" or "car"
  • Shot Change Detection: Detect scene changes within the video
  • SafeSearch Detection: Detect adult content within the video
  • Pay per minute of video processed, depending on requested detection modes

Cloud Job Discovery global

  • Helps career sites, company job boards, etc. to improve engagement & conversion
  • Pre-trained ML model to help job seekers search job posting databases
  • Most job sites rely on keyword search to retrieve content which often omits relevant jobs and overwhelms the job seeker with irrelevant jobs. For example, a keyword search with any spelling error returns 0 results, and a keyword search for "dental assistant" returns any "assistant" role that offers dental benefits.
  • Integrates with many job/hiring systems
  • Lots of features, such as commute distance and recognizing abbreviations/jargon
  • "Show me jobs with a 30 minute commute on public transportation from my home"

Big Data and IoT

Google Big Data Lifecycle

  • Ingest
  • Store
  • Process & Analyze
  • Explore & Visualize

Cloud Internet of Things (IoT) Core global

  • Fully-managed service to connect, manage, and ingest data from devices globally
  • Device Manager handles device identity, authentication, config, & control
  • Protocol Bridge publishes incoming telemetry to Cloud Pub/Sub for processing
  • Connect securely using IoT industry-standard MQTT or HTTPS protocols
  • CA signed certificates can be used to verify device ownership on first connect
  • Two-way device communication enables configuration & firmware updates
  • Device shadows enable querying & making control changes while defices offline
  • Pay per MB of data exchanged with devices; no per-device charge

Cloud Pub/Sub global

  • Infinitely-scalable at-least-once messaging for ingestion, decoupling, etc.
  • "Global by default: Publish... and consume from anywhere, with consistent latency."
  • Messages can be up to 10MB and undelivered ones stored for 7 days - but no DLQ
  • Push mode delivers to HTTPS endpoint & succeeds on HTTP success status code
    • "Slow-start" algirithm ramps up on success and backs off & retries, on failures
  • Pull mode delivers messages to requesting clients and waits for ACK to delete
    • Lets clients set rate of consumption, and supports batching and long-polling
  • Pay for data volume
    • Min 1KB per publish/push/pull request (not by message)

Cloud Dataprep global

  • Visually explore, clean, and prepare data for analysis without rinning servers
  • "Data Wrangling" (i.e. "ad-hoc ETL") for business analysts, not IT pros
    • Who might otherwise spend 80% of their time cleaning data
  • Managed version of Trifacta Wrangler - and managed by Trifacta, not Google
  • Source data from GCS, BQ, or file upload - formatted in CSV, JSON, or relational
  • Automatically detects schemas, datatypes, possible joins, and various anomalies
  • Pay for underlying Dataflow job, plus management overhead charge
  • Pay for other accessed services (e.g. GCS, BQ)

Cloud Dataproc zonal

  • Batch MapReduce processing via configurable, managed Spark & Hadoop clusters
  • Handles being told to scale (adding or removing nodes) even while running jobs
  • Integrated with Cloud Storage, BigQuery, Bigtable, and some Stackdriver services
  • "Image versioning" switches between versions of Spark, Hadoop, & other tools
  • Pay directly for underlying GCE servers used in the cluster - optionally preemptible
  • Pay a Cloud Dataproc management fee per vCPU-hour in the cluster
  • Best for mobing existing Spark/Hadoop setups to GCP
    • Prefer Cloud Dataflow for new data processing pipelines - "Go with the flow"

Cloud Dataflow zonal

  • Smartly-autoscaled & fully-managed batch or stream MapReduce-like processing
  • Released as open-source Apache Beam
  • Autoscales & dynamically redistributes lagging work, mid-job, to optimize run time
  • Integrated with Cloud Pub/Sub, Datastore, BQ, Bigtable, Cloud ML, Stackdriver, etc.
  • Dataflow Shuffle service for batch offloads Shuffle ops from workers for big gains
  • Effectively pay for underlying worker GCE via consolidated charges
    • Pay per second for vCPUs, RAM GBs PD/PD-SSD (more for streaming)
  • Dataflow Shuffle charged for time per GB used

Cloud Datalab regional

  • Interactive tool for data exploration, analysis, visualization, and machine learning
  • Uses Jupyter Notebook
    • "An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, and much more."
  • Supports iterative development of data analysis algorithms in Python/SQL/~JS
  • Pay for GCE/GAE instance hosting and storing (on PD) your notebooks
  • Pay for any other resources accessed (e.g. BigQuery)

Cloud Data Studio global

  • Big Data Visualization tool for dashboards and reporting
  • Meaningful data stories/presentations enable better business decision making
  • Data sources include BigQuery, Cloud SQL, other MySQL, Google Sheets, Google Analytics, Analytics 360, AdWords, DoubleClick, & YouTube channels
  • Visualizations include time series, bar charts, pie charts, tables, heat maps, geo maps, scorecards, scatter charts, bullet charts, & area charts
  • Templates for quick start; customization options for impactful finish
  • Familiar G Suite sharing and real-time collaboration
  • Pay only for services accessed

Cloud Genomics global

  • Store and process genomes and related experiments
  • Query complete genomic information of large research projects in seconds
  • Process many genomes and experiments in parallel
  • Open industry standards (e.g. from Global Alliance for Genomics and Health)
  • Supports "Requester Pays" sharing

Identity and Access - Core Security

Roles global

  • Roles are collections of Permissions to use or manage GCP resources
  • Permissions allow you to perform certain actions: Service.Resource.Verb
  • Primitive Roles: Owner, Editor, Viewer
    • Viewer is read-only; Editor can change things; Owner can control access & billing
    • Pre-date IAM service, may still be useful (e.g. dev/test envs), but often too board
    • Pre-date IAM service, may still be useful (e.g. dev/test envs), but often too board
  • Predefined Roles: Give granular access to specific GCP resources (IAM)
    • E.g.: roles/bigquery.dataEditor, roles/pubsub.subscriber
  • Custom Roles: Project- or Org-level collections you define of granular permissions

Cloud IAM global

  • Controls access to GCP resources: authorization, not really authentication/identity
  • Member is user, group, domain, service account, or the public (e.g. "allUsers")
    • Individual Google account, Google group (best), G Suite / Cloud Identity domain
    • Service account belongs to application/interface, not individual end user
    • Every identity has a unique -email address, including service accounts
  • Policies bind Members to Roles at a hierarchy level: Org, Folder, Project, Resource
    • Answer: Who can do what to which thing(s)?
  • IAM is free; pay for authorized GCP service usage

Service Accounts global

  • Special type of Google account that represents an application, not an end user
  • Can be "assumed" by applications or individual users (when so authorized)
  • "Important: For almost all cases, whether you are developing locally or in a production application, you should use service accounts, rather than user accounts or API keys."
  • Consider resources and permissions required by application; use least privilege
  • Can generate and download private keys (user-managed keys), for non-GCP, but...
  • Cloud-Platform-managed keys are better, for GCP (i.e. GCF, GAE, GCE, and GKE)
    • No direct downloading: Google manages private keys & rotates them once a day

Cloud Identity global

  • Identity as a Service (IDaaS, not DaaS) to provision and manage users and groups
  • Free Google Accounts for non-G-Suite users, tied to a verified domain
  • Centrally manage all users in Google Admin console; supports compliance
  • 2-Step verification (2SV/MFA) and enforcement, including security keys
  • Sync from Active Directory and LDAP directories via Google Cloud Directory Sync
  • Identities work with other Google services (e.g. Chrome)
  • Identities can be used to SSO with other apps via OIDC, SAML, OAuth2
  • Cloud Identity is free; pay for authorized GCP service usage

Security Key Enforcement global

  • USB or Bluetooth 2-step verification device that prevents phishing
  • Not like just getting a code via email or text message...
  • Device also verifies the target service
  • Eliminates man-in-the-middle (MITM) attacks against GCP credentials

Resource Manager global

  • Centrally manage & secure organization's projects with custom folder hierarchy
  • Organization resource is root node in hierarchy; folders per your business needs
  • Tied 1:1 to a Cloud Identity / G Suite domain, then owns all newly-created projects
    • Without this organization, specific identities (people) must own GCP projects
  • "Recycle bin" allows undeleting projects
  • Define custom IAM roles at org level
  • Apply IAM policies at organization, folder or project levels
  • No charge for this service

Cloud IAP global

  • Guards apps running on GCP via identity verification, not VPN access
  • Based on CLB & IAM, and only passes authed requests through
  • Grant access to any IAM identities, inc. groups & service accounts
  • Relatively straightforward to set up
  • Pay for load balancing / protocol forwarding rules and traffic

Cloud Audit Logging global

  • Answers the questions "Who did what, where, and when?" within GCP projects
  • Maintains non-tamperable audit logs for each project and organization:
    • Admin Activity and System Events (400 day retention)
    • Access Transparency (400 day retention)
      • Shows actions by Google support staff
    • Data Access (30 day retention)
      • For GCP-visible services (e.g. Can't see into MySQL DB on GCE)
  • Data Access logs priced through Stackdriver Logging; rest are free

Security Management — Monitoring and Response

Cloud Armor global

  • Edge-level protection from DDoS & other attacks on global HTTP(S) LB
  • Offload work: Blocked attacks never reach your systems
  • Monitor: Detailed request-level logs available in Stackdriver Logging
  • Manage IPs with CIDR-based allow/block lists (aka whitelist/blacklist)
  • More intelligent rules forthcoming (e.g. XSS, SQLi, geo-based, custom)
  • Preview effect of changes before making them live
  • Pay per policy and role configured, plus for incoming request volume

Security Scanner global

  • Free but limited GAE app vulnerability scanner with "very low false positive rates"
  • "After you set up a scan, Cloud Security Scanner automatically crawls your application, following all links within the scole of your starting URLs, and attempts to exercise as many user inputs and event handlers as possible."
  • Can identify:
    • Cross-site-scripting (XSS)
    • Flash injection
    • Mixed content (HTTP in HTTPS)
    • Outdated/insecure libraries

Cloud Data Loss Prevention API (DLP) global

  • Finds and optionally redacts sensitive info in unstructured data streams
  • Helps you minimize what you collect, expose, or copy to other systems
  • 50+ sensitive data detectors, including: credit card numbers, names, social security numbers, passport numbers, driver's license numbers (US and some other jurisdictions), phone numbers, and other personally identifiable information (PII)
  • Data can be sent directly, or API can be pointed at GCS, BQ, or Cloud DataStore
  • Can scan both text and images
  • Pay for amount of data processed (per GB) - and gets cheaper when large volume
    • Pricing for storage now very simple (June 2019), but for streaming is still a mess

Event Threat Detection (ETD) global

  • Automatically scans your Stackdriver logs for suspicious activity
  • Uses industry-leading threat intelligence, including Google Safe Browsing
  • Quickly detects many possible threats, including:
    • Malware, cryptomining, outgoing DDoS attacks, port scanning, brute-force SSH
    • Also: Unauthorized access to GCP resources via abusive IAM access
  • Can export parsed logs to BigQuery for forensic analysis
  • Integrates with SIEMs like Google's Cloud SCC or via Cloud Pub/Sub
  • No charge for ETD, but charged for its usage of other GCP services (like SD Logging)

Cloud SCC global

  • "Comprehensive security management and data risk platform for GCP"
  • Security Information and Event Management (SIEM) software
  • "Helps you prevent, detect, & respond to threats from a single pane of glass"
  • Use "Security Marks" (aka "marks") to group, track, and manage resources
  • Integrate ETD, Cloud Scanner, DLP, & many external security finding sources
  • Can alert to humans & systems; Can export data to external SIEM
  • Free! But charged for services used (e.g. DLP API, if configured)
  • Could also be charged for excessive uploads of external findings

Encryption Key Management

Cloud Key Management Service (KMS) regional multi-regional global

  • Low-latency service to mange and use cryptographic keys
  • Supports symmetric (e.g. AES) and asymmetric (e.g. RSA, EC) algorithms
  • Move secret out of code (and the like) and into the environment, in a secure way
  • Integrated with IAM & Cloud Audit Logging to authorize & track key usage
  • Rotate keys used for new encryption either automatically or on demand
    • Still keeps old active key versions, to allow decrypting
  • Key deletion has 24 hour delay, "to prevent accidental or malicious data loss"
  • Pay for active key versions stored over time
  • Pay for key use operations (i.e. encrypt/decrypt; admin operations are free)

Cloud Hardware Security Module (HSM) regional multi-regional global

  • Cloud KMS keys managed by FIPS 140-2 Level 3 certified HSMs
  • Device hosts encryption keys and performs cryptographic operations
  • Enables you to meet compliance that mandates hardware environment
  • Fully integrated with Cloud KMS
    • Same API, features, IAM integration
  • Priced like Cloud KMS: Active key versions stored & key operations
    • But some key types more expensive: RSA, EC, Long AES

Operations and Management

Google Stackdriver global

  • Family of services for monitoring, logging, & diagnosing apps on GCP/AWS/hybrid
  • Service integrations adds lots of value - among Stackdriver and with GCP
  • One Stackdriver account can track multiple:
    • GCP projects
    • AWS accounts
    • Other resources
  • Simple usage-based pricing
    • No longer previous system of tiers, allotments, and overages

Stackdriver Monitoring global

  • Gives visibility into perf, uptime, & overall health of cloud apps (based on collectd)
  • Includes built-in/custom metrics, dashboards, global uptime monitoring, & alerts
  • Follow the trail: Links from alerts to dashboards/charts to logs to traces
  • Cross-cloud: GCP, of course, but monitoring agent also supports AWS
  • Alerting policy config includes multi-condition rules & resource organization
  • Alert via email, GCP Mobile App, SMS, Slack, PagerDuty, AWS SNS, webhook, etc.
  • Automatic GCP/Annthos metrics always free
  • Pay for API calls & per MB for custom or AWS metrics

Stackdriver Logging global

  • Store, search, analyze, monitor, and alert on log data & events (based on Fluentd)
  • Collection built into some GCP, AWS support with agent, or custom send via API
  • Debug issues via integration with Stackdriver Monitoring, Trace & Error Reporting
  • Create real-time metrics from log data, then alert or chart them on dashboards
  • Send real-time log data to BigQuery for advanced analytics and SQL-like querying
  • Powerful interface to browse, search, and slice log data
  • Export log data to GCS to cost-effectively store log archives
  • Pay per GB ingested & stored for one month, but first 50GB/project free
  • Cloud Audit Logging/Access Transparency logs also free

Stackdriver Error Reporting global

  • Counts, analyzes, aggregates, & tracks crashes in helpful centralized interface
  • Smartly aggregates errors into meaningful groups tailored to language/framework
  • Instantly alerts when a new app error canot be grouped with existing ones
  • Link directly drom notifications to error details
    • Thme chart, occurences, affected user count, first/last seen dates, cleaned stack
  • Exception stack trace parser knows Java, Python, JavaScript, Ruby, C#, PHP, & Go
  • Jump from stack frames to srouce to start debugging
  • No direct charge; pay for source data in Stackdriver Logging

Stackdriver Trace global

  • Tracks and displays call tree & timings across distributed systems, to debug perf
  • Automatically captures traces from Google App Engine
  • Trace API and SDKs for Java, Node.js, Ruby, and Go capture traces from anywhere
  • Zipkin collector allows Zipkin tracers to submit data to Stackdriver Trace
  • View aggregate app latency info or dig into individual traces to debug problems
  • Generate reports on demand and get daily auto reports per traced app
  • Detects app latency shift (degradation) over time by evaluating perf reports
  • Pay for ingesting and retrieving trace spans

Stackdriver Debugger global

  • Grabs program state (callstack, variables, expressions) in live deploys; low impact
  • Logpoints repeat for up to 24h; fuller snapshots run once but can be conditional
  • Source view supports Cloud Source Repository, Github, Bitbucket, local, & upload
  • Java and Python supported and GCE, GKE, and GAE (Standard and Flex)
  • Node.js and Ruby supported on GCE, GKE, and GAE Flex; Go only on GCE and GKE
  • Automatically enabled for Google App Engine apps; agents available for others
  • Share debugging sessions with others (just send URL)
  • Free to use

Stackdriver Profiler global

  • Continuous CPI and memory profiling to improve perf & reduce cost
  • Low overhead (Typical 0.5%; Max 5%) - so use it in prod, too!
  • Supports Go, Java, Node.js, and Python (3.2+)
  • Agent-based
  • Saves profiles for 30 days
  • Can download profiles for longer-term storage
  • Free to use

Cloud Deployment Manager global

  • Create/manage resources via declarative templates: "Infrastructure as Code"
  • Declarative allows automatic parallelization
  • Templates written in YAML, Python, or Jinja2
  • Supports input and output parameters, with JSON schema
  • Create and update of deployments both support preview
  • Free service; just pay for resources involved in deployments

Cloud Billing API global

  • Programatically manage billing for GCP projects and get GCP pricing
  • Billing config
    • List billing accounts; get details and associated projects for each
    • Enable (associate), disable (disassociate), or change project's billing account
  • Pricing
    • List billable SKUs; get public pricing (including tiers) for each
    • Get SKU metadata like regional availability
  • Export of current bill to GCS or BQ is possible - but configured via console, not API

Development and APIs

Cloud Source Repositories global

  • Hosted private Git repositories, with integrations to GCP and other hosted repos
  • Supports standard Git functionality
  • No enhanced workflow support like pull requests
  • Can set up automatic sync from GitHub or Bitbucket
  • Natural integration with Stackdriver debugger for live-debugging deployed apps
  • Pay per project-user active each month (not prorated)
  • Pay per GB-month of data storage (prorated)
  • Pay per GB of data egress

Cloud Build global

  • Continuously takes source code and builds, tests, and deploys it - CI/CD service
  • Trigger from Cloud Source Repository (by branch, tag, or commit) or zip in GCS
    • Can trigger from GitHub and Bitbucket via Cloud Source Repositories RepoSync
  • Runs many builds in parallel (currently 10 at a time)
  • Dockerfile: super-simple build + push - plus scans for package vulnerabilities
  • JSON/YAML file: flexible & parallel steps
  • Push to GCR & export artifacts to GCS - or anywhere your build steps write
  • Maintains build logs and build history
  • Pay per minute of build time - but free tier is 120 minutes per day

Container Registry regional multi-regional

  • Fast, private Docker image storage (based on GCS) with Docker V2 Registry API
  • Creates & manages a multi-regional GCS bucket, then translates GCR calls to GCS
  • IAM integration simplifies builds and deployments within GCP
  • Quick deploys because of GCP networking to GCS
  • Directly compatible with standard Docker CLI; native Docker Login support
  • UX integrated with Cloud Build & Stackdriver Logs
  • UI to manage tags and search for images
  • Pay directly for storage and egress of underlying GCS (no overhead)

Cloud Endpoints global

  • Handles authorization, monitoring, logging, & API keys for APIs backed by GCP
  • Proxy instances are distributed and hook into Cloud Load Balancer
  • Super-fast Extensible Service Proxy (ESP) container based on nginx: <1 ms/call
  • Uses JWTs and integrates with Firebase, Auth0, & Google Auth
  • Integrates with Stackdriver Logging and Stackdriver Trace
  • Extensible Service Proxy (ESP) can transcode HTTP/JSON to gRPC
    • But API needs to be resource-oriented (i.e. RESTful)
  • Pay per call to your API

Apigee global

  • Full-featured & enterprise-scale API management platform for whole API lifecycle
  • Transform calls between different protocols: SOAP, REST, XML, binary, custom
  • Authenticate via OAuth/SAML/LDAP; authorize via Role-Based Access Control
  • Throttle traffic with quotas, manage API versions, etc.
  • Apigee Sense identifies and alerts administrators to suspicious API behaviors
  • Apigee API Monetization supports various revenue models/rate plans
  • Team and Business tiers are flat monthly rate with API call quotas & feature sets
  • "Enterprise" tier and special feature pricing are "Contact Sales"

Test Lab for Android global

  • Cloud infrastructure for running test matrix across variety of real Android devices
  • Production-grade decives flashed with Android version and locale you specify
  • Robo test captures log files, saves annotated screenshots & video to show steps
    • Default completely automatic but still deterministic, so can show regressions
    • Can record custom script
  • Can also run Espresso and UI Automator 2.0 instrumentation tests
  • Firebase Spark and Flame plans have daily allotment of physical and virtual tests
  • Blaze (PAYG) plan charges per device-hour - much less for virtual devices