GCP Structure & Design

Physical infrastructure

  • vCPU
  • Physical server
  • Rack
  • Data center (building)
  • Zone
  • Region
  • Multi-Region
  • Private global network
  • Points of Presence (POPs) - Network edges and CDN locations
  • Global system

Network Ingress & Egress

  • Normal network: Routes via Internet to edge location closest to destination
  • Google: Routes so traffic enters from Internet at edge closest to source
    • Enables very interesting scenarios
    • Single global IP address can load balance worldwide
    • Sidesteps many DNS issues
  • Can now opt for "normal" network routing to reduce price (and functionality)

Pricing model

  • Provisioned — "Make sure you're ready to handle X"
  • Usage — "Whandle whatever I use, and charge me for that"
  • Network traffic
    • Free on the way in (ingress)
    • Carged on the way out (egress), by GBs used
    • Egress to GCP services sometimes free
      • Depends on the destination service
      • Depends on the location of that service

Security

  • Separation of duties and physical security
  • Absolutely everything always encrypted at rest
  • Strong key and identity management
  • Network encryption
    • All control info encrypted
    • All WAN traffic to be enctypted automatically
    • Moving towards encrypting all local traffic within data centers
  • Distrust the network, always
    • BeyondCorp

Link to google cloud security design here

Organization

  • Projects are similar to AWS accounts
  • Projects own resources
  • Resurces can be shared with other projects
  • Projects can be grouped and controlled in a hierarchy

The Google Cloud Developer's Cheat Sheet

GCP Free Trial Restrictions

  • No more than 8 vCPUs (total simultaneous)
  • No GPUs (video card chips)
  • No TPUs (custom chips for TensorFlow)
  • No Quota increases
  • No cryptomining allows
  • No SLAs
  • No premium OS licenses (e.g. Windows)
  • No Cloud Launcher products with extra usage fees

Always Free Services

  • Compute
    • 24h/day of f1-micro runtime, in most US regions, only
    • 28h/day of App Engine runtime, in North America
    • 2M/month of Cloud Functions invocations (with runtime/size limits)
  • Storage
    • Storage averages over the month
    • 5GB of Regional Cloud Storage, including some operations
    • 1 GB of Cloud Datastore storage, including some operations
    • 10 GB of BigQuery storage, with 1 TB/month of query processing
    • 30 GB HDD storage on GCE and AE
    • 5 GB snapshot storage on GCE and AE
    • 5 GB of StackDriver logs with 7 day retention
  • Networking
    • Egress to China and Australia not free!
    • 1 GB/month of App Engine data egress
    • 1 GB/month of Compute Engine data egress
    • 5 GB/month of egress by Cloud Function invocations
    • 5 GB/month of egress from Cloud Storage based in North America
    • 10 GB/month of Cloud PubSub messages
  • Extras
    • 120 build-minutes/day of Google Cloud Container Builder
    • 60 minutes/month of Google Cloud Speeh API reconginiton from audio/video
    • 1000 units/month of Cloud VIsion API calls
    • 5000 units/month Google Cloud Natural Language API
    • Google Cloud Shell with 5 GB of persistent disk storage data
    • 1 GB of Google Cloud Source Repositories private hosting

Accounts

Billing IAM

  • Role: Billing Account User
  • Purpose: Link projects to billing accounts
  • Level: Organization or billing account
  • Use Case: This role has very restricted persmissions, so you can grant it broadly, typically in combination with Project Creator. These Two roles allow a user to create new projects linked to the billing account on which the role is granted.

Setting up

  • Get email address of non-admin Google account you control
    • Not the admin account created when signing up for GCP
    • This will be our "user" account
    • Could be pre-existing Google account
    • Could make Google account for any existing email address
    • Could make new Gmail account
    • Make sure to have 2FA set up for the account

Cloud Shell and Data Flows

Google Cloud Shell

Google Cloud Shell provides you with command-line access to your cloud resources directly from your browser. You can easily manage your projects and resources without having to install the Google Cloud SDK or other tools on your system. With Cloud Shell, the Cloud SDK gcloud command-line tool and other utilities you need are always available, up to date and fully authenticated when you need them.

Summary

  • Web browser access
    • No need for local terminal
      • Chromebook
      • No Putty
    • Automatic SSH key management
  • 5 GB of persistent storage
  • Easy-access to preinstalled tools
    • gcloud, bq, kubectl, docker, npm/node, pip/python, ruby, vim, emacs, bash, etc.
  • Pre-authorized and always up-to-date
  • Web preview of web app running on local port

VM is ephemeral and will be reset, but home directory is persistent

Notable Features

  • Boost mode
  • Upload fie
  • Download file
  • Code editor
  • Web preview

Data Flows

Data flow has three core components:

  • Moving (Network services)
  • Processing (Compute services)
  • Remembering (Storage services)

Basic GCP Services

GCS: Google Cloud Storage

Google Cloud Storage is a RESTful online file storage web service for storing and accessing data on Google Cloud Platform infrastructure. The service combines the performance and scalability of Google's cloud with advanced security and sharing capabilities.

Access via Storage/Storage/Browser

Create a bucket

Permissions can be edited to these objects

URL is predictable

https://storage.googleapis.com/<bucket_name>/<object_name>

Hierarchy can be achieved by Creating Folders or using / characters in the object name (adding it to the end)

Every object that has a tailing / in it's name is considered to be a folder

GCS via gsutil in Command Line

Check what project we're currently in via the terminal

gcloud config list

List storages

gsutil ls

Native GCloud storage addresses start with gs://

List objects in bucket

gsutil ls gs://<bucket_name>/

Use /** to show everything in a folder

Make a bucket

gsutil mb -l <region> gs://<bucket_name>

Ge the label of a bucket

gsutil label get gs://<bucket_name>/

Label is set from file

gsutil label set <path_to_file> gs://<bucket_name>/

Label can be appended

gsutil label ch -l "labelkey:labelvalue" gs://<bucket_name>/

Check versioning

gsutil versioning get gs://<bucket_name>/

Copy files between buckets

gsutil cp gs://<source_bucket>/** gs://<target_bucket>/

Add permissions to all users

gsutil acl ch -u AllUsers:R gs://<bucket_name>/<object_name>

Google Compute Engine: VM Setup

Get the current project

gcloud config get-value project

List all the VM instances running

gcloud compute instances list

List GCloud services

gcloud services list

Filter available services with compute substring

gcloud services list --available | grep compute

Create a VM

gcloud compute instances create <vm_name>

Delete VM

gcloud compute instances delete <vm_name>

Rundown on the gcloud command

Overview

  • Command-line tool to interact with GCP
  • Best friends with gsutil and bq
    • All share same configuration set via gcloud config
    • gsutil could have been gcloud storage
    • bq could have been gcloud bigquery
  • In general: more powerful than console, but less powerful than REST API
  • Alpha and Beta versions available via gcloud alpha and gcloud beta

Basic Syntax

gcloud <global flags> <service/product> <group/area> <command> <flags> <parameters>

Global Flags

  • --help
  • -h
  • --project ProjectID
  • --account Account
  • --filter
    • Not always available, but often better than using grep
  • --format
    • Can choose JSON, YAML, CSV, etc.
    • Can pipe | JSON to jq command for further processing
  • --quiet (or -q)
    • Skipping confirmation prompts
    • Useful for scripts

Config Properties

  • Values entered once and used by any command that needs them
  • Can be overridden on a specific command with corresponding flag
  • Used very often for account, project, region and zone
    • Set core/account or account to replace --account
    • Set core/project or project to replace --project
    • Set compute/region to replace --region
    • Set compute/zone to replace --zone

Set with

gcloud config set <property> <value>

Get with

gcloud config get-value <property>

Clear with

gcloud config unset <property>

Configurations

Can maintain groups of settings and switch between them

Most useful when using multiple projects

Interactive workflow to set common properties in a config with

gcloud init

List all properties in a configuration with

gcloud config list

IS_ACTIVE column shows which one is currently being used

Other columns list account, project, region, zone and the name of the config

Make new config with

gcloud config configurations create <config_name>

Start using config with

gcloud config configurations actuvate <config_name>

Or use for just one command with --configuration=<config_name>

Official docs here

GCE Connecting to VMs

Command for checking the currently logged in user

whoami

Curl trick to get the machine ip

curl api.ipify.org

List machine types

gcloud compute machine-types list

Create micro machine type

gcloud compute instances create --machine-type=f1-micro <name>

Test connection with PING

ping -c 3 <external_ip>

SSH into the VM

gcloud compute ssh <vm_name>