Create an internal load balancer across VPC networks

Autopilot Standard

This page explains how to create an internal passthrough Network Load Balancer on Google Kubernetes Engine (GKE) across VPC networks. Before reading this page, you should be familiar with the following concepts:

Before you begin

Before you start, make sure you have performed the following tasks:

Enable the Google Kubernetes Engine API.

Enable Google Kubernetes Engine API

If you want to use the Google Cloud CLI for this task, install and then initialize the gcloud CLI. If you previously installed the gcloud CLI, get the latest version by running gcloud components update.
Note: For existing gcloud CLI installations, make sure to set the compute/region and compute/zone properties. By setting default locations, you can avoid errors in gcloud CLI like the following: One of [--zone, --region] must be supplied: Please specify location.

Create an internal load balancer with Private Service Connect

As a service producer, you can use service attachments to make your services available to service consumers in other VPC networks using Private Service Connect. You can create, manage, and delete service attachments using a ServiceAttachment custom resource.

Requirements and limitations

Limitations for Private Service Connect apply.
You can create a service attachment in GKE versions 1.21.4-gke.300 and later.
You cannot use the same subnet in multiple service attachment configurations.
You must create a GKE service that uses an internal passthrough Network Load Balancer.
You cannot specify a subnet in a different project (Shared VPC) for GKE versions earlier than 1.22.4-gke.100. For Shared VPC, ensure all requirements for Shared VPC are met.

Create a `ServiceAttachment`

Create a subnet.

You must create a new subnet for each ServiceAttachment.
```
gcloud beta compute networks subnets create SUBNET_NAME \
    --project PROJECT_ID \
    --network NETWORK_NAME \
    --region REGION \
    --range SUBNET_RANGE \
    --purpose PRIVATE_SERVICE_CONNECT
```
Replace the following:
- SUBNET_NAME: the name of the new subnet. In GKE versions 1.22.4-gke.100 and later, you can specify a subnet in a different project by using the fully qualified resource URL for this field. You can get the fully qualified resource URL using the command gcloud compute networks subnets describe.
- PROJECT_ID: the ID of your Google Cloud project.
- NETWORK_NAME: the name of the VPC network for the subnet.
- REGION: the region for the new subnet. You must use the same region as the service that you create.
- SUBNET_RANGE: the IP address range to use for the subnet.

Deploy a workload.

The following manifest describes a Deployment that runs a sample web application container image. Save the manifest as my-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: psc-ilb
spec:
  replicas: 3
  selector:
    matchLabels:
      app: psc-ilb
  template:
    metadata:
      labels:
        app: psc-ilb
    spec:
      containers:
      - name: whereami
        image: us-docker.pkg.dev/google-samples/containers/gke/whereami:v1.2.19
        ports:
          - name: http
            containerPort: 8080
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 5
          timeoutSeconds: 1

Apply the manifest to your cluster:
```
kubectl apply -f my-deployment.yaml
```

Create a service. The following manifest describes a service that creates an internal passthrough Network Load Balancer on TCP port 8080. Save the manifest as my-service.yaml:

 apiVersion: v1
 kind: Service
 metadata:
   name: SERVICE_NAME
   annotations:
     networking.gke.io/load-balancer-type: "Internal"
 spec:
   type: LoadBalancer
   selector:
     app: psc-ilb
   ports:
   - port: 80
     targetPort: 8080
     protocol: TCP

Replace the following:

SERVICE_NAME: the name of the new service.

Apply the manifest to your cluster:
```
kubectl apply -f my-service.yaml
```
Create ServiceAttachment.

The following manifest describes a ServiceAttachment that exposes the service that you created to service consumers. Save the manifest as my-psc.yaml:
```
apiVersion: networking.gke.io/v1
kind: ServiceAttachment
metadata:
 name: SERVICE_ATTACHMENT_NAME
 namespace: default
spec:
 connectionPreference: ACCEPT_AUTOMATIC
 natSubnets:
 - SUBNET_NAME
 proxyProtocol: false
 resourceRef:
   kind: Service
   name: SERVICE_NAME
```
Note: For GKE versions earlier than 1.23.3-gke.900, use networking.gke.io/v1beta1 instead of networking.gke.io/v1.

Replace the following:
- SERVICE_ATTACHMENT_NAME: the name of the new service attachment.
- SUBNET_NAME: the name of the new subnet. In GKE versions 1.22.4-gke.100 and later, you can specify a subnet in a different project by using the fully qualified resource URL for this field. You can get the fully qualified resource URL using the command gcloud compute networks subnets describe. For a Shared VPC configuration, use the following format: projects/HOST_PROJECT_ID/regions/COMPUTE_REGION/subnetworks/SUBNET_NAME.
Note: The ServiceAttachment resource will have error events until the GKE controller creates the internal passthrough Network Load Balancer that is defined by the service. After the internal passthrough Network Load Balancer is provisioned, the errors are no longer published.

For more information about the manifest fields, see the service attachment fields.
Apply the manifest to your cluster:
```
kubectl apply -f my-psc.yaml
```

Verify that the Private Service Connect controller created the service attachment:

gcloud beta compute service-attachments list

The output shows a service attachment with an automatically generated name:

NAME        REGION       PRODUCER_FORWARDING_RULE          CONNECTION_PREFERENCE
k8s1-sa-... REGION_NAME  a3fea439c870148bdba5e59c9ea9451a  ACCEPT_AUTOMATIC

View a `ServiceAttachment`

You can view the details of a ServiceAttachment using the following command:

kubectl describe serviceattachment SERVICE_ATTACHMENT_NAME

The output is similar to the following:

 kubectl describe serviceattachment foo-sa
Name:        <sa-name>
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  networking.gke.io/v1beta1
Kind:         ServiceAttachment
Metadata:
  ...
Status:
  Forwarding Rule URL:      https://www.googleapis.com/compute/beta/projects/<project>/regions/<region>/forwardingRules/<fr-name>
  Last Modified Timestamp:  2021-07-08T01:32:39Z
  Service Attachment URL:   https://www.googleapis.com/compute/beta/projects/<projects>/regions/<region>/serviceAttachments/<gce-service-attachment-name>
Events:                     <none>

Consume a `ServiceAttachment`

To consume your service from another project, perform the following steps:

Get the URL of the ServiceAttachment:

kubectl get serviceattachment SERVICE_ATTACHMENT_NAME -o=jsonpath="{.status.serviceAttachmentURL}"

The output is similar to the following:

  serviceAttachmentURL: https://www.googleapis.com/compute/alpha/projects/<project>/region/<region>/serviceAttachments/k8s1-...my-sa

Create a Private Service Connect endpoint using the URL of the ServiceAttachment.

Verify that you can connect to the Service that you deployed in the producer project by using a curl command from a VM in the consumer project:

curl PSC_IP_ADDRESS

Replace PSC_IP_ADDRESS with the IP address of the forwarding rule in the consumer project.

The output is similar to the following:

{
  "cluster_name":"cluster",
  "host_header":"10.128.15.200",
  "node_name":"gke-psc-default-pool-be9b6e0e-dvxg.c.gke_project.internal",
  "pod_name":"foo-7bf648dcfd-l5jf8",
  "pod_name_emoji":"👚",
  "project_id":"gke_project",
  "timestamp":"2021-06-29T21:32:03",
  "zone":"ZONE_NAME"
}

Update a `ServiceAttachment`

You can update a ServiceAttachment using the following steps:

Edit the ServiceAttachment manifest in my-psc.yaml:

apiVersion: networking.gke.io/v1
kind: ServiceAttachment
metadata:
  name: my-sa
  namespace: default
spec:
  connectionPreference: ACCEPT_AUTOMATIC
  natSubnets:
  - my-nat-subnet
  proxyProtocol: false
  resourceRef:
    kind: Service
    name: ilb-service

Apply the manifest to your cluster:
```
kubectl apply -f my-psc.yaml
```

Delete a `ServiceAttachment`

You cannot delete an internal passthrough Network Load Balancer that is connected to a service attachment. You must delete the service attachment and GKE Service separately.

Delete the service attachment:
```
kubectl delete serviceattachment SERVICE_ATTACHMENT_NAME --wait=false
```
This command marks the service attachment for deletion, but the resource continues to exist. You can also wait for the deletion to finish by omitting the --wait flag.
Delete the Service:
```
kubectl delete svc SERVICE_NAME
```

Delete the subnet:

gcloud compute networks subnets delete SUBNET_NAME

`ServiceAttachment` fields

The ServiceAttachment has the following fields:

connectionPreference: the connection preference that determines how customers connect to the service. You can either use automatic project approval using ACCEPT_AUTOMATIC or explicit project approval using ACCEPT_MANUAL. For more information, see Publishing services using Private Service Connect.
natSubnets: a list of subnetwork resource names to use for the service attachment.
proxyProtocol: when set to true, the consumer source IP and Private Service Connect connection ID are available in the requests. This field is optional and defaults to false if not provided.
consumerAllowList: the list of consumer projects that are allowed to connect to the ServiceAttachment. This field can only be used when connectionPreference is ACCEPT_MANUAL. For more information about this field, see Publishing services using Private Service Connect.
- project: the project ID or number for the consumer project.
- connectionLimit: the connection limit for the consumer project. This field is optional.
- forceSendFields: the field names to send to include in API requests. This field is optional.
- nullFields: the field names to include in API requests with a null value. This field is optional.
consumerRejectList: the list of consumer project IDs or numbers that are not allowed to connect to the ServiceAttachment. This field can only be used when connectionPreference is ACCEPT_MANUAL. For more information about this field, see Publishing services using Private Service Connect.
resourceRef: a reference to the Kubernetes resource.
- kind: the type of Kubernetes resource. You must use Service.
- name: the name of the Kubernetes resource that must be in the same namespace as the internal passthrough Network Load Balancer.
Note: After a service attachment is connected to an internal passthrough Network Load Balancer, you cannot delete or update the internal passthrough Network Load Balancer. If you attempt to delete or update the internal passthrough Network Load Balancer, an error message appears. For more information, see Troubleshooting.

Troubleshooting

You can view error messages using the following command:

kubectl get events -n NAMESPACE

Replace NAMESPACE with the namespace of the internal passthrough Network Load Balancer.

An error message similar to the following occurs if you try to delete an internal passthrough Network Load Balancer that is being used by a service attachment. You must delete the ServiceAttachment before you can delete the internal passthrough Network Load Balancer.

Error syncing load balancer: failed to ensure load balancer: googleapi:
Error 400: The forwarding_rule resource '<fwd-rule-URL>' is already being used
by '<svc-attachment-URL>', resourceInUseByAnotherResource.

Create an internal load balancer across VPC networks

Before you begin

Create an internal load balancer with Private Service Connect

Requirements and limitations

Create a ServiceAttachment

View a ServiceAttachment

Consume a ServiceAttachment

Update a ServiceAttachment

Delete a ServiceAttachment

ServiceAttachment fields

Troubleshooting

What's next

Create a `ServiceAttachment`

View a `ServiceAttachment`

Consume a `ServiceAttachment`

Update a `ServiceAttachment`

Delete a `ServiceAttachment`

`ServiceAttachment` fields