MySQL, on the source server:

1
2
3
4

mysqldump --databases app -h db.example -u dumper -p \
  --hex-blob --single-transaction --set-gtid-purged=OFF \
  --default-character-set=utf8mb4 --routines --triggers --events \
  | sed -E 's/DEFINER=`[^`]+`@`[^`]+`//g' | gzip > app.sql.gz

PostgreSQL, on the source server:

1
2

pg_dump app -h db.example -U dumper --format=plain --no-owner --no-acl \
  | sed -E 's/(DROP|CREATE|COMMENT ON) EXTENSION/-- \1 EXTENSION/g' | gzip > app.sql.gz

And in both cases:

1
2
3
4
5
6
7

SA=$(gcloud sql instances describe my-instance --format='value(serviceAccountEmailAddress)')
gcloud storage buckets add-iam-policy-binding gs://my-dumps \
  --member="serviceAccount:$SA" --role=roles/storage.objectAdmin
gcloud storage cp app.sql.gz gs://my-dumps/

gcloud sql import sql my-instance gs://my-dumps/app.sql.gz --database=app                 # MySQL
gcloud sql import sql my-instance gs://my-dumps/app.sql.gz --database=app --user=postgres  # PostgreSQL

gcloud sql import sql takes plain SQL or .gz: the custom format of pg_dump goes through pg_restore (below).

The errors and where they come from

On Cloud SQL nobody has SUPER (MySQL) or is a superuser (PostgreSQL); the admin is root@% or postgres, a member of cloudsqlsuperuser. Everything a normal dump takes for granted on your own server breaks right there:

MySQL: what to do with DEFINER clauses

First see how many there are on the source, and whose:

1
2
3
4

SELECT DISTINCT DEFINER FROM INFORMATION_SCHEMA.VIEWS;
SELECT DISTINCT DEFINER FROM INFORMATION_SCHEMA.TRIGGERS;
SELECT DISTINCT DEFINER FROM INFORMATION_SCHEMA.ROUTINES;
SELECT DISTINCT DEFINER FROM INFORMATION_SCHEMA.EVENTS;

Three ways out, from most to least faithful to the original:

1. Create the users on the instance before importing, with the same user@host: gcloud sql users create app --host=% --instance=my-instance --password=…. This is what the documentation recommends, and it keeps views and procedures exactly as they were. root@localhost is the exception: it cannot be created on Cloud SQL, so a DEFINER with that value must be changed no matter what.
2. Strip the DEFINER with the sed above: objects get created with the importing user as definer. Enough when the source users will never exist on Cloud SQL.
3. Switch to SQL SECURITY INVOKER (sed -E 's/SQL SECURITY DEFINER/SQL SECURITY INVOKER/g' on top of the previous one): the view or procedure runs with the caller’s privileges and the definer stops mattering. It changes the permission model; check that the application users can read the underlying tables.

The sed is deliberately broad because DEFINER= also shows up inside conditional comments (/*!50013 DEFINER=… SQL SECURITY DEFINER */); deleting just the DEFINER= leaves the rest intact and valid.

MySQL: the flags, one by one

If the user you dump with lacks PROCESS, mysqldump 8.0.21+ fails right away with “Access denied; you need (at least one of) the PROCESS privilege(s)” while querying tablespaces: add --no-tablespaces.

PostgreSQL: plain or custom

--no-owner --no-acl make everything imported belong to --user, with no GRANT to phantom roles. If the application connects as another role, create the role first and use --user=app, or reassign afterwards:

1

REASSIGN OWNED BY postgres TO app;   -- connected as postgres, in the imported database

Create the extensions the database needs (pg_trgm, postgis, uuid-ossp…) before importing, as postgres, in the target database: CREATE EXTENSION IF NOT EXISTS pg_trgm;. Only the ones on Cloud SQL’s supported extensions list work; plpgsql is already there. With that in place, commenting out the dump’s CREATE EXTENSION lines removes nothing.

If you prefer the custom format (-Fc), the extension filter goes into the pg_restore table of contents, exactly as Google documents it, and you restore through the Cloud SQL Auth Proxy:

1
2
3

pg_dump app -h db.example -U dumper --format=custom --no-owner --no-acl > app.dmp
pg_restore --list app.dmp | sed -E 's/(.*EXTENSION )/; \1/g' > app.toc
pg_restore --use-list=app.toc --no-owner --no-acl -j 4 -h 127.0.0.1 -U postgres -d app app.dmp

-j 4 restores in parallel, which the import from the bucket does not do.

Traps

• The bucket role goes to the instance’s service account, not yours: the docs ask for storage.objectAdmin (for import alone storage.objects.get and list, that is objectViewer, are enough). Without it, “The service account does not have the required permissions for the bucket” even though you can read the object yourself.
• One operation at a time per instance: with a backup or export running, the import fails with “operation already in progress”. gcloud sql operations list --instance=my-instance shows it.
• Large dumps: the import is serial and slow; launch it with --async and watch it with gcloud sql operations wait OPERATION_ID --timeout=unlimited, or gcloud dies on you along the way.
• pg_dump version: use the same major version as the target or lower. pg_dump 17 writes SET transaction_timeout = 0;, which a PostgreSQL 16 rejects with “unrecognized configuration parameter” and aborts the import.
• utf8mb4 on the instance: --default-character-set=utf8mb4 in the dump does not fix an instance created with latin1; check character_set_server in the instance flags before importing.
• The DEFINER sed also touches data: if some row literally contains DEFINER=`x`@`y` (rare, but a CMS that stores SQL can), skip it with a pattern anchored to CREATE, or review the dump diff.

See also

• PostgreSQL: admin cheat sheet — roles, pg_dump, REASSIGN OWNED and friends.
• Import and export using SQL dump files (MySQL), DEFINER clause and pg_dump / pg_restore (PostgreSQL) in the Cloud SQL documentation.

1

GRANT CHANGE NOTIFICATION TO app_user;

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

import oracledb

def on_change(message):
    for query in message.queries:
        for table in query.tables:
            for row in table.rows:
                print(table.name, row.operation, row.rowid)

conn = oracledb.connect(user="app_user", password=pw,
                        dsn="db.example:1521/orclpdb1", events=True)

sub = conn.subscribe(callback=on_change,
                     operations=oracledb.OPCODE_INSERT | oracledb.OPCODE_UPDATE | oracledb.OPCODE_DELETE,
                     qos=oracledb.SUBSCR_QOS_QUERY | oracledb.SUBSCR_QOS_ROWIDS)
sub.registerquery("select id, status, updated_at from orders")   # explicit columns, never SELECT *

input("Ctrl+C to exit\n")

Every COMMIT that changes the result of that query fires on_change with the affected rowids. No polling, no triggers, no GoldenGate.

Why

When another system must react to changes in Oracle (invalidate a cache, replicate to another database, kick off a process), the usual approach is asking every N seconds with WHERE updated_at > :last. It works until it does not: rows without a date column, deletes that leave no trace, useless load on the database.

Continuous Query Notification flips the flow: you register a query and Oracle calls you when it changes. In query mode (SUBSCR_QOS_QUERY) it notifies only if the result of that query changes; in object mode (without that flag) it notifies on any change to the tables involved. SUBSCR_QOS_ROWIDS includes the rowids in the notification, so the callback can read exactly those rows.

The case: replicating Oracle into PostgreSQL

oracle_fdw lets you query Oracle tables from PostgreSQL as if they were local, and it is tempting to think a trigger on the foreign table will fire when data changes in Oracle. It does not: triggers on a foreign table only fire for writes made from PostgreSQL through it. The oracle_fdw author says it plainly: there is no automatic way to keep a PostgreSQL table in sync with an Oracle table from the PostgreSQL side; Oracle has to trigger it.

With CQN, the small Python service is the missing piece:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

def on_change(message):
    rowids = [row.rowid for q in message.queries for t in q.tables for row in t.rows]
    with conn.cursor() as cur:
        cur.execute("select id, status, updated_at from orders where rowid in (%s)"
                    % ",".join(f"'{r}'" for r in rowids))
        for id_, status, updated_at in cur:
            pg.execute("insert into orders (id, status, updated_at) values (%s, %s, %s) "
                       "on conflict (id) do update set status = excluded.status, updated_at = excluded.updated_at",
                       (id_, status, updated_at))
    pg.commit()

Deletes arrive with row.operation & oracledb.OPCODE_DELETE and a rowid that no longer exists: delete in PostgreSQL by the key you stored against that rowid, or flag the row.

Requirements

• Permission: GRANT CHANGE NOTIFICATION TO app_user, besides SELECT on the tables.
• events=True on the connection that subscribes; without it subscribe() fails.
• Network back-channel. By default the database opens a connection towards your application to deliver notifications: the client machine needs an IP reachable from the Oracle server and an open port (subscribe(..., ip_address=…, port=…) pins which). Behind NAT, inside a container, or with a firewall in between, notifications simply never arrive and nobody tells you.
• client_initiated=True removes that problem: your app opens the connection. It requires Thick mode (oracledb.init_oracle_client() with Instant Client 19.4 or later) and a 19.4+ database. In Thin mode, python-oracledb’s default, CQN works but only in the classic mode.
• job_queue_processes greater than zero on the database: notifications are dispatched through the job queue.

Traps

• ORA-29977: Unsupported column type for query registration in guaranteed mode: in query mode some column types are not allowed (LOBs, among others), and SELECT * drags them in. List the columns you need, or drop SUBSCR_QOS_QUERY and work in object mode.
• Notifications are not durable: if your service is down during a change, it does not get it. On startup, reconcile first (by updated_at, or in full) and subscribe afterwards. timeout=0 on subscribe keeps the subscription until the connection dies.
• One subscription per live connection: the events=True connection must stay open; give it keepalives and reconnect with a re-subscribe if it drops.
• Heavy volume (thousands of commits per second) is not CQN territory: that is where redo-log CDC comes in (GoldenGate, or Debezium with its Oracle connector over LogMiner).
• SELECT with non-deterministic functions or over complex views may not register in query mode; simplify the query or fall back to object mode.

Reference: Continuous Query Notification in the python-oracledb documentation and the oracle_fdw discussion.

1
2
3

; /etc/pgbouncer/pgbouncer.ini
[pgbouncer]
ignore_startup_parameters = extra_float_digits, options

1

systemctl reload pgbouncer          # or RELOAD; on the admin console: psql -p 6432 -U pgbouncer pgbouncer

extra_float_digits can be ignored with no effect. options (what JDBC and DataGrip use for -c search_path=…) is ignored too, but then the client’s search_path never arrives: pin it on the role with alter role app set search_path = app, public.

The error

1

FATAL: unsupported startup parameter: extra_float_digits

or, depending on the client, pooler error: unsupported startup parameter: extra_float_digits. Straight to port 5432 it connects; through pgbouncer’s 6432, it does not. JDBC sends it (everything using the PostgreSQL Java driver: DBeaver, DataGrip, Spring, Jenkins), as do Npgsql and some ORMs; other clients send options.

Why it happens

When a client opens a connection, it puts the session parameters it wants into the startup packet (extra_float_digits=3, options=-c search_path=app). pgbouncer, in transaction or statement mode, shares one server connection among many clients, so it only lets through the parameters it knows how to store per client and restore on the server when that client’s turn comes: client_encoding, datestyle, timezone, standard_conforming_strings, application_name. Anything else would be a setting leaking from one client to another; rather than that, pgbouncer rejects the connection.

Two ways to resolve it:

extra_float_digits is harmless since PostgreSQL 12: float output is already the shortest exact representation and the parameter changes nothing. options with search_path does matter: ignored, the client sees the database’s default search_path and unqualified queries fail with relation does not exist. The clean fix is the role:

1

alter role app set search_path = app, public;

’no such database'

1

pooler error: no such database: app_db

pgbouncer only routes databases listed under [databases]. Either add the entry, or a wildcard:

1
2
3

[databases]
app_db = host=db.example port=5432 dbname=app_db
* = host=db.example port=5432

With *, a client asking for other_db gets host=db.example dbname=other_db; those automatic entries clean themselves up after autodb_idle_timeout. And check the entry’s dbname=: pointing app_db at another database through a copy-paste is common.

Traps

• RELOAD is enough for ignore_startup_parameters and [databases]; listen_* and auth_type need a restart.
• transaction mode and SET: a manual SET search_path in the session is lost as soon as the server connection moves to another client; use SET LOCAL inside the transaction or the alter role.
• Named prepared statements (JDBC prepareThreshold, PREPARE) do not work in transaction mode on old pgbouncer; since 1.21, max_prepared_statements = 200 supports them.
• SCRAM: with auth_type = scram-sha-256 the userlist.txt needs the SCRAM hash exactly as it comes out of pg_authid, or auth_query so pgbouncer looks it up.
• SHOW CONFIG; and SHOW DATABASES; on the admin console tell you what is really loaded, not what you think you wrote.

See also: PostgreSQL: admin cheat sheet. Reference: pgbouncer.ini.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

-- Exact rows of every table in a schema, in a single query
select table_schema, table_name,
       (xpath('/row/cnt/text()', xml_count))[1]::text::bigint as row_count
from (
  select table_schema, table_name,
         query_to_xml(format('select count(*) as cnt from %I.%I', table_schema, table_name), false, true, '') as xml_count
  from information_schema.tables
  where table_schema = 'public' and table_type = 'BASE TABLE'
) t
order by row_count desc;

-- Instant estimate (does not read the tables)
select relname, n_live_tup from pg_stat_user_tables order by n_live_tup desc;

-- Read-only user (PostgreSQL 14 or later)
create user reporting with password '…';
grant pg_read_all_data to reporting;

Counting rows

An exact count(*) on each table means scanning it whole; on a database of hundreds of GB it takes a while. What you almost always want is the estimate autovacuum maintains, refreshed on every ANALYZE and usually within a couple of points of the real number:

1
2
3

select schemaname, relname, n_live_tup, n_dead_tup, last_autoanalyze
from pg_stat_user_tables
order by n_live_tup desc;

When you need the exact number for every table without writing a DO block in PL/pgSQL, the query_to_xml trick: it runs a dynamic count(*) per table, returns the result as XML, and xpath extracts the number. It is plain SQL, works from any client, and can be filtered by schema or name. Just remember it runs the counts serially: on big tables, patience.

Sizes

1
2
3
4
5
6
7
8
9

select pg_size_pretty(pg_database_size(current_database()));

select n.nspname as schema, c.relname as table,
       pg_size_pretty(pg_total_relation_size(c.oid)) as total,   -- data + indexes + TOAST
       pg_size_pretty(pg_relation_size(c.oid)) as data
from pg_class c join pg_namespace n on n.oid = c.relnamespace
where c.relkind = 'r' and n.nspname not in ('pg_catalog', 'information_schema')
order by pg_total_relation_size(c.oid) desc
limit 20;

Sessions

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

-- Who is connected and what they are doing
select pid, usename, state, client_addr, application_name, left(query, 80) as query
from pg_stat_activity
where datname = current_database();

-- Cancel a query (the session stays)
select pg_cancel_backend(12345);

-- Kick everyone but yourself (before a DROP or a restore)
select pg_terminate_backend(pid)
from pg_stat_activity
where datname = 'app_db' and pid <> pg_backend_pid();

-- Or, since PostgreSQL 13, directly:
drop database app_db with (force);

-- Uptime
select date_trunc('second', current_timestamp - pg_postmaster_start_time()) as uptime;

Read-only user

On 14 or later, a predefined role solves it for every schema, including tables created tomorrow:

1
2

create user reporting with password '…';
grant pg_read_all_data to reporting;

On earlier versions, or if you want to limit it to one schema, the classic path:

1
2
3
4
5
6
7
8
9

grant connect on database app_db to reporting;
grant usage on schema public to reporting;
grant select on all tables in schema public to reporting;
-- future tables: only covers those created by the role that runs this (or FOR ROLE app_owner)
alter default privileges for role app_owner in schema public grant select on tables to reporting;

-- Undo
revoke all on all tables in schema public from reporting;
alter default privileges for role app_owner in schema public revoke select on tables from reporting;

alter default privileges does not touch what already exists and only applies to objects created by the given role: if another account creates tables, reporting will not see them. Hence the value of pg_read_all_data.

Passwords, timezone, and psql

1
2
3
4
5

\password app_user                                  -- from psql, without leaving the password in the history
alter user app_user with password '…';

alter database app_db set timezone to 'America/Mexico_City';   -- new connections to that database
alter role reporting set timezone to 'UTC';                    -- new sessions of that role

1
2

psql -v ON_ERROR_STOP=1 -f data.sql app_db      # a script that stops at the first error
psql -P pager=off app_db                        # no pager (or \pset pager 0 inside)

And inside psql, \x auto (vertical columns when they do not fit) and \timing.

Minimal maintenance

1
2

vacuum (verbose, analyze) orders;
reindex table concurrently orders;               -- 12+, without blocking writes

VACUUM FULL rewrites the table and locks it meanwhile; to reclaim space in production, pg_repack.

Traps

• n_live_tup is an estimate: freshly analyzed it is faithful; on a table with heavy churn it can drift until the next autoanalyze.
• pg_read_all_data does not bypass RLS and grants no DDL; it is reading tables, views, and sequences, nothing else.
• pg_terminate_backend cuts transactions: the application sees an error and reconnects. To only abort a query, pg_cancel_backend.
• alter role … set timezone applies to new sessions; the one already open stays the same.
• quote_ident/format('%I') whenever you build SQL with table names: a name with capitals or dashes breaks the query without them.

See also

• pgbouncer: unsupported startup parameter
• Cloud SQL: importing dumps without DEFINER, GTID or extension errors