reindex hangs or times out on a large catalog, what do you do?

A indexer:reindex that hangs is usually memory pressure, a slow database, or the EAV/flat tables fighting. We profile the slow indexer, raise the relevant batch sizes, run heavy indexers individually instead of all at once, and switch noisy indexers to "Update by Schedule" so they reindex incrementally via MView instead of full rebuilds. On very large catalogs we schedule the full reindex for off-peak.