ON SETS INVARIANT UNDER THE ACTION OF THE DIAGONAL GROUP

ELON LINDENSTRAUSS AND BARAK WEISS

1. Introduction

Starting with Hillel Furstenberg’s seminal paper [F], it has been noted that many natural actions of ℝ^{k}, ℤ^{k} or the semigroup ℤ_{+}^{k} for k ≥ 2 (say on a space X) display remarkable rigidity properties. In particular, in stark contrast to hyperbolic actions of ℤ or ℝ, hyperboliclike actions of these k-dimensional groups display a scarcity both of closed invariant subsets of X and of invariant measures on X.

In this paper we consider the action of A, the positive diagonal matrices in G = SL(n, ℝ) on G/Γ, where Γ is a lattice in G. This action is defined by a(gΓ) = agΓ. For n = 2, i.e., when dim A = 1, it is known that this action has many irregular invariant subsets — in fact for any α ∈ [1, 3] there is a point x ∈ G/Γ such that \overline{Ax} has Hausdorff dimension α (this is an unpublished result of Furstenberg and Benjamin Weiss).

For n ≥ 3 it was conjectured by Gregory A. Margulis (see [M1], p. 203) that typically an orbit-closure for A is an orbit of a group containing A, i.e.,

\overline{Ax} = Hx

for some Lie subgroup H < G. Some care must be taken in this conjecture to rule out some irregular A-orbits arising from rank-1 actions which appear as factors. For a recent formulation of Margulis’ Conjecture see [M3, Conjecture 1.1]. Related positive results restricting the A-invariant measures on G/Γ were established by Anatole Katok and Ralf Spatzier in [KS] (see [KS] also for more on conjectures regarding invariant measures).

In this paper we consider the rather special case of A-orbits whose closures contain compact orbits. For general lattices, we prove the following theorem:

Theorem 1.1. Let G = SL(n, ℝ), n ≥ 3, let A, Γ be as above and let y ∈ G/Γ. Assume that F = \overline{Ay} contains a point x satisfying:

1. Ax is compact; 2. For every 1 ≤ i < j ≤ n, N_{ij} = {diag(a_{1}, …, a_{n}) ∈ A : a_{i} = a_{j}}

acts ergodically on Ax (since N_{ij} is of co-dimension 1 in A, this simply means that N_{ij}x is not compact).

Then there exists a reductive subgroup H, containing A, such that F = Hy and F carries an H-invariant probability measure.

The proof we present for Theorem 1.1 is valid not only for SL(n, ℝ) but also for products of SL(n_{i}, ℝ). To simplify the notation we restrict ourselves to SL(n, ℝ). Interestingly, it does not work when G is any other noncompact simple Lie group (see Remark following the proof of Lemma 4.2); this might be seen as an indication of the subtlety of the questions considered here.

A result similar to Theorem 1.1 was obtained in the S-arithmetic context by Shahar Mozes in [Mo] for the case G = PGL(ℚ_{p})×PGL(ℚ_{ℓ}) and Γ an irreducible lattice.

In unpublished work, Mary Rees has given an example of a cocompact lattice in SL(3, ℝ) for which the assumption 2 is not satisfied, and irregular orbits for the action of A do arise. Using her example as a model we prove:

Theorem 1.2. Let G, A, Γ be as above. If x ∈ G/Γ is such that Ax is compact but there are some i < j such that N_{ij} does not act ergodically on Ax then there is y ∈ G/Γ with Ax ⊂ \overline{Ay} but \overline{Ay} is not of the form Hy for any Lie subgroup of G.

For the case Γ = SL(n, ℤ), i.e., for the action on the space of lattices in ℝ^{n}, we are able to weaken our assumptions and strengthen our conclusions.

Theorem 1.3. Let G and A be as in Theorem 1.1. Let Γ = SL(n, ℤ), y ∈ G/Γ, F = \overline{Ay}. Assume that F contains a compact A-orbit. Then there are integers k and d with n = kd and a permutation matrix P such that F = Hy, where

H = \{P ( \begin{matrix}B_{1} & 0 & ⋯ & 0 \\ 0 & B_{2} & ⋯ & 0 \\ . & . & . \\ 0 & 0 & ⋯ & B_{d} \end{matrix} ) P^{−1} : B_{i} ∈ GL(k, ℝ)\} ∩ G,

(here the 0’s stand for 0 matrices in M_{k}(ℝ)). Moreover, if F ≠ Ay then F is not compact.

Corollary 1.4. Assume in addition to the hypotheses of Theorem 1.3 that n is prime. Then Ay is either compact or dense.

When n is not a prime, there are orbits whose closure contain a compact orbit that are neither dense nor closed. We give an explicit description of all such possible orbit-closures. These orbit-closures correspond to pairs of number fields K and K′ where K′ < K and [K: ℚ] = n.

From Corollary 1.4 we draw the following strengthening of an isolation theorem due to Cassels and Swinnerton-Dyer (see [CaS-D], Theorem 2 and also [M2], §2.1). Recall that {v_{1}, …, v_{m}} ⊂ ℤ^{n}, m < n is said to be primitive if it can be completed to a set of n generators of ℤ^{n}.

Corollary 1.5. Let n ≥ 3 be prime, and let f = L_{1}·. . .·L_{n} be a product of n independent linear forms on ℝ^{n}. Assume that the coefficients of f are integers, and that f does not represent 0 nontrivially over ℚ, that is

f(q) ≠ 0 for all nonzero q ∈ ℚ^{n}.

Then for any open V ⊂ ℝ^{n−1} there is a neighborhood U of f (in the space of products of n linear forms) such that for every h ∈ U which is not a multiple of f there exists a primitive set v_{1}, …, v_{n−1} ∈ ℤ^{n} such that

(h(v_{1}), …, h(v_{n−1})) ∈ V.

The argument deducing Corollary 1.5 from Corollary 1.4 follows along the lines of an argument given by Armand Borel and Gopal Prasad in [BoPr].

There is a number–theoretic motivation for the isolation theorem proved by Cassels and Swinnerton-Dyer, namely the following elementary conjecture due to Littlewood:

Conjecture 1 (Littlewood). For every α, β ∈ ℝ, we have

lim inf_{n→∞} n{nα}{nβ} = 0

(where {x} is the distance from x to the nearest integer).

Using their isolation theorem, Cassels and Swinnerton-Dyer (albeit in a different ‘language’, see [M2] §2.1), showed that Littlewood’s Conjecture follows from the statement that any compact A-invariant subset of SL(3, ℝ)/ SL(3, ℤ) contains a compact orbit.

Finally, we wish to mention some of the progress made on problems related to rigidity of hyperbolic actions of higher-rank abelian groups. In [KS], Katok and Spatzier addressed, inter alia, the problem of describing the ergodic invariant measures for the action of A on G/Γ. Their results place severe restrictions on the invariant measures which have positive entropy with respect to any one parameter subgroup of A.

An analogue to the action of A on G/Γ is the action of a (large enough) semigroup of commuting endomorphisms on tori. For such actions the topological questions regarding the closed invariant subsets have been completely solved by Furstenberg [F] for the one dimensional torus 𝕋^{1} and by Daniel Berend (see [B]) for 𝕋^{d}, d ≥ 2.

A key part of Furstenberg’s and Berend’s argument was proving that any closed invariant subset containing a non–isolated periodic point is the complete torus. They use this result twice. First, in conjunction with a general disjointness argument due to Furstenberg (for which, unfortunately, there is no clear analogue in the case of G/Γ), to show that any minimal closed invariant subset of the torus is a periodic orbit. Then, in view of the fact that any closed invariant subset contains a minimal one, to show that any closed invariant subset of the torus is either a periodic orbit or the whole torus. Our result can be seen as the analogue of this key ingredient in Furstenberg’s and Berend’s proof. In many cases it reduces the problem of describing a general orbit–closure for the action of the diagonal group to that of classifying the minimal compact invariant subsets for the action.

Overview: After some notations and preliminaries, we prove Theorem 1.1 in §4. A key component of the proof is Lemma 4.2, which shows in particular that if \overline{Ay} contains a compact orbit Ax then there is a unipotent subgroup U such that Ux ⊂ \overline{Ay}. This argument was used implicitly in [CaS-D]. We then use Marina Ratner’s orbit closure theorem [R] to show that there is some closed orbit Hx ⊂ \overline{Ay} where H is a reductive Lie subgroup of G (Lemma 4.1). If Hx ≠ \overline{Ay} we then find a higher dimensional unipotent subgroup U′ such that U′x ⊂ \overline{Ay} (this is what is proved in steps 4.7–4.9).

We discuss Rees’ example and prove Theorem 1.2 in §5. In §6 we consider the case of Γ = SL(n, ℤ), and prove Theorem 1.3. We also provide a general construction of orbit closures that contain a compact orbit but are neither a compact orbit nor the whole space. We prove Corollary 1.5 in §7. The appendix contains the proof of an auxilliary result due to Nimish Shah, which has been included for the reader’s convenience.

Acknowledgments: We thank Gregory Margulis and Shahar Mozes for useful and stimulating discussions. In particular, we would like to thank G. Margulis for very helpful suggestions on steps 4.7–4.9 in the proof of Theorem 1.1.

2. Notation and Definitions

Throughout this paper G = SL(n, ℝ) with n ≥ 3. Let k be a subfield of ℝ. By a k-subgroup of G we mean a subgroup of matrices which satisfy a set of polynomial equations with coefficients in k, where the variables of the polynomials are the n^{2} matrix entries. If H is a ksubgroup and R is a ring in k then H(k) (respectively, H(R)) denotes the matrices in H with entries in k (respectively, R). A field automorphism σ of k acts on a matrix M = (m_{ij}) with entries in k by

σ(M) = M^{σ} = (σ(m_{ij})),

and on a polynomial P (x) = ∑ a_{i}x^{i} with coefficients in k by

P^{σ}(x) = ∑ σ(a_{i})x^{i}.

If H is a k-subgroup of G then H^{σ} is the group obtained by acting on the polynomials defining H. The above actions are compatible in the sense that

M ∈ H(k) ⟺ M^{σ} ∈ H^{σ}(k).

For a subgroup H of G, Lie(H) denotes the Lie algebra of H, N_{G}(H) denotes the normalizer of H in G, and

N_{G}^{1}(H) = {g ∈ N_{G}(H) : | det(Ad(g)|_{Lie(H)})| = 1}.

For any subset S ⊂ G, C_{G}(S) denotes the centralizer of S in G. For any x ∈ G/Γ,

H_{x} = {h ∈ H : hx = x}.

Let U^{+} (respectively U^{−}) denote the group of upper-triangular (resp., lower triangular) matrices with all diagonal elements equal to 1. For 1 ≤ i ≠ j ≤ n let λ_{ij} be the linear functional on Lie(A) = {diag(a_{1}, …, a_{n}) : ∑ a_{i} = 0} given by

λ_{ij} : diag(a_{1}, …, a_{n}) ↦ a_{i} − a_{j}

and let U_{ij} be the one-parameter subgroup such that [a, u] = λ_{ij}(a)u for every a ∈ Lie(A), u ∈ Lie(U_{ij}). Thus Φ = {λ_{ij}} is the set of roots and Lie(U_{ij}) = 𝔊_{ij} are the root subspaces of Lie(G). We will sometimes also let Φ denote the set {(i, j) : 1 ≤ i ≠ j ≤ n}.

Throughout this paper u^{+}, u^{−}, u^{+}, u^{−}, u_{ij}, u_{ij}, a, a denote elements of U^{+}, U^{−}, Lie(U^{+}), Lie(U^{−}), U_{ij}, Lie(U_{ij}), A, and Lie(A) respectively. Fix nonzero elements w_{ij} ∈ Lie(U_{ij}). Then each w ∈ Lie(G) can be written uniquely as a+∑ c_{ij}w_{ij}, and we denote the projection w ↦ c_{ij} by P_{ij}. We choose some norm ∥ · ∥_{A} on Lie(A) and define a norm on Lie(G) by

∥w∥ = max{∥a∥_{A}, |c_{ij}|}.

For every z ∈ G/Γ, we can find neighborhoods V^{0}, V^{+}, V^{−} of the identity in Lie(A), Lie(U^{+}), Lie(U^{−}) respectively so that the maps

Ψ : V^{+} × V^{−} → G/Γ, Ψ(u^{+}, u^{−}) = exp(u^{+}) exp(u^{−})z

and

Ψ̂ : V^{0} × V^{+} × V^{−} → G/Γ, Ψ̂(a, u^{+}, u^{−}) = exp(a) exp(u^{+}) exp(u^{−})z

are diffeomorphisms onto their images, and the image of Ψ̂ is a neighborhood of z in G/Γ. We call Ψ and Ψ̂ box maps for z.

Recall that two lattices Γ_{1} and Γ_{2} are commensurable if their intersection is of finite index in both, and that the commensurator of Γ is

Comm_{G}(Γ) = {g ∈ G : Γ and gΓg^{−1} are commensurable}.

3. Preliminaries

In this section we collect some of the results we will need.

The following is probably well–known:

Proposition 3.1. Let H be a closed connected subgroup of G normalized by A. Then

Lie(H) = Lie(A ∩ H) ⊕ ⨁_{λ_{ij}∈Δ} 𝔊_{ij}

for some Δ ⊂ Φ. If H is reductive then H is in ‘block form’, i.e., there is a partition {1, …, n} = B_{1} ⊔ … ⊔ B_{k} such that

Lie(H) = Lie(A ∩ H) ⊕ ⨁_{l=1}^{k}( ⨁_{i,j∈B_{l},i≠j} 𝔊_{ij}), (1)

and H is equal to the connected component of the identity in C_{G}(A ∩ C_{G}(H)).

Proof: The subalgebras 𝔊_{ij}, which are the eigenspaces for the action of Ad(A) on Lie(G), are one-dimensional. Thus an Ad(A)-invariant subspace of Lie(G) is a sum of the 𝔊_{ij}. This proves the first assertion.

For the second assertion, define a relation R on {1, …, n} by

iRj ⟺ i = j or Lie(U_{ij}) ⊂ Lie(H).

Transitivity of R follows from the fact that H is closed under the Lie bracket, and symmetry follows from the fact that H is reductive and normalized by A. Thus R is an equivalence relation, and the desired partition of {1, …, n} is given by the equivalence classes for R.

The third statement follows easily from the second.

We will need the following ergodicity condition:

Theorem 3.2. Moore’s Ergodicity Theorem: Let H be a connected semisimple Lie group with no compact factors and finite center, and let Δ be a lattice in H. Let A be a subgroup of H such that A ∩ N is noncompact for each nontrivial normal subgroup N of H. Then A acts ergodically on H/Δ (with respect to the H-invariant measure).

Proof: In case Δ is irreducible, this is a well–known theorem of C. C. Moore (see [Z], Theorem 2.2.6). The general case follows by reduction to this case: By [Ra], Theorem 5.22, we can write H = H_{1} · . . . · H_{k} with H_{i} ∩ H_{j} discrete and central for i ≠ j, and find Δ_{i} ⊂ H_{i} ∩ Δ, such that Δ_{i} is an irreducible lattice in H_{i}, and [Δ : Δ_{1} ·. . .·Δ_{k}] < ∞. Since A ∩ H_{i} is noncompact, it acts ergodically on H_{i}/Δ_{i} and therefore A acts ergodically on H_{1}/Δ_{1} × ⋯ × H_{k}/Δ_{k}. Therefore the factor action on H/Δ is also ergodic.

Our results depend essentially on the following result, which is a special case of Theorems A and B in [R]:

Theorem 3.3. Ratner’s Orbit Closure Theorem: Let U be a connected Lie subgroup of G generated by unipotent elements, and let x ∈ G/Γ. Then there is a closed connected subgroup H containing U such that \overline{Ux} = Hx and H_{x} is a lattice in H. Moreover, if U = \{u(t) : t ∈ ℝ\} is a one-parameter subgroup, the positive semi–orbit {u(t)x : t ≥ 0} is dense in Hx.

We will need some additional properties of the group H appearing in the conclusion of Theorem 3.3:

Theorem 3.4. Let the notation be as in Theorem 3.3 (recall that G = SL(n, ℝ) and n ≥ 3). Then:

1. H is an ℝ-subgroup of G.

2. H_{x} is a Zariski dense lattice in H.

3. The unipotent radical of H is equal to the radical of H.

4. Comm_{H}(H_{x}) is dense in H.

Proof: This was proved by Nimish Shah in section 3 of [Sh], but not stated explicitly. For the reader’s convenience, we show in Appendix A how to deduce this result from Shah’s results.

We also need the following ([Sh], Lemma 2.2):

Proposition 3.5. If H_{1} and H_{2} are two closed subgroups of G such that H_{i}z is closed for some z ∈ G/Γ, then (H_{1} ∩ H_{2})z is also closed.

Proposition 3.6. Let z ∈ G/Γ. Then for any finite subset Λ ∈ G_{z}, C_{G}(Λ)z is closed.

Proof: This is Lemma 1.14 of [Ra].

4. Invariant Sets for General Lattices

In this section we will prove Theorem 1.1. We will need some lemmas.

Lemma 4.1. Let V be a connected unipotent subgroup normalized by A, and let z ∈ G/Γ be such that Az is compact. Then \overline{V z} = Hz, where H is connected, semisimple, and A ⊂ N_{G}^{1}(H).

Proof: From Theorem 3.3 it follows that there exists a connected closed subgroup H such that \overline{V z} = Hz and H_{z} is a lattice in H.

Let a ∈ A_{z}. We have

aHa^{−1}z = aHz = a\overline{V z} = \overline{aV z} = \overline{V az} = \overline{V z} = Hz,

and since Γ is discrete this implies that Ad(a)(Lie(H)) = Lie(H). Since H is connected, a ∈ N_{G}(H), and since A_{z} is Zariski dense in A, A normalizes H. Conjugation by elements of A_{z} preserves H_{z} and therefore preserves H/H_{z}, which has finite volume. H is unimodular since it contains the lattice H_{z}, and therefore the volume of H/H_{z} is just the haar measure of any fundamental domain. The Jacobian for the volume change when conjugating by a is | det(Ad(a)|_{Lie(H)})| and this implies that for a ∈ A_{z}, det(Ad(a)|_{Lie(H)})^{2} = 1. Since A_{z} is Zariski dense we obtain that A ⊂ N_{G}^{1}(H).

Let R denote the unipotent radical of H. By Theorem 3.4, R is also the radical of H. It is invariant under automorphisms of H and therefore A normalizes R. By Lie’s theorem, the group AR can be brought into triangular form, and thus if R is nontrivial then a ↦ det(Ad(a)|_{Lie(R)}) is a nontrivial character on A. Thus to show that R is trivial it will suffice to show that

(2) | det(Ad(a)|_{Lie(H)})| = | det(Ad(a)|_{Lie(R)})|.

To see this, notice that a induces an automorphism of H/R, that is it acts as a bundle automorphism of the bundle R → H → H/R. Since H/R is semisimple it is unimodular and has a finite group of outer automorphisms, thus an automorphism of H/R does not affect haar measure on H/R. Now considering H as the total space of the bundle R → H → H/R and disintegrating the haar measure on H into the haar measure on R and H/R yields (2).

Thus R must be trivial and therefore H is semisimple.

The following lemma is the core of our argument.

Lemma 4.2. Suppose z ∈ F is such that Az is compact and for every 1 ≤ i ≠ j ≤ n, N_{ij} acts ergodically on Az. Suppose H is a semisimple subgroup of G, normalized by A, such that Hz is closed, H_{z} is a lattice in H, and AHz is properly contained in F. Then there exists 1 ≤ i ≠ j ≤ n such that U_{ij} ⊄ H and U_{ij}z ⊂ F.

Proof: The proof of the lemma will be divided into steps.

Step 4.1. Let A_{0} = A ∩ C_{G}(H). Then A_{0}z is compact.

Since Az is compact, by Proposition 3.5 it suffices to show that C_{G}(H)z is closed. By Theorem 3.4 H_{z} is Zariski dense in H, hence C_{G}(H) = C_{G}(H_{z}) and by the descending chain condition for algebraic groups, there is a finite subset Λ ⊂ H_{z} such that C_{G}(H) = C_{G}(Λ). Now we may use Proposition 3.6.

Step 4.2. Let

Δ = {(i, j) : 1 ≤ i ≠ j ≤ n, U_{ij} ⊂ H ∪ U^{−}}.

With no loss of generality we may assume that there is there is an element a = exp(a) such that az = z and

λ_{ij}(a) > 0 whenever (i, j) ∉ Δ, and λ_{ij}(a) ≤ 0 whenever (i, j) ∈ Δ. (3)

First notice that neither the conditions nor the conclusions of the Lemma are affected if we replace z by P_{σ}z, H by P_{σ}HP_{σ}^{−1} and F by P_{σ}F, where P_{σ} is a permutation matrix corresponding to some σ ∈ S_{n}. So using Proposition 3.1 we may assume that H is in ‘block form’ and moreover that the B_{i} in (1) all consist of consecutive integers. Under this assumption Lie(A_{0}) consists of traceless diagonal matrices diag(a_{1}, …, a_{n}) with a_{j} = a_{k} whenever j, k ∈ B_{i} for some i, and by Proposition 3.1 H is of finite index in C_{G}(A_{0}) and hence Ad(A_{0}) acts nontrivially on Lie(U_{ij}) whenever U_{ij} ⊄ H. This implies that there is a ∈ A_{0}, a = exp(a) such that (3) is satisfied. Since (A_{0})_{z} is cocompact in A_{0} by Step 4.1, we can choose such an a so that in addition az = z.

Step 4.3. Let Ψ, Ψ̂ be box maps for z. There is a sequence of elements (u_{k}^{+}, u_{k}^{−}) ∈ V^{+} × V^{−}, k = 1, 2, …, such that at least one of u_{k}^{+}, u_{k}^{−} is not in Lie(H) and such that Ψ(u_{k}^{+}, u_{k}^{−}) ∈ F and (u_{k}^{+}, u_{k}^{−}) → 0.

If y_{k} ∈ F − AHz satisfy y_{k} → z, we can assume that for all k, y_{k} is in the image of Ψ̂, and write

Ψ̂(0) = z ← y_{k} = Ψ̂(a_{k}, u_{k}^{+}, u_{k}^{−}).

Then (u_{k}^{+}, u_{k}^{−}) are nontrivial, at least one of them is not in Lie(H), they converge to 0, and Ψ(u_{k}^{+}, u_{k}^{−}) ∈ F since y_{k} ∈ F and F is A-invariant.

Step 4.4. There is a nontrivial u ∈ Lie(U^{+}) ∪ Lie(U^{−}) such that exp(u)z ∈ F and

U_{ij} ⊂ H ⇒ P_{ij}(u) = 0.

By Step 4.3, there are elements of the form u_{k}^{+}u_{k}^{−}z in F, with u_{k}^{+} → e and u_{k}^{−} → e, and at least one of u_{k}^{+}, u_{k}^{−} is not in H. Assume with no loss of generality that for all k, u_{k}^{+} ∉ H, and that both u_{k}^{+} and u_{k}^{−} are nontrivial for all k (otherwise there is nothing to prove). Let

Δ = {(i, j) : 1 ≤ i ≠ j ≤ n, U_{ij} ⊂ H ∪ U^{−}}.

It will suffice to find R > 1 such that for each ϵ > 0, there is an element of F in

V_{ϵ,R,H} = {Ψ(u^{+}, u^{−}) : 1 ≤ ∥u^{+}+u^{−}∥ ≤ R, (i, j) ∈ Δ ⇒ |P_{ij}(u^{+}+u^{−})| < ϵ}.

Let a = exp(a) be as in step 4.2. Let

R_{1} = min{e^{λ_{ij}(a)} : (i, j) ∉ Δ}

and

R_{2} = max{e^{λ_{ij}(a)} : (i, j) ∉ Δ}.

We have 1 < R_{1} ≤ R_{2}. Since

au^{+}u^{−}z = au^{+}a^{−1} au^{−}a^{−1}z,

we get that for all (u^{+}, u^{−}) ∈ V^{+} × V^{−}, aΨ(u^{+}, u^{−}) = Ψ(u_{1}^{+}, u_{1}^{−}) with

R_{1}|P_{ij}(u^{+}+u^{−})| ≤ |P_{ij}(u_{1}^{+}+u_{1}^{−})| ≤ R_{2}|P_{ij}(u^{+}+u^{−})| whenever (i, j) ∉ Δ

and

|P_{ij}(u_{1}^{+} + u_{1}^{−})| ≤ |P_{ij}(u^{+} + u^{−})| whenever (i, j) ∈ Δ.

Therefore if we choose (u^{+}, u^{−}) such that Ψ(u^{+}, u^{−}) ∈ F, u^{+} + u^{−} ∉ Lie(H) and ∥u^{+}+u^{−}∥ < ϵ, for some natural k we will have a^{k}Ψ(u^{+}, u^{−}) ∈ V_{ϵ,R_{2},H}.

Step 4.5. For some 1 ≤ i ≠ j ≤ n, there is u_{ij} ≠ 0 such that exp(u_{ij})z ∈ F and exp(u_{ij}) ∉ H.

Let u be an element as in Step 4.4. Define

Δ′ = {λ_{ij} : P_{ij}(u) ≠ 0}.

By Step 4.4,

λ_{ij} ∈ Δ′ ⇒ U_{ij} ⊄ H.

Let λ = λ_{rs} ∈ Δ′ be extremal in the sense that it is not in the convex cone over Δ′ − {λ} (in Lie(A)^{∗}). Then we can find a ∈ Lie(A) ≅ Lie(A)^{∗∗} such that λ(a) = 0 and α(a) < 0 for every α ∈ Δ′ − {λ}. Since Az is a torus, the orbit of z under any element of A is recurrent and therefore there is a subsequence k_{ℓ} → ∞ such that a^{k_{ℓ}}z → z, where a = exp(a). Thus:

F ∋ a^{k_{ℓ}} exp(u)z

= a^{k_{ℓ}} exp( ∑_{λ_{ij}∈Δ′} u_{ij})a^{−k_{ℓ}}a^{k_{ℓ}}z

= exp(Ad(a)^{k_{ℓ}}( ∑_{λ_{ij}∈Δ′} u_{ij}))a^{k_{ℓ}}z

= exp( ∑_{λ_{ij}∈Δ′} e^{k_{ℓ}λ_{ij}(a)}u_{ij})a^{k_{ℓ}}z

⟶_{ℓ→∞} exp(u_{rs})z.

Thus (r, s) satisfies the required conclusions.

Step 4.6. Let 1 ≤ i ≠ j ≤ n be as in 4.5. Then U_{ij}z ⊂ F.

Let λ = λ_{ij},

Lie(A)_{z} = {a ∈ Lie(A) : exp(a)z = z},

and

Q = {λ(a) : a ∈ Lie(A)_{z}} ⊂ ℝ.

Since Lie(A)_{z} is a lattice in Lie(A), Q is either discrete or dense. But if Q is discrete then N_{ij} ∩ A_{z} is a lattice in N_{ij}, and this contradicts the assumption that N_{ij} acts ergodically on Az. Therefore Q is dense.

Now we have for q ∈ Q:

exp(e^{q}u_{ij})z = exp(e^{λ_{ij}(a)}u_{ij})z
= exp(Ad(a)u_{ij})z
= a exp(u_{ij})a^{−1}z
= aua^{−1}z
= auz ∈ F,

where a = exp(a) ∈ A_{z}, u = exp(u_{ij}).

Therefore by continuity, exp(tu_{ij})z ∈ F for all t ≥ 0, and now it follows from the last assertion in Theorem 3.3 that exp(tu_{ij})z ∈ F for all t ∈ ℝ.

Remark: The main obstacle to extending the results of this paper to noncompact simple groups G other than SL(n, ℝ) is the fact that the statement that H is the connected component of C_{G}(A ∩ C_{G}(H)) holds for G = SL(n, ℝ) but not for other G. This was used in an in the proof of Step 4.2 of the preceding Lemma.

Proof of Theorem 1.1: Let V be a connected subgroup of G of maximal possible dimension satisfying the following:

1. V is unipotent.

2. V x ⊂ F.

3. V is normalized by A. By Ratner’s theorem 3.3 there is a connected closed subgroup H of G such that \overline{V x} = Hx and H_{x} is a lattice in H. By Lemma 4.1, H is semisimple and normalized by A. Let H′ = AH. Then H′ is reductive. From Proposition 3.1 we see that A = (A ∩ H) · A_{0}, where A_{0} = A ∩ C_{G}(H), hence H′ = A_{0} · H. From Lemma 4.2, step 4.1 it follows that A_{0}x is compact. It follows from this that H′x is closed; indeed, if a_{k}h_{k}x → z where a_{k} ∈ A_{0} and h_{k} ∈ H, let a_{k}x = a′_{k}x where a′_{k} are in a compact subset of A_{0} and a′_{k} → a_{0}. Then

z ← a_{k}h_{k}x = h_{k}a_{k}x = h_{k}a′_{k}x = a′_{k}h_{k}x,

hence a_{0}h_{k}x → z so z ∈ H′x. Also, H′x supports the finite H′-invariant measure obtained by pushing forward the A_{0} ×H-invariant measure on A_{0}x × Hx via the map (a_{0}x, hx) ↦ a_{0}hx.

Thus the theorem will be proved if we show that F = H′x. Suppose the contrary is true; we will reach a contradiction by finding a connected subgroup V′ of G satisfying conditions 1,2,3 above and such that dim V′ > dim V. Once again we divide our argument into steps.

Step 4.7. The set

D = {z ∈ Hx : Az is compact, N_{ij} acts ergodically on Az}

is dense in Hx.

By Theorem 3.4, Comm_{H}(H_{x}) is dense in H. For each q ∈ Comm_{H}(H_{x}), A_{qx} ∩ A_{x} is of finite index in A_{x} and is therefore cocompact in A, and similarly, for 1 ≤ i ≠ j ≤ n, (N_{ij})_{qx} is not a lattice in N_{ij}. Thus qx ∈ D for every q ∈ Comm_{H}(H_{x}).

Step 4.8. There exist 1 ≤ i ≠ j ≤ n such that U_{ij} ⊄ H and U_{ij}Hx ⊂ F.

For each 1 ≤ i ≠ j ≤ n such that U_{ij} ⊄ H, let

D_{ij} = {z ∈ Hx : U_{ij}z ⊂ F }.

Then D_{ij} is closed. By step 4.7, there is a dense set of points in Hx for which the hypotheses of Lemma 4.2 are satisfied. Therefore Hx ⊂ ⋃ D_{ij}. By the Baire category theorem, one of the D_{ij}’s contains an open subset of Hx. Each of the D_{ij} is A ∩ H-invariant and by Moore’s Theorem 3.2 A∩H acts topologically transitively on Hx; thus there is 1 ≤ i ≠ j ≤ n for which Hx = D_{ij}, proving the claim.

Suppose with no loss of generality that U_{ij} ⊂ U^{+}. Let us also replace V with H ∩ U^{+}, which we may since dim H ∩ U^{+} = dim V. Denote dim V by d. Write

U_{ij} = {u_{ij}(t) : t ∈ ℝ}.

By step 4.8, for all t,

u_{ij}(t)V u_{ij}(t)^{−1}u_{ij}(t)x ⊂ F.

By Ratner’s theorem there is a sequence t_{k} → ∞ such that u_{ij}(t_{k})x →_{k→∞} x. This implies that if g is a limit point of

{u_{ij}(t_{k})Hu_{ij}(t_{k})^{−1} : k = 1, 2, … },

then gx ∈ F. Therefore we can increase d, and conclude the proof, by taking V′ = U_{ij}V_{0} where V_{0} is as in the following

Step 4.9. The limit lim_{t→∞} Ad(u_{ij}(t))(Lie(V)) exists in the Grassmannian manifold G of d-dimensional subspaces of Lie(G), and is the Lie subalgebra of a unipotent subgroup V_{0} of G not containing U_{ij}, normalized by A and by U_{ij}.

The map t ↦ Ad(u_{ij}(t))(Lie(V)) is a polynomial map from ℝ to the projective variety G; i.e., considering G as a subset of projective space ℙ^{M} for suitable M, this map is the composition of a polynomial ϕ : ℝ → ℝ^{M+1} with the natural projection ℝ^{M+1} → ℙ^{M}. Let N be the highest degree of any of the polynomials determining the coordinates of ϕ, and let p be the point in ℙ^{M} corresponding to the coefficients of the monomials of degree N. Then it is clear that p = lim_{t→∞} Ad(u_{ij}(t))(Lie(V)).

Since the Lie subalgebras form a closed subset of G, p corresponds to a Lie subalgebra of Lie(G). Let V_{0} be the corresponding connected subgroup of U^{+}. A normalizes V_{0} and by definition it is clear that U_{ij} normalizes V_{0}. Since V_{0} ⊂ U^{+}, V_{0} is unipotent.

It remains to show that U_{ij} ⊄ V_{0}. Write 𝔘^{0} = Lie(U^{+}), 𝔘^{i+1} = [𝔘^{0}, 𝔘^{i}], and let k be the largest integer for which Lie(U_{ij}) ⊂ 𝔘^{k}. Let 𝔙 = Lie(V) + 𝔘^{k+1}. Then 𝔙 is invariant under Ad(U_{ij}) and does not contain U_{ij}. For every t, Ad(u_{ij}(t))(Lie(V)) ⊂ 𝔙, and therefore p ∈ 𝔙. Therefore Lie(U_{ij}) ⊄ p = Lie(V_{0}).

5. Rees’ Example and Related Constructions

In an unpublished manuscript Rees showed how to construct uniform lattices Γ in G = SL(3, ℝ) such that Γ_{i} = Γ∩H_{i} is a lattice in H_{i}, where

H_{1} = \{( \begin{matrix}∗ & ∗ & 0 \\ ∗ & ∗ & 0 \\ 0 & 0 & ∗ \end{matrix} )\}

and

H_{2} = \{( \begin{matrix}∗ & ∗ & 0 \\ ∗ & ∗ & 0 \\ 0 & 0 & 1 \end{matrix} )\} .

From this construction Rees obtained A-invariant closed subsets of G/Γ which are contained in H_{1}/Γ_{1} and are circle bundles over an arbitrary orbit-closure for A_{2} = A ∩ H_{2} in H_{2}/Γ_{2}. Note that H_{2} is isomorphic to SL(2, ℝ). Thus any irregular orbit-closure for the action of A_{2} on H_{2}/Γ_{2} – that is, for the geodesic flow on the unit tangent bundle to a finite volume Riemann surface – can appear as the base of a bundle which is an orbit-closure for A in G/Γ. As is well-known, for these one-parameter flows many irregular orbit-closures can appear –e.g., orbit-closures which have fractional Hausdorff dimensions, orbitclosures which contain but are not equal to periodic orbits, etc.

In this section we show that a construction similar to Rees’ is possible for the action of the diagonal matrices on SL(n, ℝ)/Γ whenever assumption 1 but not assumption 2 of Theorem 1.1 holds. That is, we show

Proposition 5.1. Let G, A, Γ be as in Theorem 1.1. Let x ∈ G/Γ such that Ax is compact and N_{ij} does not act ergodically on Ax. Then there exists a subgroup H_{2} of G which is isomorphic to SL(2, ℝ) and normalized by A such that:

1. H_{2}x is closed and admits an H_{2}-invariant finite measure.

2. Let H_{1} = AH_{2}. Then H_{1}x is closed.

3. Let A_{0} = A ∩ C_{G}(H_{2}) and A_{2} = A ∩ H_{2}. Then for any y ∈ H_{2}x, A_{0}y is compact and \overline{Ay} = A_{0}\overline{A_{2}y}.

Proof: Since Ax is compact and N_{ij} does not act ergodically on Ax, (N_{ij})_{x} is a cocompact lattice in N_{ij}. By Zariski density, we obtain the existence of γ ∈ (N_{ij})_{x} such that γ = diag(a_{1}, …, a_{n}) with a_{i} = a_{j} and a_{k} ≠ a_{l} whenever {k, l} ≠ {i, j}. Let H_{1} be the connected component of the identity in C_{G}(γ), and let H_{2} be the subgroup generated by U_{ij} and U_{ji}. Then obviously H_{2} is isomorphic to SL(2, ℝ), A normalizes H_{2} and AH_{2} ⊂ H_{1}. By considering dimensions we see that H_{1} = AH_{2}.

By Proposition 3.6, H_{1}x is closed. By Theorem 3.3, \overline{U_{ij}x} = Hx for some subgroup H of G containing U_{ij}, and there is a finite H-invariant measure on H_{x}. Since H_{1}x is closed we must have that H ⊂ H_{1}, and since, by Lemma 4.1, H is semisimple, we must have that H = H_{2}.

Arguing as in Step 4.1 in the proof of Lemma 4.2 we see that A_{0}x is compact. Therefore for any y ∈ H_{2}x, say y = h_{2}x for h_{2} ∈ H_{2}, we get that

A_{0}y = A_{0}h_{2}x = h_{2}A_{0}x

is compact. Now let a_{k}y → z, where a_{k} ∈ A. Write a_{k} = a_{k}^{2}a_{k}^{0} with a_{k}^{2} ∈ A_{2} and a_{k}^{0} ∈ A_{0}. Then for each k there is a_{k}^{1} ∈ A_{0} such that a_{k}^{1}y = a_{k}^{0}y and a_{k}^{1} → a_{0}. Thus

z ← a_{k}^{2}a_{k}^{0}y = a_{k}^{2}a_{k}^{1}y = a_{k}^{1}a_{k}^{2}y

and therefore

a_{k}^{2}y → a_{0}^{−1}z.

This implies that z ∈ A_{0}\overline{A_{2}y}.

Proposition 5.1 shows that any irregular orbit for the geodesic flow on SL(2, ℝ)/Λ (where Λ is an appropriate lattice) yields an irregular orbit for the action of A on G/Γ. For instance, Theorem 1.2 now follows from Proposition 5.1 and the following simple

Lemma 5.2. Let Λ be a lattice in SL(2, ℝ) and D = {diag(e^{t}, e^{−t}) : t ∈ ℝ}. Then for any x ∈ SL(2, ℝ)/Λ such that Dx is a periodic orbit, there exists y ∈ SL(2, ℝ)/Λ such that Dy is open in \overline{Dy} and x ∈ \overline{Dy}.

Proof: Let U^{+} (respectively, U^{−}) be the subgroup of SL(2, ℝ) consisting of upper (respectively lower) triangular unipotent matrices. Then U^{−}DU^{+} is a dense open subset of SL(2, ℝ). Since the periodic Dorbits in SL(2, ℝ)/Λ are dense (see [A]), there is z ∈ SL(2, ℝ)/Λ, z = u^{−}du^{+}x, such that Dz is periodic and Dz ≠ Dx. Let y = u^{+}x. From the commutation relations for D, U^{+}, U^{−} it is then obvious that \overline{Dy} = Dy ∪ Dz ∪ Dx.

6. Orbit-Closures for Γ = SL(n, ℤ)

In this section we examine the case Γ = SL(n, ℤ) in more detail. First we show how to explicitly construct orbit-closures for A in G/Γ that contain a compact A-orbit, and are not equal to an A-orbit or to G/Γ. We then prove Theorem 1.3. In the proof of Theorem 1.3 it develops that all orbit-closures containing a compact orbit are like the ones in the example. Example:

First let us construct some compact orbits. These orbits are closely related to the norm tori of [PlRa, page 54]. Let K be a totally real number field, with [K : ℚ] = n, and let L be the splitting field for K. We take O(K) to be the ring of algebraic integers of K, and we recall that O(K) considered as an additive group is isomorphic to ℤ^{n}. Now take R < O(K) an additive subgroup of finite index. Let α_{1}, …, α_{n} be generators for R. We further set Λ = Gal(L/ℚ), and Λ_{K} the subgroup of Λ that acts trivially on K. Recall that [Λ : Λ_{K}] = n. Let {σ_{1} = id, σ_{2}, …, σ_{n}} be a set of representatives of the cosets of Λ/Λ_{K}. For any unit θ ∈ O(K)^{∗}, take

a(θ) = diag(σ_{1}(θ), …, σ_{n}(θ)) ∈ A.

We now take g ∈ GL(n, ℝ) to be

g = (σ_{i}(α_{j}))_{i,j=1...n}. (4)

and

ḡ = det(g)^{−1/n}gΓ.

We note that det(g) is fixed by Λ, so det(g) ∈ ℚ ∩ O(L) = ℤ.

A_{0} = g^{−1}Ag is a norm torus in the sense of [PlRa, page 54]. There it is shown that A_{0} is defined over ℚ and has no ℚ-characters. By [BoH], A_{0} ∩ Γ is a lattice in A_{0}, so Aḡ is compact.

Suppose that a = diag(a_{1}, …, a_{n}) ∈ A_{0} ∩ Γ. Then ag = gγ for some γ ∈ Γ. In particular (carrying out the matrix multiplication in the first row),

a_{1}α_{j} = ∑_{i=1}^{n} α_{i}γ_{ij},

so a_{1} = α_{j}^{−1} ∑_{i=1}^{n} α_{i}γ_{ij} ∈ K, and also a_{1}R ⊂ R so a_{1} ∈ O(K). Computing similarly in the ℓth row we get

a_{1}σ_{ℓ}(α_{j}) = ∑_{i=1}^{n} σ_{ℓ}(α_{i})γ_{ij},

and hence

a_{ℓ} = σ_{ℓ}(α_{j}^{−1}) ∑_{i=1}^{n} σ_{ℓ}(α_{i})γ_{ij} = σ_{ℓ}(α_{j}^{−1} ∑_{i=1}^{n} α_{i}γ_{ij}) = σ_{ℓ}(a_{1}).

Therefore ∏_{ℓ=1}^{n} σ_{ℓ}(a_{1}) = det a = 1, and we have a_{1} ∈ O(K)^{∗}. Thus a = a(a_{1}) ∈ a(O(K)^{∗}).

Note that by Dirichlet’s Unit Theorem [Sa, Theorem 4.4.1], a(O(K)^{∗}) is also a lattice in A, so Aḡ = g(A_{0} ∩ Γ)g^{−1} is a subgroup of finite index in a(O(K)^{∗}).

Now let us construct a closed A-invariant set containing such a compact orbit. Assume there is some intermediate field ℚ < K′ < K. Set

d = [K′ : ℚ]
k = [K : K′].

Since Aḡ is of finite index in a(O(K)^{∗}) ≅ O(K)^{∗}, there is

θ′ ∈ A^{−1}O(K′) ∩ Aḡ

such that ℚ(θ′) = K′. Let H = C_{G}(a(θ′)). Since a(θ′) ∈ Aḡ, we know that a(θ′)ḡ = ḡ, hence by Proposition 3.6 Hḡ is closed, and in addition since A ⊂ H it contains the compact orbit Aḡ. Since ℚ(θ′) = K′, θ′ has exactly d conjugates, and so the n-tuple

σ_{1}(θ′), σ_{2}(θ′), …, σ_{n}(θ′)

consists of d different elements of L each appearing with multiplicity k. This shows that for some permutation matrix P,

H = \{P ( \begin{matrix}B_{1} & 0 & ⋯ & 0 \\ 0 & B_{2} & ⋯ & 0 \\ . & . & . \\ 0 & 0 & ⋯ & B_{d} \end{matrix} ) P^{−1} : B_{1}, …, B_{d} ∈ GL(k, ℝ)\} ∩ G. (5)

We now show that Hḡ is an orbit–closure of some y ∈ G/Γ.

Proposition 6.1. H/Hḡ admits a finite H-invariant measure, and A acts ergodically H/Hḡ ≅ Hḡ. In particular, there is a dense orbit in Hḡ.

Remark: We remark that even though H/Hḡ has finite volume as we shall see later it is not compact.

Proof: Let H_{0} = g^{−1}Hg = C_{G}(g^{−1}a(θ′)g) and A_{0} = g^{−1}Ag where g is as in (4), and let H′_{0} = [H_{0}, H_{0}]. By our choice of θ′, we have g^{−1}a(θ′)g ∈ Γ and hence H_{0} is defined over ℚ. Therefore H_{0} is also defined over ℚ. It follows from (5) that H_{0} = A_{0}·H′_{0}, thus any character on H_{0} whose restriction to A_{0} and H′_{0} is trivial must be trivial. Thus, since A_{0} has no ℚ-characters, and H′_{0} has no characters at all, H_{0} does not have ℚ-characters.

Now using a Theorem of Borel and Harish-Chandra ([BoH]) we obtain that H_{0}/(H_{0} ∩ Γ) (respectively, H′_{0}/(H′_{0} ∩ Γ)) admits a finite H_{0}- (respectively, H′_{0}-) invariant measure. From Moore’s Theorem 3.2 we then conclude that A′_{0} = A_{0} ∩ H′_{0} acts ergodically on H′_{0}/(H′_{0} ∩ Γ), and since H_{0} = A_{0} · H′_{0}, we obtain that A_{0} acts ergodically on H_{0}/(H_{0} ∩ Γ) and hence that A acts ergodically on Hḡ.

We now prove Theorem 1.3. In the course of the proof it will develop that any orbit-closure for A containing a compact orbit arises as in the example above. We will first need the following result, which is probably well–known, and gives a more explicit description of all compact orbits in G/Γ:

Lemma 6.2. Let ḡ = gΓ ∈ G/Γ be such that Aḡ is compact. Let A_{0} = g^{−1}Ag. Then there is a Galois number field L and a matrix g_{0} ∈ GL(n, L) such that g_{0}g is a diagonal matrix, A_{0} = g_{0}^{−1}Ag_{0}, and Gal(L/ℚ) permutes the rows of g_{0} transitively. In particular, if a_{1}, …, a_{n} ∈ ℚ then the Galois group Gal(L/ℚ) acts on g_{0}^{−1}diag(a_{1}, …, a_{n})g_{0} ∈ A_{0}(L), by permuting the a_{i}’s transitively.

Proof: Since Γ_{0} = Γ ∩ A_{0} is Zariski dense in A_{0}, there is γ ∈ Γ_{0} with distinct eigenvalues. Let L be a splitting field for the minimal polynomial P_{γ} of γ. Let us show that P_{γ} is irreducible over ℚ. If not, there is a nontrivial γ-invariant proper ℚ-subspace V ⊂ ℝ^{n} (see e.g. [HK], Theorem 6.12). Since γ has distinct eigenvalues and A_{0} = C_{G}(γ), V is also an A_{0}-invariant subspace. The character a ↦ det(a|_{V}) is then a nontrivial ℚ-character and since it is trivial on integer matrices and Γ_{0} is Zariski-dense, it must be trivial on A_{0}. This however implies that dim A_{0} ≤ n − 2, which is impossible.

As a consequence Λ = Gal(L/ℚ) acts transitively on the eigenvalues of γ.

We know that there are θ_{1}, …, θ_{n} ∈ L such that

gγ = diag(θ_{1}, …, θ_{n})g.

Thus the rows of g are left eigenvectors for γ. On the other hand, since γ ∈ SL(n, ℤ) there is a row vector v_{1} ∈ ℚ(θ_{1})^{n} satisfying

v_{1}γ_{1} = θ_{1}v_{1}.

The group Gal(L/ℚ) permutes the eigenvalues of γ, and so the orbit of v_{1} under Gal(L/ℚ) are n linearly independent eigenvectors: v_{1} with eigenvalue θ_{1}, v_{2} with eigenvalue θ_{2}, etc. Let

g_{0} = ( v_{1} ; ⋮ ; v_{n} ) ,

i.e., the v_{i}’s are the rows of the matrix g_{0}. It is clear that g_{0} ∈ GL(n, L). Also, since each row of g_{0} spans a one-dimensional eigenspace for γ, and the eigenvalues are permuted transitively, so are the eigenspaces. In particular, the ith row of g is the same as v_{i} up to multiplication by a scalar, and so g_{0}g^{−1} is a diagonal matrix. This implies that g_{0}^{−1}Ag_{0} = g^{−1}Ag. Remark: Lemma 6.2 shows in particular that any compact orbit arises as in the example, with K = ℚ(α) for some eigenvalue α of γ.

We are now ready for the

Proof of Theorem 1.3: Our first step is show that the assumptions of Theorem 1.1 are satisfied automatically:

Step 6.1. If Ax is compact then for any 1 ≤ i ≠ j ≤ n, N_{ij}x is not compact.

Since Ax is a torus, it is enough to find some z ∈ Ax such that N_{ij}z is not compact. Thus we can replace x with z = ĝ_{0}Γ, where g_{0} is as in Lemma 6.2, and

ĝ_{0} = (\frac{1}{det g_{0}} )^{1/n}g_{0}.

If N_{ij}z is compact then N_{0} = g_{0}^{−1}N_{ij}g_{0} intersects Γ co-compactly and is therefore defined over ℚ. Therefore the action of Λ leaves N_{0}(L) invariant. By Lemma 6.2, Λ permutes the eigenvalues of elements of the dense subset g_{0}^{−1}A(ℚ)g_{0}. Therefore for any σ ∈ Λ there is a permutation (also denoted by σ) of {1, …, n} such that N_{0}^{σ} = g_{0}^{−1}N_{σ(i)σ(j)}g_{0}. That is, for every σ ∈ Λ, {σ(i), σ(j)} = {i, j}. This contradicts the fact that the action of Λ on {1, …, n} is transitive and n ≥ 3.

Step 6.2. Let H be as in the conclusion of theorem 1.1. Then H is in ‘equal blocks form’, i.e., there is a partition

{1, …, n} = B_{1} ⊔ … ⊔ B_{d}, (6)

where all the B_{ℓ} are of the same cardinality, such that

Lie(H) = Lie(A) ⊕ ⨁_{ℓ=1}^{d}( ⨁_{i,j∈B_{ℓ}} 𝔊_{ij}). (7)

The proof of Theorem 1.1 shows that H = A · S where S is semisimple, normalized by A, and S_{x} is a lattice in S. S consists of those block matrices in H such that the determinant of each block is 1. Moreover, if x = ĝ_{0}Γ where ĝ_{0} = (det g_{0})^{−1/n}g_{0}, g_{0} as in Lemma 6.2, then Theorem 3.4 shows that S_{0} = g_{0}^{−1}Sg_{0} is defined over ℚ and S_{0}(ℤ) is a Zariski dense lattice in S_{0}. Using Lemma 6.2 we can also assume that there is a Galois field L over ℚ such that Λ = Gal(L/ℚ) acts on g_{0}^{−1}S(ℚ)g_{0} by permuting the rows of g_{0}. Thus for each σ ∈ Λ there is an associated permutation (also denoted by σ) of {1, …, n} and a corresponding permutation matrix P_{σ}, such that for

h ∈ g_{0}^{−1}S_{0}(ℚ)g_{0}, h = g_{0}^{−1}h′g_{0}

we have

h^{σ} = g_{0}^{−1}P_{σ}h′P_{σ}^{−1}g_{0}.

Thus the action of σ on {1, …, n} preserves the partition ( 6). Since Λ is transitive on {1, …, n}, for every 1 ≤ ℓ_{1} ≠ ℓ_{2} ≤ d there is σ ∈ Λ such that B_{ℓ_{1}} ∩ σ(B_{ℓ_{2}}) ≠ ∅ and thus σ(B_{ℓ_{1}}) = B_{ℓ_{2}}. Therefore all the B_{ℓ} have the same cardinality.

Step 6.3. If H ≠ A then Hx is noncompact.

If H ≠ A there is some 1 ≤ i ≠ j ≤ n such that U_{ij} ⊂ H. Thus it will suffice to prove that AU_{ij}zΓ is an unbounded subset of G/Γ for any z ∈ G. For this we adapt the argument of [CaS-D].

We claim first that there are z′ ∈ AU_{ij}z, v ∈ ℤ^{n} − {0}, and some 1 ≤ ℓ ≤ n such that < z′_{ℓ}, v >= 0, where z′_{i} is the ith row of z′ and < ·, · > is the standard inner product on ℝ^{n}. Indeed if for z there are no such v then for any v ∈ ℤ^{n} − \{0\} we can let t = −\frac{<z_{j},v>}{<z_{i},v>} and z′ = (1 + tE_{ij})z. Then < z′_{i}, v >=< z_{j} + tz_{i}, v >= 0.

Now we claim that Az′Γ is unbounded. Recall that by Mahler’s compactness criterion (see [Ra], Corollary 10.9) it is enough to find a_{k} ∈ A and v_{k} ∈ ℤ^{n} −{0} such that ∥a_{k}z′v_{k}∥ →_{k→∞} 0. This is satisfied by v_{k} = v and any a_{k} = diag(a_{1}^{k}, …, a_{n}^{k}) ∈ A such that for all i ≠ ℓ, a_{i}^{k} →_{k→∞} 0.

Remarks:

1. The proof of Theorem 1.3 shows that any orbit-closure for A arises as in the example, with K′ = ℚ(α), where α is an eigenvalue of a generic matrix in C_{G}(H) ∩ A_{x}.

2. If one is only interested in a proof of Corollary 1.4, the following shorter proof is available. Suppose \overline{Ay} ≠ Ay. Then from Lemma 4.2 (which is easier in this case since H = {e}) there is U such that \overline{Ux} ⊂ \overline{Ay}. Then \overline{Ux} = Hx by Ratner’s theorem, and by Lemma 4.1 and Step 6.2, H = G.

7. An Isolation Result

In this section n ≥ 3 is prime and Γ = SL(n, ℤ). We first state a corollary similar to Corollary 1.4:

Corollary 7.1. Let F be a closed A-invariant subset of G/Γ containing a compact orbit Ax, and suppose that F − Ax is not closed. Then F = G/Γ.

Proof: Arguing as in the proof of Lemma 4.2, we obtain a unipotent subgroup U such that Ux ⊂ F. By Ratner’s theorem \overline{Ux} = Hx and by Lemma 4.1 and Step 6.2, H = G.

Now we turn to

Proof of Corollary 1.5: Any product h of n linearly independent forms can be represented by n vectors in ℝ^{n}. If we place these vectors as rows in a matrix, which we denote by h̃, we get an element of GL(n, ℝ). Note that h̃ is not uniquely defined by h, but this will be of no importance. For our argument it will suffice to restrict attention to those h̃ ∈ G.

Let

R(h) = {(h(v_{1}), …, h(v_{n−1})) : {v_{1}, …, v_{n−1}} is primitive}.

Note that for every a ∈ A and every γ ∈ Γ, R(h̃) = R(ah̃γ). If the assertion is untrue, there is an open V ⊂ ℝ^{n−1} and a sequence of forms h_{k} such that for each k, V ∩ R(h̃_{k}) = ∅, h_{k} → f in the space of forms, and the h_{k} are not multiples of f. By multiplying h̃_{i} and f̃ with the inverse of its determinant, we may assume that h̃_{i} ∈ G for all i ∈ ℕ and f̃ ∈ G. Let

F = \overline{A\{h̃_{k} : k = 1, 2, … \}Γ}.

Our assumption imples that V ∩ R(h̃) = ∅ for every h̃ ∈ F. The projection of F to G/Γ is an A-invariant closed set. It contains f̃Γ. We will show momentarily that Af̃Γ is compact. This will complete the proof since F −Af̃Γ is not closed and then by Corollary 7.1, F = G, contradicting the easily established fact there exists g ∈ G such that V ∩ R(g̃) ≠ ∅.

Let us show that Af̃Γ is closed, or equivalently that ÃΓ is closed, where Ã = f̃^{−1}Af̃. Since f is a product of n linearly independent form, Ã is the connected component of A_{f} the stabilizer of f in G, where G acts on the right by f^{g}(v) = f(gv). Since Ã is normal and of finite index in A_{f}, to prove that ÃΓ is closed it is enough to prove that A_{f}Γ is closed. So let a_{k} ∈ A_{f}, γ_{k} ∈ Γ such that a_{k}γ_{k} → z. Then

f^{z} ← f^{a_{k}γ_{k}} = f^{γ_{k}}.

For each k the form f^{γ_{k}} also has integer coefficients and is therefore contained in a discrete subset of the set of products of n forms. Hence there is γ_{0} such that f^{z} = f^{γ_{0}}, and therefore zγ_{0}^{−1} stabilizes f. Thus z ∈ A_{f}Γ.

Now suppose Af̃Γ is unbounded. By Mahler’s compactness criterion (see [Ra], Corollary 10.9) this implies that there are a_{k} ∈ A and v_{k} ∈ ℤ^{n} − {0} such that ∥a_{k}f̃v_{k}∥ → 0. If we write a_{k} = diag(a_{1}^{k}, …, a_{n}^{k}) and f = L_{1} · . . . · L_{n}, where L_{i} =< w_{i}, · >, then we get

max_{i} |a_{i}^{k}L_{i}(v_{k})| →_{k→∞} 0

and therefore

ℤ − {0} ∋ f(v_{k}) = ∏_{i=1}^{n} a_{i}^{k}L_{i}(v_{k}) →_{k→∞} 0.

Appendix A. Proof of Theorem 3.4

In [Sh], Shah proves this theorem uncer the hypotheses that G is an algebraic group defined over ℚ, Γ = G(ℤ) and x is the coset Γ. More precisely, assertions 1 and 3 of the proposition are proved in [Sh], Proposition 3.2. Assertion 4 follows since

H(ℚ) ⊂ Comm_{H}(H(ℤ)),

and assertion 2 is Corollary 2.13 of [Sh]. We are interested in the case G = SL(n, ℝ), n ≥ 3. This case can be easily reduced to Shah’s results using Margulis’ arithmeticity theorem (see [Z], Chapter 6), as follows:

Suppose x = gΓ. Replacing H with g^{−1}Hg and U with g^{−1}Ug shows that there is no loss of generality in assuming that g = e. Recall that the arithmeticity theorem states that there is an algebraic group G′ defined over ℚ and a surjective homomorphism ρ : G′ → G such that K′ = ker ρ is compact and Γ is commensurable with ρ(G′(ℤ)). Let U′ ⊂ G′ be a connected unipotent subgroup such that ρ(U′) = U (such a group exists since there is an embedding Lie(G) ⊂ Lie(G′)). Let Γ′ be a subgroup of finite index in G′(ℤ) such that ρ(Γ′) is a finite index subgroup of Γ, and let x′ be the coset Γ′. Then the map ρ̄: G′/Γ′ → G/Γ defined by ρ̄(g′Γ′) = ρ(g′)Γ is a proper map.

Let H′ be a connected subgroup containing U′ such that \overline{U′G′(ℤ)} = H′G′(ℤ). By [Sh], H′ satisfies the conclusions of the Theorem. Let us show that also \overline{U′x′} = H′x′. By Ratner’s orbit-closure theorem, \overline{U′x′} = H′′x′. Since H′G′(ℤ) is closed and the map G′/Γ′ → G′/G′(ℤ) is continuous and equivariant, H′x′ is closed and contains U′x′, and hence H′′ ⊂ H′. Also

H′ ⊂ \overline{U′G(ℤ)} = \overline{⋃_{γ} U′Γ′γ} = ⋃_{γ} \overline{U′Γ′}γ = ⋃_{γ} H′′Γ′γ,

where the union is over a finite set of representatives of cosets in Γ′\G′(ℤ). Therefore we must have dim H′ = dim H′′ and hence by connectedness H′ = H′′.

By Ratner’s orbit-closure theorem there is a connected group H such that \overline{Ux} = Hx. We claim that H = ρ(H′). Indeed, since ρ̄ is proper, \overline{Ux} ⊂ ρ(H′)x and since U′x′ is dense in H′x′, its image ρ̄(U′x′) = Ux is dense in ρ̄(H′x′) = ρ(H)x. Thus Hx = ρ̄(H′)x and since Γ is discrete and H and ρ(H′) are connected, H = ρ(H′). Thus the assertions that H is an ℝ-subgroup and that the unipotent radical of H is equal to its radical follow from the corresponding ones for H′.

Now we claim that ρ(H′_{x′}) is of finite index in H_{x}. It is obvious that ρ(H′_{x′}) ⊂ H_{x} and since both are lattices in H, the assertion follows.

Thus the remaining assertions about H_{x} follow from the corresponding ones for H′_{x′}.

References

[A] D. V. Anosov, Geodesic Flows on Closed Riemannian Manifolds of Negative Curvature, Proc. Steklov Math. Inst. 90 (1967), no. 5.

[B] D. Berend, Multi-Invariant Sets on Tori, Trans. Amer. Math. Soc. 280 no. 2 (1983) 509-532.

[BoH] A. Borel and Harish-Chandra, Arithmetic Subgroups of Algebraic Groups, Annals of Math. 75 (1962) 485-535.

[BoPr] A. Borel and G. Prasad, Values of Isotropic Quadratic Forms at S-integral Points, Compositio Math. 83 (1992) 347-372.

[CaS-D] J. W. S. Cassels and H. P. F. Swinnerton-Dyer, On the Product of Three Homogeneous Forms and Indefinite Ternary Quadratic Forms, Phil. Trans. Roy. Soc. of London, ser. A 248 (1955) 73-96.

[F] H. Furstenberg, Disjointness in Ergodic Theory, Minimal Sets, and a Problem in Diophantine Approximation, Math. Sys. Theory, Vol. 1 1 (1966) 1-49.

[HK] K. Hoffman and R. Kunze, Linear Algebra (1961) Prentice Hall, Englewood Cliffs, NJ.

[KS] A. Katok and R. J. Spatzier, Invariant Measures for Higher Rank Hyperbolic Abelian Actions, Erg. Th. and Dyn. Sys. 16 (1996) 751-778, and Corrections to ‘Invariant Measures for Higher Rank Hyperbolic Abelian Actions’, Erg. Th. and Dyn. Sys. 18 (1998), no. 2, 503–507.

[M1] G. A. Margulis, Dynamical and Ergodic Properties of Subgroup Actions on Homogeneous Spaces with Applications to Number Theory, Proc. of ICM (Kyoto, 1990), Math. Soc. of Japan and Springer (1991) 193-215.

[M2] G. A. Margulis, Oppenheim Conjecture, Fields Medallists’ lectures, 272–327, World Sci. Ser. 20th Century Math., 5, World Sci. Publishing, River Edge, NJ, 1997.

[M3] G. A. Margulis, Problems and Conjectures in Rigidity Theory, preprint (1999).

[Mo] S. Mozes, On Closures of Orbits and Arithmetic of Quaternions, Isr. J. of Math. 86 (1994) 195-209.

[PlRa] V. Platonov and A. Rapinchuk, Algebraic Groups and Number Theory, (1994) Academic Press, San Diego, CA.

[Ra] M. S. Raghunathan, Discrete Subgroups of Lie Groups (1972) Springer, Berlin.

[R] M. Ratner, Raghunathan’s Topological Conjecture and Distribution of Unipotent Flows, Duke J. of Math. 63 (1991) 235-280.

[Sa] P. Samuel, Algebraic Theory of Numbers 1970 Hermann, Houghton, Mifflin, Boston.

[Sh] N. A. Shah, Uniformly Distributed Orbits of Certain Flows on Homogeneous Spaces, Math. Ann. 289 (1991) 315-334.

[Z] R. J. Zimmer, Ergodic Theory and Semisimple Groups, 1984, Birkhauser, Boston.