-->

Wednesday, July 27, 2016

Exchange 2016 Installing Exchange In A Resource Forest: Part 4

This is a continuation from Part 3 of my Installing Exchange 2016 in a Resource Forest series.

I've also created this guide as an eBook which you can grab here.

Request and Import a new UC certificate

Generate the Cert Request (CSR)

1.      On one Mailbox Server, open the Exchange Admin Center (EAC) and navigate to Servers > Certificates.
       2.      Hit “+” button
       3.      Choose to create a request for a certificate from a certification authority
       4.      Enter a friendly name for the cert (can be anything you want). You’ll see this name in the list of certificates installed on the server, so make it something that you will easily recognize; maybe call your new certificate something like “Exchange 2016 UC Cert”.

Although you can technically use a Wildcard cert, don’t select Wildcard – it makes things painful later on.

5.      Choose a server to store the cert request on. This server will be used to complete the request, and will be the first server that has the certificate installed.
       6.      Now, just hit “Next” because we’ll select the SSL names on the next screen.
       7.      At the next step you can select and remove any unwanted names, edit existing names, or add more names to the certificate request.

The most important ones we need will be:

Mail.domain.com
Autodiscover.domain.com
            Domain.com

**Note** The server name will be present, and won’t hurt anything. You’ll also need to add any extra domains you will be using. For instance us.domain.com or domain.org.

You will need to pay for each additional domain name, but it depends on your provider, so it’s best to only include the ones you need.

8.      Enter your Organization info.
       9.      Enter a UNC path to save the request on: \\EXCH-MBX-01\c$\temp\exchcert.req
       10.  Click Finish and submit that .req to your Certificate Authority like DigiCert or GoDaddy.

Complete the Pending Cert Request

1.      Download the cert file provided from your CA to C:\Temp
       2.      In the EAC, go to Servers > Certificates
       3.      Click the Pending Request, and in the right-hand pane, click Complete near the bottom.
       4.      Enter the UNC for the cert file: \\EXCH-MBX-01\c$\temp\newcert.cer

If successful, it will show as “Valid”.

Assign the Cert to Services


Once your cert is installed, you can assign it to Exchange services such as IIS, SMTP, etc.

1.      Still in Servers > Certificates, select the new SSL cert, and click the “Pencil” button.
       2.      Check every box that you need – most times you’ll need IIS, SMTP, POP, IMAP. If you run UM, check those too.
       3.      Click “Save”.
       4.      You will be prompted to overwrite the existing SMTP service, click “Yes”.

Configure Outlook Anywhere


1.      In the EAC, go to Servers, and double-click your first server.
       2.      Choose Outlook Anywhere.
       3.      Set your namespace for Internal and External host names to match your namespaces you used in the Set Namespace section i.e. Mail.domain.com
       4.      And set NTLM for the auth method.

Import certificates on the Load Balancer

Each Load Balancer is different, but for Kemp follow these steps:

1.      Export your certificate from Exchange.

On the Load Balancer:
      
       2.      Go to Virtual Services > View/Modify Services.
       3.      Click the Add New button under the Certificate Installed Column.
       4.      Click Import Certificate in the upper-right, then next to Certificate File, click Browse.
       5. Select your .pfx file that you exported from Exchange, input the passphrase, and specify the cert name: Exchange 2016 UC Cert
       6.      Click “Save
       7.      On the Cert Config screen, select the VIP in the “Available VSs” and hit the right arrow to move it to the Assigned VSs box.
       8.     Save Changes
       9.      Now back at the View/Modify Services page, you can see the cert is assigned to the VIP.

Now test your pings and nslookups to ensure that mail and autodiscover resolve to the LB and open OWA and Outlook to ensure you don’t get any cert prompts.
                
All Done!

You should now have your Exchange Resource Forest set up and functioning behind your Load Balancer, with mail flowing in and out through your Edge server.


Now create some linked mailboxes by following my previous post:

Exchange 2016 Installing Exchange In A Resource Forest: Part 3

This is a continuation from Part 2 of my Installing Exchange 2016 in a Resource Forest series.

I've also created this guide as an eBook which you can grab here.
 
Setup POP and IMAP

If using POP and IMAP, run these cmdlets on each server:

Set-POPSettings -ExternalConnectionSetting {mail.domain.com:995:SSL}

Set-ImapSettings -ExternalConnectionSetting {mail.domain.com:993:SSL}

Set-POPSettings -X509CertificateName mail.domain.com

Set-IMAPSettings -X509CertificateName mail.domain.com

Next, start the POP and IMAP services and set to Automatic on each server

Redistribute (balance) the Database across the DAG


To redistribute the database across the DAG according to activation preference, run the following cmdlets in the EMS:

cd $exscripts

 Then, run:

.\RedistributeActiveDatabases.ps1 -DagName "DAG01" -BalanceDbsByActivationPreference -Confirm:$False  

Edge Server Setup

On a stand-alone server in the DMZ (not joined to the domain), configure the server name with a DNS suffix matching your Exchange Forest:

Right-click “This PC” > Properties > Advanced System Settings > Computer Name > Change > More.

Enter the DNS suffix to match your Exchange Forest, like so:

EXCH-EDG-01.resourcedomain.com

**Note** We’re only changing the suffix, leave the server in the workgroup it is currently in.

Reboot the Edge server for the name to take effect.

In the NIC properties, set a static IP and add your Exchange Forest DC\DNS servers as DNS providers, and mark the “register the IP in DNS” checkbox.

The Edge Server needs to be able to resolve the mailbox servers by name and vice-versa.

Ports that need to be open in your firewalls:

Port TCP 25 in/out between the Edge and the internet

Port TCP 25 in/out between the Edge and the internal LAN

Port TCP 50636 from internal LAN to the DMZ

Next install the Exchange Edge pre-reqs by running the cmdlet in an Elevated Windows PowerShell:

Install-WindowsFeature ADLDS

Run Windows Update and install all update except for .Net 4.6.1 – its best to hide that update.

Download the most current Exchange 2016 CU ISO to the C:\TEMP\CU folder and extract it.

In an Elevated CMD prompt, run:

cd C:\TEMP\CU

And then run the following command:

setup /m:install /r:et /IAcceptExchangeServerLicenseTerms

Reboot the server and install any other needed updates.

Edge Subscription Creation


On the Edge server, in the EMS, run:

New-EdgeSubscription -FileName C:\Temp\Edge.xml

Copy the EdgeSubscription.xml file to C:\Temp on any Mailbox server.

On the mailbox server you copied the .xml file to, in the EMS, run:

New-EdgeSubscription -FileData ([byte[]]$(Get-Content -Path "C:\Temp\Edge.xml" -Encoding Byte -ReadCount 0)) -Site " resourcedomain.com/Configuration/Sites/Default-First-Site-Name"

On your Mailbox Severs in the EAC > Mail Flow > Send Connectors, you will see the new EdgeSync Send Connectors.

**Note** If you previously had any Send Connectors set up, you’ll need to remove those.

You will need to ensure that your MX records point to the Public IP of your Edge Server – which should be NAT’d behind your firewall on Port 25.

Verify inbound/outbound mailflow.

Load Balancer Setup

Gareth Gudger (SuperTekBoy) has written an excellent guide on setting up a Kemp Load Balancer for Exchange.

Follow his guide here:


CAS Setup

Set DNS Entries


We will be using Split-DNS – meaning on your internal DNS mail.domain.com will resolve to your internal VIP of your Load Balancer; while on the External DNS, mail.domain.com will resolve to the Public IP of your Firewall, which will NAT to your Load Balancer VIP.

Create the DNS A-Record for “mail” on Internal and External DNS.

This will be an A-Record for mail, pointing to the VIP of your Load Balancer, for instance:

Internally - Mail > 10.10.128.120

Externally – mail.domain.com > 74.74.72.190

Configure your autodiscover record pointing to your Load Balancer for both Internal and External DNS.

Internal – autodiscover > 10.10.128.120

External – autodiscover.domain.com > 74.74.72.190

Create Namespace


Use Paul Cunningham’s (ExchangeServerPro) awesome script to automatically set your namespaces in one shot.


Next, follow Part 4 here

Exchange 2016 Installing Exchange In A Resource Forest: Part 2

This is a continuation from Part 1 of my Installing Exchange 2016 in a Resource Forest series.

I've also created this guide as an eBook which you can grab here.

Create an IP-less DAG

 **Note** You must create a Witness upon initial setup because Exchange 2016 on Server 2012R2 uses "dynamic quorum" for when a node goes down.

Create the Witness Server

**Note** I always run the Witness Share on a server that runs Exchange Management Tools – that is not an Exchange Server.

Stand up a member server called something like EXCH-MGMT-WIT, and add the following permissions:

Since the Witness Share resides on a non-Exchange server, you need to add the Exchange Trusted Subsystem group to the Local Administrators Group on the server - this means it cannot be on a Domain Controller since there are no local groups.

Create the DAG and set the Witness Server to EXCH-MGMT-WIT on C:\DAG01FSW, by running the following in the EMS:

New-DatabaseAvailabilityGroup -Name DAG01 -DatabaseAvailabilityGroupIPAddresses ([System.Net.IPAddress]::None) -WitnessServer EXCH-MGMT-WIT.resourcedomain.com –WitnessDirectory “C:\ DAG01FSW“

Add Mailbox Servers to the DAG

Run the following in the EMS:
 
             Add-DatabaseAvailabilityGroupServer -identity DAG01 –MailboxServer "EXCH-MBX-01"
             Add-DatabaseAvailabilityGroupServer -identity DAG01 –MailboxServer "EXCH-MBX-02"
             Add-DatabaseAvailabilityGroupServer -identity DAG01 –MailboxServer "EXCH-MBX-03"

Enable DAC (datacenter activation coordination) mode on the DAG to prevent split-brain syndrome during fail-back(s) by running:

         Set-DatabaseAvailabilityGroup -Identity DAG01 -DatacenterActivationMode DagOnly

Create DAG Mount Points

Perform the following steps on each Exchange server – it is very important that volumes and folders match exactly on each server.

You should already have your E: and F: Volumes presented to your servers as drives.

1.      On the C: drive, create a folder called EXVols – this folder will be used to mount our E: (Volume1) and F: (Volume2).
          2.      Next, on the C: drive, create a folder called ExDBs – this folder will hold the Database mount points.
          3.      Creating the Volumes
          4.      Within the ExVols folder, create two new folders called Volume1 and Volume2.
          5.      Open Windows Disk Management to mount our two volumes to our ExVols folders.
          6.      Right-click E: and select Change Drive Letter and Paths…
          7.      Click Add and browse to the location of the Volume1 folder – C:\ExVols\Volume1
          8.      Click OK, twice
          9.      Right-click F: and select Change Drive Letter and Paths…
         10.  Click Add and browse to the location of the Volume2 folder – C:\ExVols\Volume2

You should see the folders with Disk icons meaning they are now Mount Points. 

Creating the Database Folders

Under the C:\ExDBs folder, create the new Database folders for as many DB’s as you plan to have. In my case, we have 6, so we’ll create the following folders:

C:\ExDBs\DB01
          C:\ExDBs\DB02
          C:\ExDBs\DB03
          C:\ExDBs\DB04
          C:\ExDBs\DB05
          C:\ExDBs\DB06

After you have your folders set, open an Elevated command prompt, and run:
        
          mountvol

This will list the available volumes for use.

In our case we know the one we want is \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\
because we can see the folder Volume1 is mounted to it

Run the following command to mount DB01:

Mountvol DB01 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

If you go to your C:\ExDBs, you’ll notice the folder icon for DB01 has changed to a mount point icon.

Now mount your other DB’s:

Mountvol DB02 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB03 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB04 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB05 \\?\Volume{eeadb719-54af-4384-9c90-78dbf04acf86}\

Mountvol DB06 \\?\Volume{03cf7f78-ed05-4bb7-a4f0-0914f9575bdd}\

If you run mountvol again, you’ll see all DB’s mounted under the Volume1 folder.

Now set up your Archive DB’s on Volume2 like the above.

Run mountvol, and for my setup, the F: Volume2 is \\?\Volume{b4d8eb69-2c04-11e6-80d8-806e6f6e6963}\

Run the following commands to mount those Arch DB’s:

Mountvol Arch01 \\?\Volume{b4d8eb69-2c04-11e6-80d8-806e6f6e6963}\

Mountvol Arch02 \\?\Volume{b4d8eb69-2c04-11e6-80d8-806e6f6e6963}\

Mountvol Arch03 \\?\Volume{b4d8eb69-2c04-11e6-80d8-806e6f6e6963}\

Mountvol Arch04 \\?\Volume{b4d8eb69-2c04-11e6-80d8-806e6f6e6963}\

Mountvol Arch05 \\?\Volume{b4d8eb69-2c04-11e6-80d8-806e6f6e6963}\

Mountvol Arch06 \\?\Volume{b4d8eb69-2c04-11e6-80d8-806e6f6e6963}\

Creating the Database Directory Structure

Next, we’ll create database directory structure; each folder will have 2 folders beneath it: one folder for the Database .edb file and one for the Logs.

**Note** It is best practice to keep database and log files on the same volume, as long as it is separated from the System Volume. So, all DB’s and Logs will be on Volume1 and Archive DB’s and Logs will be on Volume2.

You can create these folders directly from Volume1 (E:) or by going to C:\ExDBs\DB01 through DB06 (they will have the same folders).

In E:\ExDBs\DB01, create a new folder named DB01.db and new folder called DB01.log.

In E:\ExDBs\DB02, create a new folder named DB02.db and new folder called DB02.log.

In E:\ExDBs\DB03, create a new folder named DB03.db and new folder called DB03.log.

In E:\ExDBs\DB04, create a new folder named DB04.db and new folder called DB04.log.

In E:\ExDBs\DB05, create a new folder named DB05.db and new folder called DB05.log.

In E:\ExDBs\DB06, create a new folder named DB06.db and new folder called DB06.log.

Do the same for your Archive Databases:

In F:\ArchDBs\Arch01, create a new folder named Arch01.db and new folder called Arch01.log.

In F:\ArchDBs\Arch02, create a new folder named Arch02.db and new folder called Arch02.log.

In F:\ArchDBs\Arch03, create a new folder named Arch03.db and new folder called Arch03.log.

In F:\ArchDBs\Arch04, create a new folder named Arch04.db and new folder called Arch04.log.

In F:\ArchDBs\Arch05, create a new folder named Arch05.db and new folder called Arch05.log.

In F:\ArchDBs\Arch06, create a new folder named Arch06.db and new folder called Arch06.log.

Create Mailbox Databases

We’ll be creating our six Databases, and evenly distributing them across our servers.

DB01 and DB04 on Server01

DB02 and DB05 on Server02

DB03 and DB06 on Server03.

Create the Databases by running the following cmdlets in Exchange Management Shell (EMS):

New-MailboxDatabase –Name DB01 –Server EXCH-MBX-01 –LogFolderPath C:\ExDBs\DB01\DB01.log –EdbFilePath C:\ExDBs\DB01\DB01.db\DB01.edb

New-MailboxDatabase –Name DB02 –Server EXCH-MBX-02 –LogFolderPath C:\ExDBs\DB02\DB02.log –EdbFilePath C:\ExDBs\DB02\DB02.db\DB02.edb

New-MailboxDatabase –Name DB03 –Server EXCH-MBX-03 –LogFolderPath C:\ExDBs\DB03\DB03.log –EdbFilePath C:\ExDBs\DB03\DB03.db\DB03.edb

New-MailboxDatabase –Name DB04 –Server EXCH-MBX-01 –LogFolderPath C:\ExDBs\DB04\DB04.log –EdbFilePath C:\ExDBs\DB04\DB04.db\DB04.edb

New-MailboxDatabase –Name DB05 –Server EXCH-MBX-02 –LogFolderPath C:\ExDBs\DB05\DB05.log –EdbFilePath C:\ExDBs\DB05\DB05.db\DB05.edb

New-MailboxDatabase –Name DB06 –Server EXCH-MBX-03 –LogFolderPath C:\ExDBs\DB06\DB06.log –EdbFilePath C:\ExDBs\DB06\DB06.db\DB06.edb

Do the same for your Archive Databases:

New-MailboxDatabase –Name Arch01 –Server EXCH-MBX-01 –LogFolderPath C:\ArchDBs\Arch01\Arch01.log –EdbFilePath C:\ArchDBs\Arch01\Arch01.db\Arch01.edb

New-MailboxDatabase –Name Arch02 –Server EXCH-MBX-02 –LogFolderPath C:\ArchDBs\Arch02\Arch02.log –EdbFilePath C:\ArchDBs\Arch02\Arch02.db\Arch02.edb

New-MailboxDatabase –Name Arch03 –Server EXCH-MBX-03 –LogFolderPath C:\ArchDBs\Arch03\Arch03.log –EdbFilePath C:\ArchDBs\Arch03\Arch03.db\Arch03.edb

New-MailboxDatabase –Name Arch04 –Server EXCH-MBX-01 –LogFolderPath C:\ArchDBs\Arch04\Arch04.log –EdbFilePath C:\ArchDBs\Arch04\Arch04.db\Arch04.edb

New-MailboxDatabase –Name Arch05 –Server EXCH-MBX-02 –LogFolderPath C:\ArchDBs\Arch05\Arch05.log –EdbFilePath C:\ArchDBs\Arch05\Arch05.db\Arch05.edb

New-MailboxDatabase –Name Arch06 –Server EXCH-MBX-03 –LogFolderPath C:\ArchDBs\Arch06\Arch06.log –EdbFilePath C:\ArchDBs\Arch06\Arch06.db\Arch06.edb

**Note** You will get a Warning that The Information Store must be restarted after DB creation - this is by design. Exchange 2013/2016 uses different memory management so that store.exe does not use all available RAM. MS suggests DB creation is during maintenance window, since restarting the store.exe service dismounts databases active on that server…even though that is annoying.

Add Database Copies


We will use Postpone Seeding to allow the copy creation to finish before seeding.

Note the Activation Preference (AP), which mounts the copy according to server:

DB
EXCH-MBX-01 -AP
EXCH-MBX-02 -AP
EXCH-MBX-03 -AP
DB01
1
2
3
DB02
3
1
2
DB03
3
2
1
DB04
1
2
3
DB05
3
1
2
DB06
3
2
1
ArchDB
EXCH-MBX-01 -AP
EXCH-MBX-02 -AP
EXCH-MBX-03 -AP
Arch01
1
2
3
Arch02
3
1
2
Arch03
3
2
1
Arch04
1
2
3
Arch05
3
1
2
Arch06
3
2
1

Run the following cmdlets in the EMS to create the DB copies according to activation preference:

**Note** Run each cmdlet separate on each line.

Add-MailboxDatabaseCopy -Identity DB01 -MailboxServer EXCH-MBX-02 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB01 -MailboxServer EXCH-MBX-03 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB02 -MailboxServer EXCH-MBX-03 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB02 -MailboxServer EXCH-MBX-01 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB03 -MailboxServer EXCH-MBX-02 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB03 -MailboxServer EXCH-MBX-01 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB04 -MailboxServer EXCH-MBX-02 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB04 -MailboxServer EXCH-MBX-03 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB05 -MailboxServer EXCH-MBX-01 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB05 -MailboxServer EXCH-MBX-01 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB06 -MailboxServer EXCH-MBX-02 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity DB06 -MailboxServer EXCH-MBX-03 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch01 -MailboxServer EXCH-MBX-02 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch01 -MailboxServer EXCH-MBX-03 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch02 -MailboxServer EXCH-MBX-03 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch02 -MailboxServer EXCH-MBX-01 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch03 -MailboxServer EXCH-MBX-02 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch03 -MailboxServer EXCH-MBX-01 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch04 -MailboxServer EXCH-MBX-02 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch04 -MailboxServer EXCH-MBX-03 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch05 -MailboxServer EXCH-MBX-03 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch05 -MailboxServer EXCH-MBX-01 -ActivationPreference 3 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch06 -MailboxServer EXCH-MBX-02 -ActivationPreference 2 -SeedingPostponed

Add-MailboxDatabaseCopy -Identity Arch06 -MailboxServer EXCH-MBX-01 -ActivationPreference 3 -SeedingPostponed

Check Database file Creation

Go to C:\ExDBs on Server01 and get the Properties on the folder, it shouldn’t be taking up any space (should be 0 bytes) – this is because the data is actually stored on Volume1 not on C:\.

Now open C:\ExDBs\DB01\DB01.db and DB01.log and you should see the .edb file and the logs in their respective folders.

Delete default databases from each server


Run the following cmdlets in the EMS to delete the default Databases:

Get-mailbox -database "default database name" –arbitration

Get-mailbox -database "default database name" -arbitration  | new-moverequest -targetdatabase "new database name"

**Note** You’ll need to do this for each server (the cmdlet can be run from one server though). Also change “default database name” to the default database you are moving from, and the “new database” you are moving to.

**Note** Exchange 2013/2016 creates a mailbox for the admin account that is used for the install; you’ll need to move or delete that mailbox before deleting the Default Database.

You cannot use get-mailbox -database "db name" | new-moverequest -targetdatabase "database name" because the pipe somehow cannot connect to the First Administrative Group

You must use new-moverequest -identity "admin user alias" -targetdatabase "new database name"
        
After moving arbitration and admin mailboxes, delete the databases and delete the .edb files and logs from each server.

If required for Unified Messaging, download and install any Exchange 2016 Language Packs on each server from the following link:


Disable Autoprovision on Archive DBs


Disable auto-provision on Archive databases, so your provisioning script doesn't put regular user mailboxes in those by letting Exchange pick the most available DB

Set-MailboxDatabase "archive database name" -IsExcludedFromProvisioning $true

Create a New Anonymous Relay Connector


If you need printers and devices to be able to send anonymous messages through Exchange, you’ll need a new Frontend transport Receive Connector to allow those connections.

Run the following in EMS:

New-ReceiveConnector "Internal Anonymous Relay" -usage custom -bindings 0.0.0.0:25 -TransportRole FrontendTransport -RemoteIPRanges $connector.RemoteIPRanges

Set quotas on databases DB01-DB06 and Arch01-Arch06

Get-MailboxDatabase | Set-MailboxDatabase -IssueWarningQuota 3GB  -ProhibitSendReceiveQuota 5GB -ProhibitSendQuota unlimited

**Note** You can set whatever quotas you need, mine is just an example. You aslo must set the ProhibitSendQuota as it has to be populated, even if unlimited.