Zimbra Disaster Recovery 8.8.15

Zimbra recovery from the backup folder.

Suppose we have an email service at mymail.reducted.com . But we faced disaster and our email server was lost. So how can we recover the service from this incident?

Fortunately, we have the Zimbra folder backup. So we go for a try to recover the mail server with this folder. So I am going to discuss the processes we can follow to restore the email system.

In response to the catastrophic incident that affected The email server hosted on mymail.reducted.com, we can execute a comprehensive disaster recovery plan. The recovery plan was formulated with the primary goal of restoring email functionality and ensuring minimal data loss, given that we had access to a backup of the /opt/zimbra folder. Below is an assessment of the steps taken during the recovery process:

• Infrastructure Alignment: We initiated the recovery process by setting up a new server with the same operating system as the previous email server. This ensured compatibility with the restored Zimbra data.
• Zimbra Installation: We proceeded to install Zimbra version 8.8.15, replicating the configuration that existed on the previous server. This step was crucial for maintaining consistency in our email system.
• Data Restoration: The backup of the /opt/zimbra folder was replaced on the new server, ensuring that we had a complete snapshot of the previous email system.
• Dependency Resolution: We addressed any broken dependencies or missing components required for Zimbra’s proper functioning on the new server.
• Linux Environment Fixes: We meticulously resolved any issues in the Linux environment that could potentially impede the operation of Zimbra, guaranteeing a stable foundation.
• Service Restart: Services associated with Zimbra were restarted to bring the email system back online, while closely monitoring for any issues or errors.
• Mailbox Database Recovery: Special attention was given to the recovery of mailbox databases to ensure that users’ email data was restored without data loss.
• Custom Upgrade Script: To enhance the upgrade process of the mail server, we developed and executed a customized installation script of Zimbra to upgrade the Zimbra installation as part of our recovery process.
• Post-Configuration: Following the successful recovery of the email server, we conducted post-configuration tasks to optimize performance, security, and resilience.

This is mentioned to use the same OS and Zimbra version to get recovery from this folder.

So firstly, we installed the CentOs 7, recommended to configure the machine as before.

as our mail server was mymail.reducted.com our configure was

sudo hostnamectl set-hostname mymail --static

Need host entry. Our IP was 10.10.123.123. so edit /etc/hosts

10.10.123.123 mymail.reducted.com mymail

Zimbra 8.8.15 foss edition. After downloading the Zimbra installation archive we extracted it. All the installation commands ran as root.

bash install.sh -s

Running this command will ask for some agreement, we have to go with that flow. This will install only the required software package on the machine. This won’t start any Zimbra services like ldap, zimlet, mailboxd etc.

After that, we moved the installed Zimbra folder from /opt/zimbra location to /opt/zimbra_old. The we moved our backup zimbra folder to the location /opt/.

There may be some permission error, so we have fixed the permission switched to Zimbra user and checked for the error.

/opt/zimbra/libexec/zmfixperms --verbose --extended
su - zimbra
postfix check

So we have found the java is unable to found that Java is unable to locate a library call. Then we added the shell variable so that Java could find that as Zimbra user.

vi ~/.bashrc
#added this line
export LD_LIBRARY_PATH=/opt/zimbra/common/lib/jvm/java/lib:$LD_LIBRARY_PATH
#This source the file
source ~/.bashrc

Then we tried the run the Zimbra service with:

zmcontrol start

After this command, some of the services got errors as below:

1. LDAP was unable to read due to dhparam file.
2. zmconfigd due to java cacert file error
3. mailboxd was unable to start due to MySQL error
4. nginx was not starting due to dhaparam file missing
5. webapps was not running due to mailboxd failure.

when we started the zmcontrol we continuously checked the logs.

tail -f /opt/zimbra/log/*.log
tail -f /var/log/zimbra*.log

So we have fixed this issue by the below processes.

For ldap:

dhparam generate

openssl dhparam -out /opt/zimbra/conf/dhparam.pem 2048

if password-related problem -

https://wiki.zimbra.com/wiki/Upgrade_Script_Check_-_Validating_LDAP_configuration#:~:text=to do it.-,Workaround,-Note%3A Please before

For zmconfigd:

cacerts fix:

cp /opt/zimbra_old/common/lib/jvm/java/lib/security/cacerts /opt/zimbra/common/lib/jvm/java/lib/security/

For mailboxd:

During maildboxd startup, we faced MySQL having broken tablespace issues in the log and the manager restarting multiple times. so that it's sure that some of the dependency is broken.

For nginx:

nginx won't start with dhaparam file. so generate it

openssl dhparam -out /opt/zimbra/conf/dhparam.pem 2048

Stop all the services by running the below command as Zimbra user:

zmcontrol stop

Mysql data recovery

As we have faced the data tablespace is broken. so we have to recover the mysql.

So we did:-

Mysql Crash Recovery — Zimbra :: Tech Center

Meanwhile, we saw that some shell variables need to be set as below:

mysql_root_password="<mysql_password>"

mysql_socket=/opt/zimbra/data/tmp/mysql/mysql.sock

mysql_root_password can be found at /opt/zimbra/conf/localconfig.xml

Zimbra reinstallation and upgrade to recovery:

To upgrade Zimbra, The installation script checks for LDAP connection. During the installation script run it faces some issues so we will fix that problem before running the script.

LDAP TLS connectivity off:

su - zimbra 
zmlocalconfig -e ssl_allow_untrusted_certs=true 
zmlocalconfig -e ldap_starttls_supported=0
zmlocalconfig -e ldap_starttls_required=false
zmlocalconfig -e ldap_common_require_tls=0
ldap restart

ref: https://wiki.zimbra.com/wiki/Unable_to_create_a_successful_TLS_connection_to_the_ldap_masters#:~:text=certificates are used.-,Method 3%3A Disabling all TLS connections,-Run this on

Run ldap by zimbra user:

#check ldap status
ldap status

#start ldap. if already started no need to start again
ldap start

Modifying installation script:

We faced that while running the installation script, it checked for the LDAP connectivity. But though ldap is already running it somehow fails to connect to ldap using user. because this script is unable to find the ldap user's password from the environment variable somehow. So we processed to hardcode the credential to the installation script. The bin/zmValidateLdap.pl file in the installation directory is responsible for checking the LDAP connectivity. So we changed some of the code.

my $ldap_master             = "<ldap_master_url>";
my $admin_user              = "<zimbra_ldap_userdn>";
my $admin_password          = "<zimbra_ldap_password.";
my $ldap_starttls_supported = <ldap_starttls_supported>;
my $upgradeOK               = 1;

my $ldap_root_password = "<ldap_root_password>";

my $admin_user     = "<zimbra_ldap_userdn>";
my $admin_password = "<zimbra_ldap_password>";

This value can be found at /opt/zimbra/conf/localconfig.xml

Deletion of java:

While running the installation script it sometimes fails sometimes due to the Java installation directory. So its better to delete the java directory.

rm -rf /opt/zimbra/common/lib/jvm/*

Now Take a deep breath , hold your finger crossed and run the installation script as root user.

bash install.sh --platform-override --force-upgrade

This will show a log file location. Keep an eye on that. and check for any errors that occur. Also, check the Zimbra log at /var/log directory.

Hope the Zimbra mail service will run again.

Post upgradation task:

Remote Manager Exception

For Zimbra internal communication it uses ssh. so regenerate the SSH key by the below tutorial

RemoteManager exception — Zimbra:: Tech Center

Relay access denied:

If sending an email shows an error as similar:

method:	[unknown]
msg:	Invalid address: test@mymail.reducted.com. com.zimbra.cs.mailbox.MailSender$SafeSendFailedException: MESSAGE_NOT_DELIVERED; chained exception is: com.zimbra.cs.mailclient.smtp.InvalidRecipientException: RCPT failed: Invalid recipient test@mymail.reducted.com: 554 5.7.1 <test@mymail.reducted.com>: Relay access denied

ref: https://forums.zimbra.org/viewtopic.php?t=70159

Then need to check if the configured IP is the previous IP. check postconf using

postconf -n | grep "mynetworks"

If the IP is different then change as below with the appropriate IP and subnet.

change postconf using

zmprov ms mymail.reducted.com zimbraMtaMyNetworks '127.0.0.0/8 [::1]/128 10.10.123.0/26'

for ldap

zmprov gs mymail.reducted.com zimbraMtaMyNetworks '127.0.0.0/8 [::1]/128 10.10.123.0/26'

ref: https://community.zextras.com/how-to-change-ip-address-of-zimbra-server/