Tutorials‎ > ‎

Install the Oracle GlassFish v3 High Availability Cluster with Oracle Database on RedHat Enterprise Linux 6

Abbreviation  Description
 fqdn  Fully qualified domain name (FQDN), sometimes also referred as an absolute domain name
 dns  The Domain Name System (DNS) is a hierarchical distributed naming system for computers, services, or any resource connected to the Internet or a private network.

Enable XA on the Oracle Database Server

To prepare the database for XA, perform these steps:
  1. Log on to sqlplus as system user, for example, sqlplus sys/CHANGE_ON_INSTALL@<DATABASE ALIAS NAME>
  2. Execute the following command: @xaview.sql (The xaview.sql script resides in the $ORACLE_HOME/rdbms/admin directory)
  3. Grant the following permissions:
    • grant select on v$xatrans$ to public (or <user>);
    • grant select on pending_trans$ to public;
    • grant select on dba_2pc_pending to public;
    • grant select on dba_pending_transactions to public;
    • grant execute on dbms_system to <user>; (when using the Oracle Thin driver 10.1.0.3 or later)

If the above steps are not performed on the database server, normal XA database queries and updates may work fine. However, when the GlassFish Server Transaction Manager performs recovery on a re-boot after a crash, recover for the Oracle resource fails with XAER_RMERR. Crash recovery is a standard operation for an XA resource.

Oracle GlassFish Domain Administration Server (DAS)

The Domain Administration Server (DAS), is a specially-designated application server instance that hosts the administrative applications. The DAS authenticates the administrator, accepts requests from administration tools, and communicates with server instances in the domain to carry out the requests.

The DAS is sometimes referred to as the admin server or default server. It is referred to as the default server because it is the only server instance that gets created on Oracle Glassfish Enterprise Server installation and can be used for deployments. The DAS is simply a server instance with additional administration capabilities.

Each Admin Console session allows you to configure and manage a single domain. If you created multiple domains, you must start an additional Admin Console session to manage the other domains. When specifying the URL for the Admin Console, be sure to use the port number of the DAS associated with the domain to be administered.

Install the Oracle Java Development Kit (JDK)

Download the last JDK 7 RPM: http://www.oracle.com/technetwork/java/javase/downloads/index.html

Run the command below under sudo or root:
rpm -Uv jdk-7u9-linux-x64.rpm

/usr/sbin/alternatives --install /usr/bin/java java /usr/java/latest/jre/bin/java 20000
/usr/sbin/alternatives --install /usr/bin/javaws javaws /usr/java/latest/jre/bin/javaws 20000
/usr/sbin/alternatives --install /usr/bin/javac javac /usr/java/latest/bin/javac 20000
/usr/sbin/alternatives --install /usr/bin/jar jar /usr/java/latest/bin/jar 20000

Edit the file “/etc/profile” to set the environment variable JAVA_HOME:
JAVA_HOME=/usr/java/latest
PATH=$PATH:$JAVA_HOME/bin
export PATH JAVA_HOME

Install the Domain Administration Server

Download the Oracle Glassfish v3.1.2 (do not use 3.1.2.2, this version cannot be patched!) from: http://www.oracle.com/technetwork/middleware/glassfish/downloads/index.html

Download the "das-initscript.txt" below and save it as "/etc/init.d/glassfish" and run the following command under sudo or root:
cd /opt
unzip ogs-3.1.2.zip

groupadd webgroup
useradd -g webgroup -s /bin/bash webuser
chown -R webuser:webgroup /opt/glassfish3

chmod 0755 /etc/init.d/glassfish
/sbin/chkconfig --add glassfish
/sbin/chkconfig --level 234 glassfish on

Export the DAS SSL Certificate

Export the DAS certificate, this is needed to auto update load balance settings from the DAS:
keytool -export -rfc -alias s1as -keystore /opt/glassfish3/glassfish/domains/domain1/config/keystore.jks -file /opt/glassfish.crt -storepass changeit

Add additional JDBC drivers

Download the Oracle ODJBC6.jar library from http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html and copy the jar to the directories:
  • /opt/glassfish3/glassfish/domains/domain1/lib
  • /opt/glassfish3/mq/lib/ext

Activate the OpenMQ binaries

run the following command under sudo or root:
chmod +x /opt/glassfish3/mq/bin/*

Configure the defaults for the Oracle GlassFish High Availability cluster

Create the file "/opt/glassfish-passwords" as the user "webuser" with the following content:
AS_ADMIN_MASTERPASSWORD=changeit 
AS_ADMIN_PASSWORD=changeit 
AS_ADMIN_NEWPASSWORD=changeit

Create the file "/opt/glassfish-defaults" as the user "webuser" with the following content:
# Enable security settings
set server.network-config.protocols.protocol.http-listener-1.http.xpowered-by=false
create-jvm-options '-Dcom.sun.enterprise.tools.admingui.NO_NETWORK=true'
create-jvm-options -Dproduct.name="" --target default-config

# Configure the EJB timer database
create-jdbc-connection-pool --datasourceclassname oracle.jdbc.pool.OracleConnectionPoolDataSource --restype javax.sql.ConnectionPoolDataSource --property url='jdbc\:oracle\:thin\:@${database-fqdn}\:1521\:${database}':user=${user}:password=${password} ClusteredTimerPool
create-jdbc-resource --connectionpoolid ClusteredTimerPool jdbc/ClusteredTimer

# Connect the EJB container to the database
set default-config.ejb-container.ejb-timer-service.timer-datasource=jdbc/ClusteredTimer

# Configure the JMS failover
set default-config.jms-service.type=LOCAL
set default-config.availability-service.jms-availability.db-vendor=oracle
set default-config.availability-service.jms-availability.db-url='jdbc:oracle:thin:@${database-fqdn}:1521:${database}'
set default-config.availability-service.jms-availability.db-username=${user}
set default-config.availability-service.jms-availability.db-password=${password}
set default-config.availability-service.jms-availability.message-store-type=jdbc

# Set the LDAP Security Realm
create-auth-realm --classname com.sun.enterprise.security.auth.realm.ldap.LDAPRealm --property jaas-context=ldapRealm:directory='ldap\://:@${ldap-fqdn}\:389':base-dn='ou\=Users,dc\=example,dc\=org':assign-groups=authenticated:search-bind-dn='cn\=${user},ou\=Users,dc\=example,dc\=org':search-bind-password='${password}':group-base-dn='ou\=Users,dc\=example,dc\=org':group-search-filter='member\=%d':search-filter='cn\=%s' --target default-config LDAPREALM
set default-config.security-service.default-realm=LDAPREALM
set default-config.security-service.activate-default-principal-to-role-mapping=true

# Set the mailing resources
create-javamail-resource --mailhost ${smtp-fqdn} --mailuser ${user} --fromaddress donotreply@example.org --property mail.smtp.host=${smtp-fqdn}:mail.smtp.port=${smtp-port} resource/MailerSession

Run the following command under sudo or root on the DAS:
chown webuser:webgroup /opt/glassfish-* 
chmod 0600 /opt/glassfish-* 
su - webuser
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords change-admin-password
./asadmin start-domain
./asadmin --user admin --passwordfile /opt/glassfish-passwords enable-secure-admin
./asadmin restart-domain
./asadmin --user admin --passwordfile /opt/glassfish-passwords --secure multimode --file /opt/glassfish-defaults
exit
service glassfish restart

Synchronization of application configuration files

Unfortunately we need the run a separate process to keep the configuration files for the application running on top of Oracle Glassfish up to date, without compromising the high availability.

Create the directory "/var/log/sync_nodes" and download "sync_nodes_app.sh" below and save the file in the directory "/opt/".

Setting the correct file permissions (as root):
chown webuser:webgroup /opt/sync_nodes.sh
chown webuser:webgroup /var/log/sync_nodes
chmod 0700 /opt/sync_nodes.sh
chmod 0700 /var/log/sync_nodes

Run the script every 5 minutes:
su -c "crontab -e" - webuser
*/5 * * * * /opt/sync_nodes.sh

Apache HTTPD with the Oracle load balancer

The Oracle Glassfish Load balancer will not work with the Apache HTTPD server provided by Red Hat Enterprise Linux. So we need to build Apache HTTPD from source.

Install from the Red Hat Enterprise Linux sources

Install the needed development packages:
yum install gcc gcc-c++ autoconf automake openssl-devel rpm-build xmlto db4-devel expat-devel apr-devel apr-util-devel pcre-devel distcache-devel compat-libstdc++-33

Get and unpack the sources:
rpm -ivh http://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/SRPMS/httpd-2.2.3-65.el5_8.3.src.rpm

cd /usr/src/redhat/SPECS
rpmbuild -bp httpd.spec

Compile Apache HTTPD:
cd /usr/src/redhat/BUILD/httpd-2.2.3
export CFLAGS="-m64"
export CXXFLAGS="${CFLAGS}"
export LDFLAGS="${CFLAGS}"
export LD_LIBRARY_PATH=/usr/lib64/
./configure --with-mpm=worker --with-included-apr --with-ssl=/usr --prefix=/opt/httpd --enable-ssl --enable-so --with-expat=builtin
make
make install
mkdir /opt/httpd/run

Create the apache user:
groupadd apache
useradd -g apache -d /opt/httpd/htdocs -s /sbin/nologin apache

Setting directory permissions:
chown apache:apache /opt/httpd/logs
chown apache:apache /opt/httpd/run

Open the "/opt/httpd/conf/httpd.conf" file and change:
User daemon
Group daemon
In:
User apache
Group apache

Download "http-initscript.txt" below and save the file as "/etc/init.d/httpd" and run the following command under sudo or root:
chmod 0755 /etc/init.d/httpd
/sbin/chkconfig --add httpd
/sbin/chkconfig --level 234 httpd on

Install the Oracle Java Development Kit (JDK)

Download the last JDK 7 RPM: http://www.oracle.com/technetwork/java/javase/downloads/index.html

Run the command below under sudo or root:
rpm -Uv jdk-7u9-linux-x64.rpm

/usr/sbin/alternatives --install /usr/bin/java java /usr/java/latest/jre/bin/java 20000
/usr/sbin/alternatives --install /usr/bin/javaws javaws /usr/java/latest/jre/bin/javaws 20000
/usr/sbin/alternatives --install /usr/bin/javac javac /usr/java/latest/bin/javac 20000
/usr/sbin/alternatives --install /usr/bin/jar jar /usr/java/latest/bin/jar 20000

Edit the file “/etc/profile” to set the environment variable JAVA_HOME:
JAVA_HOME=/usr/java/latest
PATH=$PATH:$JAVA_HOME/bin
export PATH JAVA_HOME

Install the Oracle GlassFish load balancer

Before you begin, copy the "/opt/glassfish.crt" from the DAS server to the Apache HTTPD Server.

Download the Glassfish load balancer plugin from http://www.oracle.com/technetwork/middleware/glassfish/downloads/index.html and copy the file to the directory "/opt".
WARNING: Be sure that the version of the load balancer matches the version of the DAS!

Download "glassfish-installer xml" below and save the file as "/opt/glassfish-installer.xml" and start the Glassfish load balancer installation by using:
java -jar /opt/glassfish-lbconfigurator-*.jar /opt/glassfish-installer.xml

Configure the Oracle GlassFish load balancer

Create the loadbalancer configuration directory:
mkdir /opt/httpd/conf/loadbalancer
mv /opt/httpd/conf/glassfish-loadbalancer_1_3.dtd /opt/httpd/conf/loadbalancer
mv /opt/httpd/conf/sun-loadbalancer_* /opt/httpd/conf/loadbalancer
cp /opt/httpd/conf/loadbalancer.xml.example /opt/httpd/conf/loadbalancer/loadbalancer.xml
chmod 0644 /opt/httpd/conf/loadbalancer/loadbalancer.xml
chown -R apache:apache /opt/httpd/conf/loadbalancer

In "/opt/httpd/conf/httpd.conf" change:
<IfModule apachelbplugin_module>
                config-file "/opt/httpd/conf/loadbalancer.xml"
                locale en
</IfModule>
To:
<IfModule apachelbplugin_module>
                config-file "/opt/httpd/conf/loadbalancer/loadbalancer.xml"
                locale en
</IfModule>

Create a self-signed SSL certificate

Run the following commands as root:
cd /root
openssl genrsa -des3 -out server.key 1024
# Enter pass phrase for server.key: changeit

openssl req -new -key server.key -out server.csr
# Country Name (2 letter code) [GB]:NL
# State or Province Name (full name) [Berkshire]:Noord Brabant
# Locality Name (eg, city) [Newbury]:'s-Hertogenbosch
# Organization Name (eg, company) [My Company Ltd]:Example
# Organizational Unit Name (eg, section) []:Information Technology
# Common Name (eg, your name or your server's hostname) []:
# Email Address []:
# A challenge password []:
# An optional company name []:

cp server.key server.key.org
openssl rsa -in server.key.org -out server.key
# Enter pass phrase for server.key.org: changeit

openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

cp /root/server.crt /opt/httpd/conf/server.crt
cp /root/server.key /opt/httpd/conf/server.key

Configure the virtualhost

Disable the "<VirtualHost ${all-ipaddresses}:443>" in "/opt/httpd/conf/extra/httpd-ssl.conf".

Edit the "/opt/httpd/conf/extra/httpd-vhosts.conf" file:
NameVirtualHost *:80
<VirtualHost *:80>
    ServerAdmin admin@example.org
    Alias /probe /opt/httpd/htdocs/probe

    ErrorLog "logs/error_log"
    CustomLog "logs/access_log" common
</VirtualHost>

NameVirtualHost *:443
<VirtualHost *:443>
    ServerAdmin admin@example.org
    Alias /probe /opt/httpd/htdocs/probe

    SSLEngine on
    SSLProtocol all -SSLv2
    SSLCipherSuite HIGH:MEDIUM:!aNULL:!MD5
    SSLCertificateFile "/opt/httpd/conf/server.crt"
    SSLCertificateKeyFile "/opt/httpd/conf/server.key"

    BrowserMatch ".*MSIE.*" \
        nokeepalive ssl-unclean-shutdown \
        downgrade-1.0 force-response-1.0

    ErrorLog "logs/ssl-error_log"
    CustomLog "logs/ssl-access_log" common
</VirtualHost>

Add load balancer to the DAS

Run the following commands from the DAS as the webuser:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords create-http-lb --devicehost ${httpd-fqdn} --deviceport 443 --monitor lb-${httpd-hostname}

Show all load balancers available on the DAS

Run the following commands from the DAS as the webuser:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords list-http-lbs

Push the DAS cluster configuration to the Apache load balancers

Run the following commands from the DAS as the webuser:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords apply-http-lb-changes lb-${httpd-fqdn}

Oracle GlassFish nodes

A node represents a host on which the Oracle Glassfish Server software is installed. A node must exist for every host on which Glassfish Server instances reside. A node's configuration contains information about the host such as the name of the host and the location where the Glassfish Server is installed on the host.

Install the Oracle Java Development Kit (JDK)

Download the last JDK 7 RPM: http://www.oracle.com/technetwork/java/javase/downloads/index.html

Run the command below under sudo or root:
rpm -Uv jdk-7u9-linux-x64.rpm

/usr/sbin/alternatives --install /usr/bin/java java /usr/java/latest/jre/bin/java 20000
/usr/sbin/alternatives --install /usr/bin/javaws javaws /usr/java/latest/jre/bin/javaws 20000
/usr/sbin/alternatives --install /usr/bin/javac javac /usr/java/latest/bin/javac 20000
/usr/sbin/alternatives --install /usr/bin/jar jar /usr/java/latest/bin/jar 20000

Edit the file “/etc/profile” to set the environment variable JAVA_HOME:
JAVA_HOME=/usr/java/latest
PATH=$PATH:$JAVA_HOME/bin
export PATH JAVA_HOME

Prepare Node

Create the webuser on the node:
groupadd webgroup
useradd -g webgroup -s /bin/bash webuser
passwd webuser
# Changing password for user webuser.
# New UNIX password: changeit
# Retype new UNIX password: changeit

su - webuser
mkdir ~/.ssh
chmod 700 ~/.ssh

Create root glassfish directory on the node:
mkdir /opt/glassfish3
chown webuser:webgroup /opt/glassfish3

SSH pair the DAS and the node

Generate a key pair file on the DAS:
su - webuser
ssh-keygen
# Generating public/private rsa key pair.
# Enter file in which to save the key (/home/webuser/.ssh/id_rsa):
# Created directory '/home/webuser/.ssh'.
# Enter passphrase (empty for no passphrase): ENTER
# Enter same passphrase again: ENTER
# Your identification has been saved in /home/webuser/.ssh/id_rsa.
# Your public key has been saved in /home/webuser/.ssh/id_rsa.pub.
# The key fingerprint is:
# 13:58:d1:63:92:26:6b:54:5f:96:78:c0:de:f3:1e:de webuser@${node-fqdn}

Push the key pair files from the DAS to the node:
ssh-copy-id -i ~/.ssh/id_rsa.pub ${node-fqdn}

Install a single Oracle GlassFish node

Before you start the directory "/opt/glassfish3" needs present, the ssh user must have read/write access to this directory and the two servers needed to be ssh key paired!

The next commands need to be executed on the DAS:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords setup-ssh ${node-fqdn}
./asadmin --user admin --passwordfile /opt/glassfish-passwords install-node --installdir /opt/glassfish3 ${node-fqdn}
./asadmin --user admin --passwordfile /opt/glassfish-passwords create-node-ssh --installdir /opt/glassfish3 --nodehost ${node-fqdn} ${node-name}

Show all Oracle GlassFish nodes on the DAS
Run the following commands from the DAS as the webuser:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords list-nodes-ssh --long=true

Configure an Oracle GlassFish High Availability cluster

Please be aware that GlassFish stores node information in the database by creating a table named "${cluster-name}${instance-name}", the "-" character is ignored.
Normally this isn't a problem, but Oracle database has a max length on table name of 30 characters!

Proposed naming convention:
 cluste name d(Development) / t(Testing) / a(Acceptance) / p(Production) - 3 letter abbreviation of the cluster purpose. e.g. p-hrm 
 instance name cluster name - i(Instance) - instance number e.g. p-hrm-i-1


Create a cluster

Run the following commands from the DAS as the webuser:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords create-cluster ${cluster-name}

Show all clusters on the DAS

Run the following commands from the DAS as the webuser:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords list-clusters

Add a node to the cluster

Run the following commands from the DAS as the webuser:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords create-instance --cluster ${cluster-name} --node ${node-name} ${instance-name}

Add the cluster to the loadbalancer

Run the following commands from the DAS as the webuser:
cd /opt/glassfish3/bin
./asadmin --user admin --passwordfile /opt/glassfish-passwords create-http-lb-ref --lbname lb-${httpd-hostname} ${cluster-name}


ċ
cluster-jsp.zip
(4k)
Ron Lievens,
Jan 21, 2014, 5:21 AM
ċ
das-initscript.txt
(1k)
Ron Lievens,
Jan 25, 2013, 1:08 AM
ċ
glassfish-installer.xml
(2k)
Ron Lievens,
Jan 26, 2013, 2:20 AM
ċ
http-initscript.txt
(2k)
Ron Lievens,
Jan 26, 2013, 2:14 AM
ċ
sync_nodes_app.sh
(2k)
Ron Lievens,
Jan 25, 2013, 1:37 AM