In this blog I’m going to show you how to safely rotate database passwords in WebLogic without breaking the entire system using the wonderful tool Hashicorp Vault. I have chosen to focus on rotating database passwords since these tend to be the most sensitive in terms of controlling access to your data and also the most commonly used set of credentials within WebLogic. The solutions described here can be used for other types of credentials as well, such as the NodeManager, Domain, or even a TDE Wallet password.
If you are not familiar with Hashicorp Vault it is a fantastic tool for managing all types of secrets: everything from usernames and passwords to SSH keys, AWS creds and PKI certificates. It can be used in automated workflows to help with bootstrapping new deployments to managing the full lifecycle of secrets.
The data stored within Vault is encrypted using a strong encryption key. The encryption key is protected with another key, called the master key. This master key is broken up in shards using Shamir’s secret sharing algorithm with each shard being distributed to trusted members within the organization.
A Solution for Password Management
The solution to making database password management more secure and successful within WebLogic is to start using dynamically generated short lived passwords rather than well-known, statically defined, long-lived credentials. This is where the powers of Vault shine.
I’ll show you how to configure Vault to generate a set of MySQL database credentials for your application that we will then use with configuring a WebLogic data source.
Everything described in this article can be scripted using Python and WLST to help you automate the secure deployment of WebLogic and password rotation.
Why use Short-lived Passwords
There are several benefits of using dynamic credentials that are short-lived from Vault:
- Short-lived passwords are more secure than long-lived passwords. The longer a password lives, the more time bad actors have to crack it.
- Dynamically generated passwords mean you get a new username and password every single time, eliminating default or well-known credentials.
- All dynamic credentials generated by Vault have a short specific time-to-live value associated with them, forcing you to rotate or renew those credentials often. This forces you into some good operational habits.
How to Minimize Downtime during Password Resets
Unfortunately for WebLogic, a database password reset requires a restart of the data source or WebLogic server that hosts the data source instance itself. In order to minimize the amount of downtime for your application, you need to consider a blue/green or rolling reset method for changing the password that only affects one WebLogic node in a cluster at a time.
Getting Started with Bootstrapping WebLogic
So here’s the step by step for preparing your environment to support password rotation with minimal downtime in Production.
- Define a password rotation period, such as 90 days, that will be used to trigger a rotation (this is usually company policy or compliance regulation, but could be implemented as part of a scheduler).
- Configure vault to generate database credentials with a TTL that is equal to or greater than your password rotation period.
- Deploy a highly available WebLogic domain using WebLogic clustering and a load balancer such as Apache, NGINX, or Oracle HTTP Server.
- Deploy data sources to the Cluster using credentials generated by Vault
The picture below describes our deployment architecture. We have a 2 node WebLogic cluster supported by a MySQL database. Our application will have its own schema in the MySQL database that we’ll call MY_APP. Weblogic will be configured with a MySQL data source that connects to this schema using a dynamically generated user from Vault.
Setting up the Environment
Here is what we’ll need to do.
- Install a MySQL database and create a system user for Vault
- Install and configure Hashicorp Vault to generate database credentials
- Deploy a two-node clustered WebLogic domain
- Machine-generate a WebLogic domain password and store in Vault
- Configure a WebLogic data source with credentials generated from Vault
Setting up the Database
For this article I’m going to be using docker to quickly spin up a MySQL database, however, you could use any WebLogic supported database. For most folks, this will be either MySQL or Oracle.
The following command will spin up a MySQL database using the latest official MySQL container image with a very secure password of welcome1.
|
1 2 3 |
docker run -p 3306:3306 --name mysql-test -e MYSQL_ROOT_PASSWORD=welcome1 -d mysql:latest |
The next step is to create a database schema for our application. Connect to your database and run the following MySQL statement to create a schema.
|
1 |
CREATE SCHEMA `MY_APP`; |
Since we will be using Hashicorp Vault to generate database users for us, it will need to connect to the database to do so. It would be a best practice to create a dedicated database user that Vault can use to connect. This dedicated database user must have the right privileges to create other users.
|
1 2 |
CREATE USER vaultuser IDENTIFIED BY 'welcome1'; GRANT SELECT, CREATE USER on *.* to 'vaultuser'@'localhost'; |
Installing Hashicorp Vault
Hashicorp Vault is a powerful secrets management tool that can be used, among many other things, to generate database credentials. Vault is delivered as a single executable that is for both the server and client.
First go ahead and download Vault for your operating system. Installation is a simple matter of unarchiving the downloaded file.
Starting Vault Server
The quickest way to get up and running with Vault is to start Vault server in development mode. This mode provides a default configuration for Vault. The most notable thing about dev mode is that all data is in memory and Vault is automatically unsealed. What this means is when you shutdown Vault server, you will lose all your data, however Vault does support several different storage backends to provide persistence.
Run the following command to start Vault server in development mode.
|
1 2 3 |
$ ./vault server -dev |
Vault server will output some important information. Make sure to jot down the root token and the URL for accessing Vault as shown below under the VAULT_ADDR environment variable.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
==> Vault server configuration: Cgo: disabled Cluster Address: https://127.0.0.1:8201 Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0 .1:8201", tls: "disabled") Log Level: info Mlock: supported: true, enabled: false Redirect Address: http://127.0.0.1:8200 Storage: inmem Version: Vault v0.9.0 Version Sha: bdac1854478538052ba5b7ec9a9ec688d35a3335 ==> WARNING: Dev mode is enabled! In this mode, Vault is completely in-memory and unsealed. Vault is configured to only have a single unseal key. The root token has already been authenticated with the CLI, so you can immediately begin using the Vault CLI. The only step you need to take is to set the following environment variables: export VAULT_ADDR='http://127.0.0.1:8200' The unseal key and root token are reproduced below in case you want to seal/unseal the Vault or play with authentication. Unseal Key: 0jnRFxo3I00pq5vuc+WDgW/B8mwkm815pF/O3DyG0r0= Root Token: bb94c913-752c-b5f5-9f7f-f6021e5fa279 |
Configuring Hashicorp Vault to Generate Database Credentials
Hashicorp Vault can generate credentials for many different backend systems, including databases, cloud platforms including AWS, TOTP and even function as a Certificate Authority. The list of supported backends keeps growing.
To generate credentials for our MySQL database, we need to set up the MySQL database backend within Hashicorp Vault. We will configure this database backend to generate database users with a certain set of GRANTS on our MY_APP schema.
Before we run the Vault command line tool (CLI), we need to set the VAULT_ADDR environment variable.
Open a separate terminal or command window and set VAULT_ADDR.
|
1 2 3 |
$ export VAULT_ADDR=http://localhost:8200 |
This variable tells the Vault CLI where to find the Vault server.
Next, mount the database backend in order to enable its functionality.
|
1 2 3 |
$ ./vault mount database |
Configure the database backend to use our MySQL database and the vaultuser user created above.
|
1 2 3 |
$ ./vault write database/config/my_app_db plugin_name=mysql-database-plugin connection_url="vaultuser:welcome1@tcp(localhost:3306)/" allowed_roles="my_app_users" |
There is quite a bit going on in the above command. Here’s the breakdown:
- database/config/my_app_db – This path identifies where the database backend configuration will live. The values after database/config are arbitrary.
- plugin_name=mysql-database-plugin – Identifies the type of database to configure.
- connection_url=”vaultuser:welcome1@tcp(localhost:3306)/” – Identifies the connection string Vault will use to connect to the database.
- allowed_roles=”my_app_users” – This parameter identifies what type of database user to create. We will create this role in just a moment.
Next you will create a database role in Vault that is associated with a SQL statement used to create database users. The SQL statement is responsible for creating database users and granting permissions on database objects and functions. You can see here we are granting CRUD operations on the MY_APP database schema only.
|
1 2 3 |
$ ./vault write database/roles/my_app_users db_name=my_app_db creation_statements="CREATE USER '{{name}}'@'%' IDENTIFIED BY '{{password}}';GRANT SELECT, INSERT, UPDATE, DELETE ON MY_APP.* TO '{{name}}'@'%';" default_ttl="1h" max_ttl="24h" |
Generate MySQL Database Credentials
OK, we’re almost there! We can finally generate a set of database credentials by reading from the database/creds/my_app_users path in Vault.
|
1 2 3 |
$ ./vault read database/creds/my_app_users |
If all goes well, Vault will respond with a username and password, along with two other values: lease_id and time-to-live. These last two values are used to manage the lifecycle of the database credentials. The database role we created earlier set a time-to-live value of 1 hour. This means the credentials are good for 1 hour. After 1 hour, the credentials will expire and Vault will remove them from the database. This is an important concept to understand with dynamically generated secrets.
You can verify this database user is actually real by connecting to your MySQL database or looking at the USERS table.
The database portion of this is all done. We have now configured Hashicorp Vault to dynamically generate a set of database users with certain privileges that can be used by our application.
Configuring a WebLogic Data Source with Generated Secrets
It is very common to write WLST scripts to configure WebLogic domains, including data sources. One challenge in writing WLST scripts is how to deal with passwords in a secure manner. We do not want passwords stored in plaintext, ever. We also want to avoid using well-known default passwords even for bootstrapping. The risk of passwords ending up in source code or leaking out on the Net is too great.
A better method is to fetch the passwords when you actually need them and only keep them in memory during the bootstrapping or configuration process.
Vault provides a first-class REST API that you can use to manage secrets. In fact, the CLI is simply a wrapper for the REST API.
The code snippet below shows how you can use the Python module hvac for interfacing with Hashicorp Vault to fetch a database credential. The module can be installed using pip.
|
1 2 3 4 5 6 7 8 9 10 11 12 |
import hvac # Set up the hvac client by specifying the Vault server endpoint and a Vault authentication token client = hvac.Client(url='http://localhost:8200', token=os.environ['VAULT_TOKEN']) # Fetch a database credential from our MY_APP Vault endpoint creds = client.read('database/creds/my_app_users') # hvac returns a dictionary of the results print creds['data']['username'] print creds['data']['password'] print creds['lease_id'] |
Notice that we have to provide an authentication token to Vault. When Vault server is started in dev mode, the root token is displayed in the output. You may use this root token, however, in a production environment you would never use the root token but generate tokens safe for use by scripts such as this.
Unfortunately, I have not had any luck getting the hvac module to load in a WLST script (which is Jython actually). You will have to take the response from Vault and pass it to a WLST script.
The snippet below is the python code I use to pass credentials to a WLST script.
|
1 2 3 4 5 6 |
print subprocess.check_output(['%s/oracle_common/common/bin/wlst.sh' % ORACLE_HOME, 'create_jdbc.py', db_creds['data']['username'], db_creds['data']['password'], 'weblogic', wl_creds['data']['password'] ]) |
Notice how I pass in both the database credentials, as well as the WebLogic admin username and password.
The WLST script being called will connect to a running WebLogic server using the WebLogic admin username and password, then configure a data source. The username and password are being set on lines 35 and 44.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
# create_jdbc.py WLST script # This script creates a data source in ONLINE mode using credentials # passed in from the command line. db_user = sys.argv[1] db_password = sys.argv[2] wl_user = sys.argv[3] wl_pw = sys.argv[4] # Connect to Weblogic connect(wl_user, wl_pw,'t3://adminserver:7001') # Start configuring domain edit() startEdit() # Create the data source cd('/') cmo.createJDBCSystemResource('my_app_db') cd('/JDBCSystemResources/my_app_db/JDBCResource/my_app_db') cmo.setName('my_app_db') cd('/JDBCSystemResources/my_app_db/JDBCResource/my_app_db/JDBCDataSourceParams/my_app_db') set('JNDINames',jarray.array([String('jndi/appdb')], String)) cd('/JDBCSystemResources/my_app_db/JDBCResource/my_app_db') cmo.setDatasourceType('GENERIC') cd('/JDBCSystemResources/my_app_db/JDBCResource/my_app_db/JDBCDriverParams/my_app_db') cmo.setUrl('jdbc:mysql://%s:%s/%s' % (DB_HOSTNAME, DB_PORT, DB_SCHEMA)) cmo.setDriverName('com.mysql.jdbc.jdbc2.optional.MysqlXADataSource') cmo.setPassword(db_pw) cd('/JDBCSystemResources/my_app_db/JDBCResource/my_app_db/JDBCConnectionPoolParams/my_app_db') cmo.setTestTableName('SQL SELECT 1\r\n') cd('/JDBCSystemResources/my_app_db/JDBCResource/my_app_db/JDBCDriverParams/my_app_db/Properties/my_app_db') cmo.createProperty('user') cd('/JDBCSystemResources/my_app_db/JDBCResource/my_app_db/JDBCDriverParams/my_app_db/Properties/my_app_db/Properties/user') cmo.setValue(db_user) cd('/JDBCSystemResources/my_app_db/JDBCResource/my_app_db/JDBCDataSourceParams/my_app_db') cmo.setGlobalTransactionsProtocol('TwoPhaseCommit') cd('/JDBCSystemResources/my_app_db') set('Targets',jarray.array([ObjectName('com.bea:Name=%s,Type=Cluster' % CLUSTER_TARGET)], ObjectName)) activate() |
With this approach, you can securely generate and set database passwords for WebLogic right at deployment time. No more static configuration files containing plaintext passwords or even encrypted files that require the protection of keys. Simply call the Vault API.
Other WebLogic Passwords
You may be wondering if you can use Vault to manage other passwords used within WebLogic. The answer is yes. Vault can be used to manage any password or even keys, whether they are statically defined like our WebLogic domain password or dynamic like the database credentials we created.
The WebLogic Domain Admin Password
Vault cannot directly generate a password for the admin user, however, it can store static data in key-value pairs. During the bootstrapping process when you are deploying a WebLogic domain, you can machine-generate the domain admin password and store this in Vault. Then when you have a script that needs to connect to WebLogic, you simply fetch the domain admin password from Vault when you need it. No need to store these credentials in a file on the file system.
Here is just one example of using openssl to generate a random string.
|
1 2 |
$ openssl rand -base64 14 TOPGp48kHp0RRba+gY8= |
Store this string in Hashicorp Vault so that it can be fetched later on for other scripts or actions that require the domain admin user.
Here’s how you could store this value in Vault using the hvac python module. Of course, don’t actually hardcode the secret in the script. Use a variable here.
|
1 |
client.write('secret/my_weblogic_domain/weblogic', secret='TOPGp48kHp0RRba+gY8=') |
Then when you want to fetch the password for the domain admin …
|
1 2 |
creds = client.read('secret/my_weblogic_domain/weblogic') print creds['data']['secret'] |
Minimizing Downtime during Password Rotation
The key to rotating WebLogic database passwords without breaking your application is two-fold: rolling updates and overlapping database credentials.
During a rolling update, the password will be reset and each node in a clustered WebLogic domain will be restarted one by one to allow for the changes to take effect. Restarting of the WebLogic server is required in order to flush all the old database connections and re-establish new ones using the new database credential. Having a multi-node WebLogic cluster allows for a highly available application that can survive a rolling update.
Overlapping database credentials basically means that two sets of database credentials are valid for a period of time. This overlap is what makes the rolling update or reset possible. While some nodes are being reset with the new password, your application running on the other nodes with the original or old password continue to operate.
The amount of overlap depends on the Vault TTL for the database credential and when you choose to rotate that password within the TTL window. For example, the database credentials we generated earlier had a 1 hour TTL. If we generate a new credential 30 minutes before that TTL expires, then we have an overlapping window of 30 minutes where both credentials are valid.
This is where the power of Vault really shines. The database backend we set up early in Vault generates credentials with a 1 hour TTL. For production WebLogic deployments, I would recommend changing this to 60 to 90 days, however your company policy may dictate differently. When it comes time (within the TTL) to rotate a database password, you simply ask Vault for another database credential like we did earlier in the Bootstrap portion of this article then update the data source with the new password and one by one, restart each of the weblogic servers in the cluster.
One Pattern for Rotating WebLogic Database Passwords
So here is one process to support minimal downtime WebLogic database password resets .
- Before the original database credential TTL expires, fetch a new database credential from Vault.
- Update the data source in WebLogic using the new database credential and activate the changes.
- Gracefully restart each WebLogic server one-by-one for the changes to take effect. If your WebLogiccluster is fronted by a load balancer, you will have to tell the load balancer that you’ve taken a member out of the load balancing pool.
Do you want to learn more about Hashicorp Vault?
This article briefly touched upon the power of Vault to manage database secrets. Our new online course does a deep-dive on deploying and using Vault in a production environment and includes hours of video lecture and hands-on tutorials on how to effectively manage all types of secrets, how to authorize access, and how to tie Vault into enterprise and cloud environments.
