Decoding IBM Spectrum Scale Object Authentication configuration with Active Directory(AD)

            One can use Installer toolkit during deploy to configure Spectrum Scale object with Active Directory or use mmuserauth CLI to configure Spectrum Scale object with Active Directory after enabling Spectrum Scale object

In this blog entry I am making use of mmuserauth CLI to configure Spectrum Scale object with Active Directory

Example of mmuserauth CLI for configuring Spectrum Scale object with Active Directory.

mmuserauth service create

–data-access-method object
–type ad
–servers 10.0.0.27
–user-dn dc=sonas,dc=com
–base-dn dc=sonas,dc=com
–user-name administrator@sonas.com
–password Passw0rd
–ks-admin-user keystoneadminuser
–ks-swift-user swiftserviceuser
–ks-swift-pwd swiftPassw0rd
[–user-objectclass organizationalPerson]
[–user_name_attrib sAMAccountName]
[–user-id-attrib CN ]
Note : [option] denotes the optional parameter with default values

There are more option to this cli, Above listed are important.
Refer mmuserauth man page for details

Things to check before configuring the Spectrum Scale object with Active Directory

  1. Is Active Directory Server is correct, Reachable and Bind details are working ?
It simply means validating following parameter from mmuserauth example given at the start
–servers 10.0.0.27
–user-dn cn=users,dc=sonas,dc=com
–user-name administrator@sonas.com
–password Passw0rd
  Ldapsearch sample command to validate AD details 

/usr/bin/ldapsearch -x -H ldap://<–server> -b <–user-dn> -s sub -o nettimeout=10 -LLL -A -D < –username > -w <–password>

The command for my environment
/usr/bin/ldapsearch -x -H ldap://10.0.0.27 -b cn=users,dc=sonas,dc=com -s sub -o nettimeout=10 -LLL -A -D administrator@sonas.com -w Passw0rd

Above command print all the user details on command line so you can use following to limit the details on screen

/usr/bin/ldapsearch -x -H ldap://10.0.0.27 -b cn=users,dc=sonas,dc=com -s sub -o nettimeout=10 -LLL -A -D administrator@sonas.com -w Passw0rd dn

OR

/usr/bin/ldapsearch -x -H ldap://10.0.0.27 -b cn=users,dc=sonas,dc=com -s sub -o nettimeout=10 -LLL -A -D administrator@sonas.com -w Passw0rd cn=administrator

Note : Provide Bind user ( ie –user-name ) in
DN format (example : CN=Administrator,CN=Users,DC=SONAS,DC=COM)
or
user logon name format (example : administrator@sonas.com)

If this step is successful then your Active Directory is correct, reachable and provided bind details are Valid.

  1. Is –ks-admin-user keystoneadminuser is present in Active Directory ?

Note : Keystoneadminuser should be present in Active Directory under provide –user-dn and It should have Valid Password

To validate –ks-admin-user following options need careful attention
[–user-objectclass organizationalPerson]
[–user_name_attrib sAMAccountName]
[–user-id-attrib CN ]

Ldapsearch sample command to validate the –ks-admin-user
/usr/bin/ldapsearch -x -H ldap://10.0.0.27 -b cn=users,dc=sonas,dc=com -s sub -o nettimeout=10 -LLL -A -D administrator@sonas.com -w Passw0rd ‘(&(objectClass=organizationalPerson)(|(sAMAccountName=keystoneadminuser)(CN=keystoneadminuser)))’ -z1 dn

This command should return something like
dn: CN=keystoneadminuser,CN=Users,DC=SONAS,DC=COM

This steps confirm that you –ks-admin-user is good to go with.

Note : Use shortname for –ks-admin-user. Do not user DN or user logon name
For example I am using keystoneadminuser not the
CN=keystoneadminuser,CN=Users,DC=SONAS,DC=COM
or
keystoneadminuser@sonas.com

  1. is–ks-swift-user and –ks-swift-pwd is correct ?

Validating –ks-swift-user exist in Active Directory
–ks-swift-user swiftserviceuser
–ks-swift-pwd Passw0rd

Ldapsearch sample command to check if –ks-swift-user exist using admin credentials
/usr/bin/ldapsearch -x -H ldap://10.0.0.27 -b cn=users,dc=sonas,dc=com -s sub -o nettimeout=10 -LLL -A -D administrator@sonas.com -w Passw0rd ‘(&(objectClass=organizationalPerson)(|(sAMAccountName=swiftserviceuser)(CN=swiftserviceuser)))’ dn

This command will return something like dn: CN=swiftserviceuser,CN=Users,DC=SONAS,DC=COM use this DN to validate the provided –ks-swift-pwd

Ldapsearch sample command to check if –ks-swift-pwd exist is correct

/usr/bin/ldapsearch -x -H ldap://10.0.0.27 -b cn=users,dc=sonas,dc=com -s sub -o nettimeout=10 -LLL -A -D CN=swiftserviceuser,CN=Users,DC=SONAS,DC=COM -w swiftPassw0rd ‘(objectClass=*)’ -z1 dn

If this is successful then you are ready to configure Spectrum Scale with Active Directory

Note : Use shortname for –ks-swift-user. Do not user DN or user logon name
For example I am using swiftserviceuser not the
CN=swiftserviceuser,CN=Users,DC=SONAS,DC=COM or
swiftserviceuser@sonas.com

  1. Importance of –user-objectclass, –user-name-attrib and –user-id-attrib ?
All these are used as initial filter for Keystone users. Only user who has these attribute will be listed in Keystone users.
Default for Active Directory, You can change it as per your environment.
[–user-objectclass organizationalPerson]
[–user-name-attrib sAMAccountName]
[–user-id-attrib CN ]
Note : Choosing –user-id-attrib, It should be the attribute used in user DN. For example in all DN of users (CN=swiftserviceuser,CN=Users,DC=SONAS,DC=COM) in my environment it is CN. Hence I am using –user-id-attrib CN

“These are my personal views and do not necessarily reflect that of my employer”

Advertisements

Configuring IBM Spectrum Scale Object with SSL enabled External Keystone server

Prerequisite :  

  • IBM Spectrum Scale 4.2.1.x
  • SSL enabled Keystone server.
  • A swift user with valid password having ‘admin‘ role in ‘service‘ tenant of ‘default‘ domain in a external Keystone server

Endpoints on External Keystone server will look like

[root@externalKeystone ~]# openstack endpoint list
+———————————-+———–+————–+————–+———+———–+—————————————–+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+———————————-+———–+————–+————–+———+———–+—————————————–+
| 78cd570651f34b848890687c4f1578a9 | None | keystone | identity | True | public | https://cesip:5000/ |
| ff82aa833eec42b7a345a1c029b74959 | None | keystone | identity | True | internal | https://cesip:35357/ |
| 692c4bb6c4a14ece9f810ba4fc1944f1 | None | keystone | identity | True | admin | https://cesip:35357/ |
| 52b559474c03454eaef67af14a3c4afe | RegionOne | swift | object-store | True | public | http://c1ces:8080/v1/AUTH_%(tenant_id)s |
| ed14c1c50f3242a0aed61b9ccdfc8c4d | RegionOne | swift | object-store | True | internal | http://c1ces:8080/v1/AUTH_%(tenant_id)s |
| f26a242b71b34128abf87cbc1e8937aa | RegionOne | swift | object-store | True | admin | http://c1ces:8080 |
+———————————-+———–+————–+————–+———+———–+—————————————–+

 

On IBM Spectrum scale, Object will be already configured.
Remove the Current Object Authentication and Id-mapping using following command.

[root@c1n3 ~]# mmuserauth service remove –data-access-method object 
mmuserauth service remove: Command successfully completed

[root@c1n3 ~]# mmuserauth service remove –data-access-method object –idmapdelete
mmuserauth service remove: Command successfully completed

Copy CA-Certificates used on External Keystone server on Protocol node of IBM Spectrum Scale at following location.

[root@c1n3 ~]# ls /var/mmfs/tmp/ks_ext_cacert.pem
/var/mmfs/tmp/ks_ext_cacert.pem

Run mmuserauth command on protocol node where CA-Certificate is copied

[root@c1n3 ~]# mmuserauth service create –data-access-method object –type userdefined –ks-swift-user swift –ks-swift-pwd passw0rd –ks-ext-endpoint https://cesip:5000/v3 –enable-ks-ssl
mmcesobjcrbase: Validating execution environment.
mmcesobjcrbase: Validating Keystone environment.
mmcesobjcrbase: Configuration complete.
Object authentication configuration completed successfully.

Note : External keystone server must be reachable from protocol nodes using the hostname used for creating SSL Certificates for external Keystone. Hostname in Keystone Endpoints should also match the hostname used for creating SSL certificates

IBM Spectrum scale Object is successfully configured with SSL enabled external Keystone server.

~/openrc file on protocol node will be updated with details of External Keystone.

[root@c1n3 ~]# cat openrc
export OS_AUTH_URL=”https://cesip:5000/v3
export OS_CACERT=”/etc/swift/ks_ext_cacert.pem”
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3
export OS_USERNAME=”admin”
export OS_PASSWORD=””
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_PROJECT_DOMAIN_NAME=Default

Update the ~/openrc with valid username and password from external Keystone server.

[root@c1n3 ~]# source ~/openrc

Verify IBM Spectrum Scale Object is working with external Keystone server.

[root@c1n3 ~]# swift stat
Account: AUTH_0557d5eb51294e48b1c5041c684b4f66
Containers: 0
Objects: 0
Bytes: 0
Content-Type: text/plain; charset=utf-8
Keep-Alive: timeout=5, max=100

“These are my personal views and do not necessarily reflect that of my employer”

 

IBM Spectrum Scale : Object Authentication

Every Object access request on IBM Spectrum Scale is authenticated before serving the data. IBM Spectrum Scale object store relies on keystone for validation of user before processing the request for object access. Keystone is the identity service used by various Openstack services for authentication. Basic Spectrum Scale Object access flow is depicted in following figure.

ObjectAccessFlow

Figure 1 : Basic Object access request flow

     IBM Spectrum Scale supports configuring Keystone with various different identity backends like Microsoft Active Directory, LDAP, Postgres. Identity backend is the source of username and password. Before sending request to Spectrum Scale object store, the user/client sends a request to Keystone for obtaining a Token required for accessing the object service. The request to Keystone contains the Username and Password. Keystone validates the username and password with the configured Identity backed. On successful validation of username and password keystone returns a token to the user/client which then further uses it for object (swift) request. The Token gets expire after the predefined time. The token expiration period is configurable. The specific Token can be revoked from Keystone by sending Token DELETE request. User request with revoked Token will be rejected by Spectrum Scale object store.

SpectumScaleObjectAuthentication

Figure 2 :  High level Object authentication configuration and Security aspects addressed by IBM Spectrum Scale

The above diagram depicts the various ways to configure the Object Authentication and the various security aspect involved.

Object Authentication can be configured with one of the following type:

  1. Local — User identities are stored locally in Postgres.
  2. AD – Keystone uses Microsoft Active Directory as Identity backend.
  3. LDAP – Keystone uses LDAP as Identity backend.
  4. Userdefined – Keystone will be hosted outside of Spectrum Scale and Configuration of identity server is left with User.

In first three types of object authentication configuration, Keystone will be running on all protocol nodes for High Availability.

  • The Keystone can be configured with https(ssl) for better security. The communication between object/keystone client and Keystone server will be over https(ssl) in this case.
  • IBM Spectrum Scale provide the option to configure the communication between Keystone server and Identity server(Active Directory and LDAP) over TLS.
  • For Advance user who want to make use of different features of keystone which are not provided by the keystone hosted on Spectrum Scale can configure the Spectrum Scale object with external keystone using type ‘userdefined’. The communication from swift to external keystone server can be over https(ssl)
  • When configuring the Spectrum Scale object protocol, one can use either the install toolkit or the mmuserauth command. The mmuserauth command is for managing the object authentication. Same command is used for File authentication. mmuserauth command has various option like create, list, remove and check. For more details of mmuserauth command please refer manpage of mmuserauth.

    “These are my personal views and do not necessarily reflect that of my employer”

Keystone with Active Directory and Multiple OU lookup

Generally the Active Directory deployments are very large and properly organised under multiple Organizational Units(OU).  OUs can be based on Department, Functioning groups etc.

Following diagram depict the sample AD environment with multiple OU.

Screenshot from 2015-11-27 22:06:42

One can configure the AD with Keystone using ldap identity provider so that users from AD are visible on Keystone.

In current keystone ldap provider there is no mechanism of providing two or more OUs. For Example today if one want to use only user from OU=Comp and OU=Admin for keystone he has to provide user OU as dc=myuniv,dc=com ie root of AD. Because of this all users from all OUs will be visible to keystone. Currently providing multiple OUs in keystone configuration in not in plan.

However one can limit the number of users visible to keystone using following two mechanisms from Active Directory.

  1. Update attribute of all users those users should be visible to keystone and update the
    For example I chose ‘description’ attribute of user on Active Directory and updated the this attribute of all user those should be visible to keystone with description=OBJECT_USERRun following on spectrum scale cluster to update the filter
    mmobj config change –ccrfile keystone.conf –section ldap –property user_filter –value ‘(description=OBJECT_USER)’
  2. Another approach apply ACL on bind user such that only required OU’s will be visible to keystone.In this deny full acl need to be added for bind user on all OU except the OU’s from which users will be visible to keystone.

    For example in my setup I dont want user from OU=org1. testuser1 is binduser hence i added  deny acl for testuser1 on OU=org1.
    Deny
    Now keystone will list all users expect users from OU=org1.

    “These are my personal views and do not necessarily reflect that of my employer”

Configuring IBM Spectrum Scale Object With External Keystone

IBM Spectrum Scale provide an option to configure Object with External Keystone.

There are two method to achieve the same First is using Installer to Configure the Object with External Keystone and Other is to use mmuserauth cli for setting up Object Authentication. 

Prerequisite :
    Following entities must be present on keystone server which is hosted outside of Spectrum Scale cluster. 

  • User ‘swift’ and ‘admin’ with valid password
  • Project ‘service’ and ‘admin’
  • Role ‘admin’
  • ‘swift’ user should have ‘admin’ role in ‘service’ tenant
  • ‘admin’ user should have ‘admin’ role in ‘admin’ tenant
  • keystone service of type ‘identity’
  • Keystone endpoint
  • Swift service of type ‘object-store’
  • Swift endpoint.

Refer Configure Openstack-Keystone for IBM Spectrum Scale Object Storage for configuring external keystone server which fulfill all above requirement.

On Spectrum Scale cluster. 

# Remove the existing authentication if any using following command

[root@swiftnode ~]# mmuserauth service remove --data-access-method object
[root@swiftnode ~]# mmuserauth service remove --data-access-method object --idmapdelete

# Configure IBM Spectrum Scale Object with external keystone using following command

[root@swiftnode ~]# mmuserauth service create --data-access-method object --type userdefined --ks-ext-endpoint http://mykeystone:35357/v3 --ks-swift-user swift --ks-swift-pwd password

Spectrum Scale Object is configured with External Keystone and Waiting for you to upload lots of Object. Hurry up … 🙂

Disclaimer: The content of this post is not approved nor endorsed by IBM.

Configure Openstack-Keystone for IBM Spectrum Scale Object Storage

Steps for configuring External keystone server for IBM Spectrum Scale Object Storage

Prerequisite :
               RHEL 7 or 7.1 host with Enabled Redhat and Openstack Kilo repository

# Install openstack-keystone rpm and other required rpms from repository

 $ yum install openstack-keystone openstack-utils openldap-clients python-openstackclient -y 

# Add required firewall rule on node or stop the firewalld

$ service firewalld stop

# Update the keystone.conf. Update admin_token for administration/configuration

$ openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token ADMIN 

# Update the database connection. Assumption: Mysql(MariaDB) will be used and same node is used as database node.

$ openstack-config --set /etc/keystone/keystone.conf database connection 'mysql://keystone:Passw0rd@localhost/keystone'

# In this setup PKI is used for token. One can choose to use UUID for token, In that case skip following steps.

$ openstack-config --set /etc/keystone/keystone.conf token provider 'keystone.token.providers.pki.Provider'
$ keystone-manage pki_setup --keystone-user keystone --keystone-group keystone

# Install mariadb and initialize the keystone database

$ /usr/bin/openstack-db --service keystone --init --password password --rootpw password

$ service openstack-keystone start 

# At this stage Openstack-keystone service will be running

$ export OS_TOKEN="ADMIN"
$ export OS_IDENTITY_API_VERSION=3
$ export OS_URL=http://localhost:35357/v3 

#  Create required User,Project,Role entries in Keystone

$ openstack project create --domain default service
$ openstack project create --domain default admin

$ openstack user create --password password admin --domain default
$ openstack user create --password password swift --domain default

$ openstack role create admin

$ openstack role add --user admin --domain default admin
$ openstack role add --user admin --project admin admin
$ openstack role add --user swift --domain default admin
$ openstack role add --user swift --project service admin

# Create Keystone endpoints

$ openstack service create --name keystone identity
$ keystoneservice=`openstack service show keystone -f value -c id`
$ keystoneendpoint='mykeystone' #---> Change this as per hostname/dnsname of keystone
$ openstack endpoint create $keystoneservice public http://${keystoneendpoint}:5000/v3
$ openstack endpoint create $keystoneservice internal http://${keystoneendpoint}:5000/v3
$ openstack endpoint create $keystoneservice admin http://${keystoneendpoint}:35357/v3 

# Create Swift endpoints

$ openstack service create --name swift object-store
$ swiftservice=`openstack service show swift -f value -c id`
$ swiftendpoint='swiftnode'  #---> Change this as per hostname/dnsname of swift
$ openstack endpoint create $swiftservice public  "http://${swiftendpoint}:8080/v1/AUTH_%(tenant_id)s"
$ openstack endpoint create $swiftservice internal  "http://${swiftendpoint}:8080/v1/AUTH_%(tenant_id)s"
$ openstack endpoint create $swiftservice admin  "http://${swiftendpoint}:8080"

# Remove admin_token from keystone.conf

$ openstack-config --del /etc/keystone/keystone.conf DEFAULT admin_token

# Restart the openstack-keystone service to pickup the admin_token deletion.

$ service openstack-keystone restart


Disclaimer: The content of this post is not approved nor endorsed by IBM.

Now the keystone server is ready to be configured with IBM Spectrum Scale Object Store in USERDEFINED configuration.

Changing Token expiration in Openstack-Keystone

Sometime there is requirement of changing the Token expiration time in Openstack-keystone so that Token remain valid for the longer/shorter time based on requirement.

[root@keystone ~]# openstack-config –set /etc/keystone/keystone.conf token expiration 86400

[root@keystone ~]# service openstack-keystone restart
Redirecting to /bin/systemctl restart openstack-keystone.service

One can confirm the expiration time is actually changed by getting new token

[root@keystone ~]# date
Tue Oct 6 10:01:37 EDT 2015          #–> Current time 

[root@keystone ~]# source openrc
[root@keystone ~]# openstack token issue -c expires
+———+—————————–+
| Field | Value |
+———+—————————–+
| expires | 2015-10-07T14:01:39.753765Z |         #—-> Token Expiration time 24Hrs from current time
+———+—————————–+