IBM Spectrum Scale : Object Authentication

Every Object access request on IBM Spectrum Scale is authenticated before serving the data. IBM Spectrum Scale object store relies on keystone for validation of user before processing the request for object access. Keystone is the identity service used by various Openstack services for authentication. Basic Spectrum Scale Object access flow is depicted in following figure.

ObjectAccessFlow

Figure 1 : Basic Object access request flow

     IBM Spectrum Scale supports configuring Keystone with various different identity backends like Microsoft Active Directory, LDAP, Postgres. Identity backend is the source of username and password. Before sending request to Spectrum Scale object store, the user/client sends a request to Keystone for obtaining a Token required for accessing the object service. The request to Keystone contains the Username and Password. Keystone validates the username and password with the configured Identity backed. On successful validation of username and password keystone returns a token to the user/client which then further uses it for object (swift) request. The Token gets expire after the predefined time. The token expiration period is configurable. The specific Token can be revoked from Keystone by sending Token DELETE request. User request with revoked Token will be rejected by Spectrum Scale object store.

SpectumScaleObjectAuthentication

Figure 2 :  High level Object authentication configuration and Security aspects addressed by IBM Spectrum Scale

The above diagram depicts the various ways to configure the Object Authentication and the various security aspect involved.

Object Authentication can be configured with one of the following type:

  1. Local — User identities are stored locally in Postgres.
  2. AD – Keystone uses Microsoft Active Directory as Identity backend.
  3. LDAP – Keystone uses LDAP as Identity backend.
  4. Userdefined – Keystone will be hosted outside of Spectrum Scale and Configuration of identity server is left with User.

In first three types of object authentication configuration, Keystone will be running on all protocol nodes for High Availability.

  • The Keystone can be configured with https(ssl) for better security. The communication between object/keystone client and Keystone server will be over https(ssl) in this case.
  • IBM Spectrum Scale provide the option to configure the communication between Keystone server and Identity server(Active Directory and LDAP) over TLS.
  • For Advance user who want to make use of different features of keystone which are not provided by the keystone hosted on Spectrum Scale can configure the Spectrum Scale object with external keystone using type ‘userdefined’. The communication from swift to external keystone server can be over https(ssl)
  • When configuring the Spectrum Scale object protocol, one can use either the install toolkit or the mmuserauth command. The mmuserauth command is for managing the object authentication. Same command is used for File authentication. mmuserauth command has various option like create, list, remove and check. For more details of mmuserauth command please refer manpage of mmuserauth.

    “These are my personal views and do not necessarily reflect that of my employer”

Advertisements

Running Openstack-Keystone under Apache

  1. Install Keystone and Apache(HTTPD) Packages
    [root@deepaknode1 ~]# yum install openstack-keystone openstack-utils openldap-clients python-openstackclient httpd mod_wsgi -y
  2. Copy keystone.py to httpd configuration
    [root@deepaknode1 ~]# mkdir /var/www/cgi-bin/keystone

    [root@deepaknode1 ~]# cp /usr/lib/python2.7/site-packages/keystone/httpd/keystone.py /var/www/cgi-bin/keystone/admin

    [root@deepaknode1 ~]# cp /usr/lib/python2.7/site-packages/keystone/httpd/keystone.py /var/www/cgi-bin/keystone/main

  3. Create /etc/httpd/conf.d/wsgi-keystone.conf with following content
    [root@deepaknode1 ~]# cat /etc/httpd/conf.d/wsgi-keystone.conf
    WSGISocketPrefix /var/run/wsgi
    SetEnv APACHE_RUN_USER keystone
    SetEnv APACHE_RUN_GROUP keystone
    Listen 5000
    <VirtualHost *:5000>
    WSGIDaemonProcess keystone-public user=keystone group=keystone display-name=keystone-public
    WSGIProcessGroup keystone-public
    WSGIImportScript /var/www/cgi-bin/keystone/main process-group=keystone-public application-group=%{GLOBAL}
    WSGIScriptAlias / /var/www/cgi-bin/keystone/main
    ErrorLog /var/log/keystone/httpd-error.log
    LogLevel info
    CustomLog /var/log/keystone/httpd-access.log combined
    LimitRequestBody 16384
    </VirtualHost>

    Listen 35357
    <VirtualHost *:35357>
    WSGIDaemonProcess keystone-admin user=keystone group=keystone display-name=keystone-admin
    WSGIProcessGroup keystone-admin
    WSGIImportScript /var/www/cgi-bin/keystone/main process-group=keystone-admin application-group=%{GLOBAL}
    WSGIScriptAlias / /var/www/cgi-bin/keystone/admin
    ErrorLog /var/log/keystone/httpd-error.log
    LogLevel info
    CustomLog /var/log/keystone/httpd-access.log combined
    LimitRequestBody 16384
    </VirtualHost>

  4. Start httpd service. Make sure openstack-keystone service is not running.
    [root@deepaknode1 ~]# service httpd start
    [root@deepaknode3 ~]# service httpd status
    Redirecting to /bin/systemctl status httpd.service
    …..
    Main PID: 16430 (httpd)
    CGroup: /system.slice/httpd.service
    ├─14749 keystone-public -DFOREGROUND
    ├─14750 keystone-admin -DFOREGROUND
    ├─14755 /usr/sbin/httpd -DFOREGROUND
    ├─14756 /usr/sbin/httpd -DFOREGROUND
    ├─14757 /usr/sbin/httpd -DFOREGROUND
  5. Continue with remaining Openstack-Keystone configuration like database setup, User/Project/Role/Service/Endpoint creation.
    Refer https://deepakrghuge.wordpress.com/2015/10/06/configure-openstack-keystone-for-ibm-spectrum-scale-object-storage/ for remaining setup. Just make sure that use httpd.service to start/stop keystone service instead of openstack-keystone.service

“These are my personal views and do not necessarily reflect that of my employer”

Setting up Openstack Keystone with Kerberos..

Prerequisite :
– Kerberos server(MIT Kerberos)
– Keystone server with Apache (Redhat 7.*)

1. On Kerberos Server

Add/Create new principle for Keystone service with Kerberos server Hostname.

[root@krbserver ~]# kadmin.local 
Authenticating as principal root/admin@PUNE-KDC.COM with password.
kadmin.local: addprinc -randkey HTTP/spectrumscale1
kadmin.local: ktadd -k /tmp/http.keytab HTTP/spectrumscale1

Copy /tmp/http.keytab on Keystone server :/etc/httpd/conf/httpd.keytab

2. On Keystone Server

  1. Install Apache kerberos module
    [root@KeystoneServer ~]# yum install mod_auth_kerb -y

  2. Update wsgi-keystone.conf with Kerberos configuration
    [root@KeystoneServer ~]# cat /etc/httpd/conf.d/wsgi-keystone.conf
    LoadModule auth_kerb_module modules/mod_auth_kerb.so
    WSGISocketPrefix /var/run/wsgi
    SetEnv APACHE_RUN_USER keystone
    SetEnv APACHE_RUN_GROUP keystone
    <VirtualHost *:5000>
    WSGIDaemonProcess keystone-public user=keystone group=keystone display-name=keystone-public
    WSGIProcessGroup keystone-public
    WSGIImportScript /var/www/cgi-bin/keystone/main process-group=keystone-public application-group=%{GLOBAL}
    WSGIScriptAlias /krb “/var/www/cgi-bin/keystone/main”
    WSGIScriptAlias / “/var/www/cgi-bin/keystone/main”
    <Location “/krb/v3/auth/tokens”>
    LogLevel debug
    AuthType Kerberos
    AuthName “Kerberos Login”
    KrbMethodNegotiate on
    KrbMethodK5Passwd off
    KrbServiceName HTTP/spectrumscale1
    KrbAuthRealms PUNE-KDC.COM
    Krb5KeyTab /etc/httpd/conf/httpd.keytab
    KrbLocalUserMapping on
    require valid-user
    </Location>
    ErrorLog /var/log/keystone/httpd-error.log
    LogLevel info
    CustomLog /var/log/keystone/httpd-access.log combined
    LimitRequestBody 16384
    </VirtualHost>

    Listen 35357

    <VirtualHost *:35357>
    WSGIDaemonProcess keystone-admin user=keystone group=keystone display-name=keystone-admin
    WSGIProcessGroup keystone-admin
    WSGIImportScript /var/www/cgi-bin/keystone/main process-group=keystone-admin application-group=%{GLOBAL}
    WSGIScriptAlias /krb “/var/www/cgi-bin/keystone/admin”
    WSGIScriptAlias / “/var/www/cgi-bin/keystone/admin”
    <Location “/krb/v3/auth/tokens”>
    LogLevel debug
    AuthType Kerberos
    AuthName “Kerberos Login”
    KrbMethodNegotiate on
    KrbMethodK5Passwd off
    KrbServiceName HTTP/spectrumscale1
    KrbAuthRealms PUNE-KDC.COM
    Krb5KeyTab /etc/httpd/conf/httpd.keytab
    KrbLocalUserMapping on
    require valid-user
    </Location>
    ErrorLog /var/log/keystone/httpd-error.log
    LogLevel info
    CustomLog /var/log/keystone/httpd-access.log combined
    LimitRequestBody 16384
    </VirtualHost>

  3. Install kerberos client package
    yum install krb5-workstation -y

  4. Configure the Kerberos client(as per your environment)
    [root@KeystoneServer ~]# cat /etc/krb5.conf
    [logging]

    [libdefaults]

    default_realm = PUNE-KDC.COM
    [realms]
    PUNE-KDC.COM = {
    kdc = krbserver.in.ibm.com:88
    admin_server = krbserver.in.ibm.com:749

    }
    [domain_realm]
    in.ibm.com = PUNE-KDC.COM
    .in.ibm.com = PUNE-KDC.COM

  5. Get kerberos token for user
    [root@KeystoneServer ~]# kinit userrw
    Password for userrw@PUNE-KDC.COM:
    [root@KeystoneServer ~]# klist
    Ticket cache: KEYRING:persistent:0:0
    Default principal: userrw@PUNE-KDC.COM
    Valid starting Expires Service principal
    2016-02-26T12:17:28 2016-02-27T11:45:15 krbtgt/PUNE-KDC.COM@PUNE-KDC.COM
    renew until 2016-02-26T12:17:28

  6. Curl request for keystone token without username and password
    [root@KeystoneServer ~]# curl -i -H “Content-Type:application/json” –negotiate -u : -d ‘{ “auth”: { “identity”: { “methods”: []}, “scope”: { “project”: { “domain”: { “name”: “Default” }, “name”: “admin” } } } }’ -X POST http://SpectrumScale1:5000/krb/v3/auth/tokens
    HTTP/1.1 401 Unauthorized
    Date: Tue, 01 Mar 2016 07:35:52 GMT
    Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
    WWW-Authenticate: Negotiate
    Content-Length: 381
    Content-Type: text/html; charset=iso-8859-1
    HTTP/1.1 201 Created
    Date: Tue, 01 Mar 2016 07:35:52 GMT
    Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_auth_kerb/5.4 PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
    X-Subject-Token: MIIHuQYJKoZIhvcNAQcCoIIHVqXmOQZM5KBfGnHhW3FMJW7nXDAELZ0X2s2WO9e6w==
    Vary: X-Auth-Token
    x-openstack-request-id: req-d6738e8b-b9b6-42e7-a615-ae03c8eb563e
    WWW-Authenticate: Negotiate YIGZBgkqhkiG9xIBAgICAG+BiTCBhqADAgEFoQMCAQ+iejB4oAMCARKicQRvwFfLUoOfh04
    …………
    …………
    3Tg5Ts8goQatVJ5JEnCYqkXIo8Yk5vYB7BWto2FRhDKzcAp75Qqciv6DgT8gnc6
    Content-Length: 1634
    Content-Type: application/json
    {“token”: {“methods”: [], “roles”: [{“id”: “e23776346ab747f4bddf7f056b8d62c9”, “name”: “admin”}], “expires_at”: “2016-03-31T07:35:52.528404Z”, “project”: {“domain”: {“id”: “default”, “name”: “Default”}, “id”: “bb86db4204e84b4b857241a0b23a62fb”, “name”: “admin”}, “catalog”: [{“endpoints”: [{“region_id”: null, “url”: “http://SpectrumScale1:35357/&#8221;, “region”: null, “interface”: “admin”, “id”: “1bd1b57b2cb14e74b5864261588a1e57”}, {“region_id”: null, “url”: “http://SpectrumScale1:35357/&#8221;, “region”: null, “interface”: “internal”, “id”: “43472eb0474d469c90d85a4dc7c6058d”}, {“region_id”: null, “url”: “http://SpectrumScale1:5000/&#8221;, “region”: null, “interface”: “public”, “id”: “6a9c87ec04f3480eb254568b60b8db07”}], “type”: “identity”, “id”: “7a22d7f78b224843811ef0708431eb46”, “name”: “keystone”} “extras”: {}, “user”: {“domain”: {“id”: “default”, “name”: “Default”}, “id”: “userrw”, “name”: “userrw”}, “audit_ids”: [“rcLqPQp5T6y9hKu12Tle2g”], “issued_at”: “2016-03-01T07:35:52.528436Z”}}

    “These are my personal views and do not necessarily reflect that of my employer

Openstack Swift with SSL(https)

Following are the steps to enable ssl for Openstack swift Proxy for secure data transfer between Openstack Swift Proxy server and the swift client.

Prerequisite :
– Set of ssl certificate (CA signed or Locally generated)
– Up and Running Openstack Keystone and Swift.

WARNING: SSL should only be enabled for testing purposes. Use external SSL termination for a production deployment. 

1. Copy the the ssl certificates under /etc/swift directory on all protocol nodes. Make sure CN in certificate is matching the swift endpoint hostname. In our case it is Node3

2. Swift user must have read permission on certificate files on all protocol nodes
[root@Node3]# ls -al /etc/swift/ssl_*
-rw——-. 1 swift swift 2864 Dec 8 23:56 /etc/swift/ssl_cert.pem
-rw——-. 1 swift swift 887 Dec 8 23:56 /etc/swift/ssl_key.pem

3 Update ssl certificate details in proxy-server.conf
[root@Node3]#openstack-config –set /etc/swift/proxy-server.conf  DEFAULT key_file ‘/etc/swift/ssl_key.pem’

[root@Node3]#openstack-config –set /etc/swift/proxy-server.conf  DEFAULT cert_file ‘/etc/swift/ssl_cert.pem’

4. Update swift endpoint with https
#content of ~/openrc

[root@Node3 ~]# cat openrc
export OS_AUTH_URL=”http://Node3:35357/v3
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3
export OS_USERNAME=”admin”
export OS_PASSWORD=”passw0rd”
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_PROJECT_DOMAIN_NAME=Default

[root@Node3 ~] source ~/openrc

Existing swift endpoints look like
[root@Node3 swift]# openstack endpoint list
+———————————-+———–+————–+————–+———+———–
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+———————————-+———–+————–+————–+———+———– |
| 93fa11d1fa7b4622abc857f964676e68 | RegionOne | swift | object-store | True | public | http://Node3:8080/v1/AUTH_%(tenant_id)s |
| 9f271a9d2b14471c8bbad7edca8c4a18 | RegionOne | swift | object-store | True | internal | http://Node3:8080/v1/AUTH_%(tenant_id)s |
| d70496da0a884381a818623ca5b7c501 | RegionOne | swift | object-store | True | admin | http://Node3:8080 |

Change the swift endpoint to https [change the endpoint ID as per your environment]
openstack endpoint set –url ‘https://Node3:8080/v1/AUTH_%(tenant_id)s‘ 93fa11d1fa7b4622abc857f964676e68
openstack endpoint set –url ‘https://Node3:8080/v1/AUTH_%(tenant_id)s‘ 9f271a9d2b14471c8bbad7edca8c4a18
openstack endpoint set –url ‘https://Node3:8080‘ d70496da0a884381a818623ca5b7c501

Updated swift endpoint look like
[root@Node3 swift]# openstack endpoint list
+———————————-+———–+————–+————–+———+———–
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+———————————-+———–+————–+————–+———+———–
| 93fa11d1fa7b4622abc857f964676e68 | RegionOne | swift | object-store | True | public | https://Node3:8080/v1/AUTH_%(tenant_id)s |
| 9f271a9d2b14471c8bbad7edca8c4a18 | RegionOne | swift | object-store | True | internal | https://Node3:8080/v1/AUTH_%(tenant_id)s |
| d70496da0a884381a818623ca5b7c501 | RegionOne | swift | object-store | True | admin | https://Node3:8080 |

5. Sample swift client command
[root@Node3]# swift –os-cacert ssl_cacert.pem stat
Account: AUTH_afcc267ea2c842e59082162118d5047e
Containers: 0
Objects: 0
Bytes: 0
X-Put-Timestamp: 1449638753.75244
X-Timestamp: 1449638753.75244
X-Trans-Id: txe9e7ceb7a31c48e193495-005667bb61
Content-Type: text/plain; charset=utf-8

“These are my personal views and do not necessarily reflect that of my employer”

Keystone with Active Directory and Multiple OU lookup

Generally the Active Directory deployments are very large and properly organised under multiple Organizational Units(OU).  OUs can be based on Department, Functioning groups etc.

Following diagram depict the sample AD environment with multiple OU.

Screenshot from 2015-11-27 22:06:42

One can configure the AD with Keystone using ldap identity provider so that users from AD are visible on Keystone.

In current keystone ldap provider there is no mechanism of providing two or more OUs. For Example today if one want to use only user from OU=Comp and OU=Admin for keystone he has to provide user OU as dc=myuniv,dc=com ie root of AD. Because of this all users from all OUs will be visible to keystone. Currently providing multiple OUs in keystone configuration in not in plan.

However one can limit the number of users visible to keystone using following two mechanisms from Active Directory.

  1. Update attribute of all users those users should be visible to keystone and update the
    For example I chose ‘description’ attribute of user on Active Directory and updated the this attribute of all user those should be visible to keystone with description=OBJECT_USERRun following on spectrum scale cluster to update the filter
    mmobj config change –ccrfile keystone.conf –section ldap –property user_filter –value ‘(description=OBJECT_USER)’
  2. Another approach apply ACL on bind user such that only required OU’s will be visible to keystone.In this deny full acl need to be added for bind user on all OU except the OU’s from which users will be visible to keystone.

    For example in my setup I dont want user from OU=org1. testuser1 is binduser hence i added  deny acl for testuser1 on OU=org1.
    Deny
    Now keystone will list all users expect users from OU=org1.

    “These are my personal views and do not necessarily reflect that of my employer”

Configuring IBM Spectrum Scale Object With External Keystone

IBM Spectrum Scale provide an option to configure Object with External Keystone.

There are two method to achieve the same First is using Installer to Configure the Object with External Keystone and Other is to use mmuserauth cli for setting up Object Authentication. 

Prerequisite :
    Following entities must be present on keystone server which is hosted outside of Spectrum Scale cluster. 

  • User ‘swift’ and ‘admin’ with valid password
  • Project ‘service’ and ‘admin’
  • Role ‘admin’
  • ‘swift’ user should have ‘admin’ role in ‘service’ tenant
  • ‘admin’ user should have ‘admin’ role in ‘admin’ tenant
  • keystone service of type ‘identity’
  • Keystone endpoint
  • Swift service of type ‘object-store’
  • Swift endpoint.

Refer Configure Openstack-Keystone for IBM Spectrum Scale Object Storage for configuring external keystone server which fulfill all above requirement.

On Spectrum Scale cluster. 

# Remove the existing authentication if any using following command

[root@swiftnode ~]# mmuserauth service remove --data-access-method object
[root@swiftnode ~]# mmuserauth service remove --data-access-method object --idmapdelete

# Configure IBM Spectrum Scale Object with external keystone using following command

[root@swiftnode ~]# mmuserauth service create --data-access-method object --type userdefined --ks-ext-endpoint http://mykeystone:35357/v3 --ks-swift-user swift --ks-swift-pwd password

Spectrum Scale Object is configured with External Keystone and Waiting for you to upload lots of Object. Hurry up … 🙂

Disclaimer: The content of this post is not approved nor endorsed by IBM.

Configure Openstack-Keystone for IBM Spectrum Scale Object Storage

Steps for configuring External keystone server for IBM Spectrum Scale Object Storage

Prerequisite :
               RHEL 7 or 7.1 host with Enabled Redhat and Openstack Kilo repository

# Install openstack-keystone rpm and other required rpms from repository

 $ yum install openstack-keystone openstack-utils openldap-clients python-openstackclient -y 

# Add required firewall rule on node or stop the firewalld

$ service firewalld stop

# Update the keystone.conf. Update admin_token for administration/configuration

$ openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token ADMIN 

# Update the database connection. Assumption: Mysql(MariaDB) will be used and same node is used as database node.

$ openstack-config --set /etc/keystone/keystone.conf database connection 'mysql://keystone:Passw0rd@localhost/keystone'

# In this setup PKI is used for token. One can choose to use UUID for token, In that case skip following steps.

$ openstack-config --set /etc/keystone/keystone.conf token provider 'keystone.token.providers.pki.Provider'
$ keystone-manage pki_setup --keystone-user keystone --keystone-group keystone

# Install mariadb and initialize the keystone database

$ /usr/bin/openstack-db --service keystone --init --password password --rootpw password

$ service openstack-keystone start 

# At this stage Openstack-keystone service will be running

$ export OS_TOKEN="ADMIN"
$ export OS_IDENTITY_API_VERSION=3
$ export OS_URL=http://localhost:35357/v3 

#  Create required User,Project,Role entries in Keystone

$ openstack project create --domain default service
$ openstack project create --domain default admin

$ openstack user create --password password admin --domain default
$ openstack user create --password password swift --domain default

$ openstack role create admin

$ openstack role add --user admin --domain default admin
$ openstack role add --user admin --project admin admin
$ openstack role add --user swift --domain default admin
$ openstack role add --user swift --project service admin

# Create Keystone endpoints

$ openstack service create --name keystone identity
$ keystoneservice=`openstack service show keystone -f value -c id`
$ keystoneendpoint='mykeystone' #---> Change this as per hostname/dnsname of keystone
$ openstack endpoint create $keystoneservice public http://${keystoneendpoint}:5000/v3
$ openstack endpoint create $keystoneservice internal http://${keystoneendpoint}:5000/v3
$ openstack endpoint create $keystoneservice admin http://${keystoneendpoint}:35357/v3 

# Create Swift endpoints

$ openstack service create --name swift object-store
$ swiftservice=`openstack service show swift -f value -c id`
$ swiftendpoint='swiftnode'  #---> Change this as per hostname/dnsname of swift
$ openstack endpoint create $swiftservice public  "http://${swiftendpoint}:8080/v1/AUTH_%(tenant_id)s"
$ openstack endpoint create $swiftservice internal  "http://${swiftendpoint}:8080/v1/AUTH_%(tenant_id)s"
$ openstack endpoint create $swiftservice admin  "http://${swiftendpoint}:8080"

# Remove admin_token from keystone.conf

$ openstack-config --del /etc/keystone/keystone.conf DEFAULT admin_token

# Restart the openstack-keystone service to pickup the admin_token deletion.

$ service openstack-keystone restart


Disclaimer: The content of this post is not approved nor endorsed by IBM.

Now the keystone server is ready to be configured with IBM Spectrum Scale Object Store in USERDEFINED configuration.