Let’s start hacking!!
Most of the time, one needs to know the heketi configuration and topology he is running with. I do this at times and today thought of sharing with a wider audience, there are more hacking that can be done once you got a handle of the things, but I put a disclaimer that, you should not do it unless you know what you are doing. Please reach out to support channels if you are playing this in production.
Use oc/kubectl get pods to know the heketi pod which is running in your setup.
[mynode~]# oc get pods NAME READY STATUS RESTARTS AGE busybox 1/1 Running 423 17d busybox2 1/1 Running 403 16d glusterblock-provisioner-dc-9-ch2lr 1/1 Running 0 6d glusterfs-bztdb 1/1 Running 0 23d glusterfs-h6sfw 1/1 Running 0 23d glusterfs-vl6l8 1/1 Running 0 23d heketi-1-dqcp2 1/1 Running 1 6d storage-project-router-2-x5dsv 1/1 Running 0 23d
We got the name of the heketi pod, now let’s describe the pod spec.
[mynode ~]# oc describe pod heketi-1-dqcp2 Name: heketi-1-dqcp2 Namespace: storage-project Node: example.com/10.70.47.1 Start Time: Fri, 02 Mar 2018 17:33:38 +0530 Labels: deployment=heketi-1 deploymentconfig=heketi glusterfs=heketi-pod heketi=pod Annotations: openshift.io/deployment-config.latest-version=1 openshift.io/deployment-config.name=heketi openshift.io/deployment.name=heketi-1 openshift.io/scc=privileged Status: Running IP: 10.128.0.41 Controlled By: ReplicationController/heketi-1 Containers: heketi: Container ID: docker://61e4ffd7d6adda7f1c9c663ddf4ba44d54bdd6197d2677684e87ef7bfa16da4f Image: X.X.X.X Image ID: docker-pullable://XXXX@sha256:6f7ab12b8d39e66dc186da3152c8dbf1e7682eb1cdecd0f80fa86b267b0221c1 Port: 8080/TCP State: Running Started: Fri, 02 Mar 2018 17:42:06 +0530 Last State: Terminated Reason: Error Exit Code: 137 Started: Fri, 02 Mar 2018 17:34:16 +0530 Finished: Fri, 02 Mar 2018 17:42:00 +0530 Ready: True Restart Count: 1 Liveness: http-get http://:8080/hello delay=30s timeout=3s period=10s #success=1 #failure=3 Readiness: http-get http://:8080/hello delay=3s timeout=3s period=10s #success=1 #failure=3 Environment: HEKETI_USER_KEY: HEKETI_ADMIN_KEY: HEKETI_EXECUTOR: kubernetes HEKETI_FSTAB: /var/lib/heketi/fstab HEKETI_SNAPSHOT_LIMIT: 14 HEKETI_KUBE_GLUSTER_DAEMONSET: y Mounts: /etc/heketi from config (rw) /var/lib/heketi from db (rw) /var/run/secrets/kubernetes.io/serviceaccount from heketi-service-account-token-mstk7 (ro) Conditions: Type Status Initialized True Ready True PodScheduled True Volumes: db: Type: Glusterfs (a Glusterfs mount on the host that shares a pod’s lifetime) EndpointsName: heketi-storage-endpoints Path: heketidbstorage ReadOnly: false config: Type: Secret (a volume populated by a Secret) SecretName: heketi-config-secret Optional: false heketi-service-account-token-mstk7: Type: Secret (a volume populated by a Secret) SecretName: heketi-service-account-token-mstk7 Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: <none> Events: <none>
Pay attention to config section of type secret. You got the secret name ie heketi-config-secret from it. Let’s see whats there in this secret.
[mynode ~]# oc get secret heketi-config-secret -o yaml apiVersion: v1 data: heketi.json: ewoJIl9wb3J0X2NvbW1lbnQiOiAiSGVrZXRpIFNlcnZlciBQb3J0IE51bWJlciIsCgkicG9ydCIgOiAiODA4MCIsCgoJIl91c2VfYXV0aCI6ICJFbmFibGUgSldUIGF1dGhvcml6YXRpb24uIFBsZWFzZSBlbmFibGUgZm9yIGRlcGxveW1lbnQiLAoJInVzZV9hdXRoIiA6IGZhbHNlLAoKCSJfand0IiA6ICJQcml2YXRlIGtleXMgZm9yIGFjY2VzcyIsCgkiand0IiA6IHsKCQkiX2FkbWluIiA6ICJBZG1pbiBoYXMgYWNjZXNzIHRvIGFsbCBBUElzIiwKCQkiYWRtaW4iIDogewoJCQkia2V5IiA6ICJNeSBTZWNyZXQiCgkJfSwKCQkiX3VzZXIiIDogIlVzZXIgb25seSBoYXMgYWNjZXNzIHRvIC92b2x1bWVzIGVuZHBvaW50IiwKCQkidXNlciIgOiB7CgkJCSJrZXkiIDogIk15IFNlY3JldCIKCQl9Cgl9LAoKCSJfZ2x1c3RlcmZzX2NvbW1lbnQiOiAiR2x1c3RlckZTIENvbmZpZ3VyYXRpb24iLAoJImdsdXN0ZXJmcyIgOiB7CgoJCSJfZXhlY3V0b3JfY29tbWVudCI6ICJFeGVjdXRlIHBsdWdpbi4gUG9zc2libGUgY2hvaWNlczogbW9jaywga3ViZXJuZXRlcywgc3NoIiwKCQkiZXhlY3V0b3IiIDogImt1YmVybmV0ZXMiLAoKCQkiX2RiX2NvbW1lbnQiOiAiRGF0YWJhc2UgZmlsZSBuYW1lIiwKCQkiZGIiIDogIi92YXIvbGliL2hla2V0aS9oZWtldGkuZGIiLAoKCQkia3ViZWV4ZWMiIDogewoJCQkicmViYWxhbmNlX29uX2V4cGFuc2lvbiI6IHRydWUKCQl9LAoKCQkic3NoZXhlYyIgOiB7CgkJCSJyZWJhbGFuY2Vfb25fZXhwYW5zaW9uIjogdHJ1ZSwKCQkJImtleWZpbGUiIDogIi9ldGMvaGVrZXRpL3ByaXZhdGVfa2V5IiwKCQkJInBvcnQiIDogIjIyIiwKCQkJInVzZXIiIDogInJvb3QiLAoJCQkic3VkbyIgOiBmYWxzZQoJCX0sCgoJCSJfYXV0b19jcmVhdGVfYmxvY2tfaG9zdGluZ192b2x1bWUiOiAiQ3JlYXRlcyBCbG9jayBIb3N0aW5nIHZvbHVtZXMgYXV0b21hdGljYWxseSBpZiBub3QgZm91bmQgb3IgZXhzaXN0aW5nIHZvbHVtZSBleGhhdXN0ZWQiLAoJCSJhdXRvX2NyZWF0ZV9ibG9ja19ob3N0aW5nX3ZvbHVtZSI6IHRydWUsCgoJCSJfYmxvY2tfaG9zdGluZ192b2x1bWVfc2l6ZSI6ICJOZXcgYmxvY2sgaG9zdGluZyB2b2x1bWUgd2lsbCBiZSBjcmVhdGVkIGluIHNpemUgbWVudGlvbmVkLCBUaGlzIGlzIGNvbnNpZGVyZWQgb25seSBpZiBhdXRvLWNyZWF0ZSBpcyBlbmFibGVkLiIsCgkJImJsb2NrX2hvc3Rpbmdfdm9sdW1lX3NpemUiOiA1MDAKCX0sCgoJImJhY2t1cF9kYl90b19rdWJlX3NlY3JldCI6IGZhbHNlCn0K private_key: null topology.json: ewogICAgImNsdXN0ZXJzIjogWwogICAgICAgIHsKICAgICAgICAgICAgIm5vZGVzIjogWwogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJub2RlIjogewogICAgICAgICAgICAgICAgICAgICAgICAiaG9zdG5hbWVzIjogewogICAgICAgICAgICAgICAgICAgICAgICAgICAgIm1hbmFnZSI6IFsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiZGhjcDQ2LTE4Ny5sYWIuZW5nLmJsci5yZWRoYXQuY29tIgogICAgICAgICAgICAgICAgICAgICAgICAgICAgXSwKICAgICAgICAgICAgICAgICAgICAgICAgICAgICJzdG9yYWdlIjogWwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICIxMC43MC40Ni4xODciCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBdCiAgICAgICAgICAgICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgICAgICAgICAgICJ6b25lIjogMQogICAgICAgICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgICAgICAgImRldmljZXMiOiBbCiAgICAgICAgICAgICAgICAgICAgICAgICIvZGV2L3NkYyIsCiAgICAgICAgICAgICAgICAgICAgICAgICIvZGV2L3NkZCIKICAgICAgICAgICAgICAgICAgICBdCiAgICAgICAgICAgICAgICB9LAogICAgICAgICAgICAgICAgewogICAgICAgICAgICAgICAgICAgICJub2RlIjogewogICAgICAgICAgICAgICAgICAgICAgICAiaG9zdG5hbWVzIjogewogICAgICAgICAgICAgICAgICAgICAgICAgICAgIm1hbmFnZSI6IFsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiZGhjcDQ3LTEubGFiLmVuZy5ibHIucmVkaGF0LmNvbSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIF0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAic3RvcmFnZSI6IFsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiMTAuNzAuNDcuMSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIF0KICAgICAgICAgICAgICAgICAgICAgICAgfSwKICAgICAgICAgICAgICAgICAgICAgICAgInpvbmUiOiAyCiAgICAgICAgICAgICAgICAgICAgfSwKICAgICAgICAgICAgICAgICAgICAiZGV2aWNlcyI6IFsKICAgICAgICAgICAgICAgICAgICAgICAgIi9kZXYvc2RjIiwKICAgICAgICAgICAgICAgICAgICAgICAgIi9kZXYvc2RkIgogICAgICAgICAgICAgICAgICAgIF0KICAgICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgICB7CiAgICAgICAgICAgICAgICAgICAgIm5vZGUiOiB7CiAgICAgICAgICAgICAgICAgICAgICAgICJob3N0bmFtZXMiOiB7CiAgICAgICAgICAgICAgICAgICAgICAgICAgICAibWFuYWdlIjogWwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICJkaGNwNDYtOTgubGFiLmVuZy5ibHIucmVkaGF0LmNvbSIKICAgICAgICAgICAgICAgICAgICAgICAgICAgIF0sCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAic3RvcmFnZSI6IFsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAiMTAuNzAuNDYuOTgiCiAgICAgICAgICAgICAgICAgICAgICAgICAgICBdCiAgICAgICAgICAgICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgICAgICAgICAgICJ6b25lIjogMQogICAgICAgICAgICAgICAgICAgIH0sCiAgICAgICAgICAgICAgICAgICAgImRldmljZXMiOiBbCiAgICAgICAgICAgICAgICAgICAgICAgICIvZGV2L3NkYyIsCiAgICAgICAgICAgICAgICAgICAgICAgICIvZGV2L3NkZCIKICAgICAgICAgICAgICAgICAgICBdCiAgICAgICAgICAgICAgICB9CiAgICAgICAgICAgIF0KICAgICAgICB9CiAgICBdCn0K kind: Secret metadata: creationTimestamp: 2018-02-13T13:00:48Z labels: glusterfs: heketi-config-secret heketi: config-secret name: heketi-config-secret namespace: storage-project resourceVersion: “28053” selfLink: /api/v1/namespaces/storage-project/secrets/heketi-config-secret uid: e969d95a-10bd-11e8-9c7b-005056a5a340 type: Opaque
From the tags, we have
heketi.json and topology.json.
The value of these tags/fields are base64 encoded values, so let us pick the heketi.json field and decode it as shown below.
echo ewoJIl9wb3J0X2NvbW1lbnQiOiAiSGVrZXRpIFNlcnZlciBQb3J0IE51bWJlciIsCgkicG9ydCIgOiAiODA4MCIsCgoJIl91c2VfYXV0aCI6ICJFbmFibGUgSldUIGF1dGhvcml6YXRpb24uIFBsZWFzZSBlbmFibGUgZm9yIGRlcGxveW1lbnQiLAoJInVzZV9hdXRoIiA6IGZhbHNlLAoKCSJfand0IiA6ICJQcml2YXRlIGtleXMgZm9yIGFjY2VzcyIsCgkiand0IiA6IHsKCQkiX2FkbWluIiA6ICJBZG1pbiBoYXMgYWNjZXNzIHRvIGFsbCBBUElzIiwKCQkiYWRtaW4iIDogewoJCQkia2V5IiA6ICJNeSBTZWNyZXQiCgkJfSwKCQkiX3VzZXIiIDogIlVzZXIgb25seSBoYXMgYWNjZXNzIHRvIC92b2x1bWVzIGVuZHBvaW50IiwKCQkidXNlciIgOiB7CgkJCSJrZXkiIDogIk15IFNlY3JldCIKCQl9Cgl9LAoKCSJfZ2x1c3RlcmZzX2NvbW1lbnQiOiAiR2x1c3RlckZTIENvbmZpZ3VyYXRpb24iLAoJImdsdXN0ZXJmcyIgOiB7CgoJCSJfZXhlY3V0b3JfY29tbWVudCI6ICJFeGVjdXRlIHBsdWdpbi4gUG9zc2libGUgY2hvaWNlczogbW9jaywga3ViZXJuZXRlcywgc3NoIiwKCQkiZXhlY3V0b3IiIDogImt1YmVybmV0ZXMiLAoKCQkiX2RiX2NvbW1lbnQiOiAiRGF0YWJhc2UgZmlsZSBuYW1lIiwKCQkiZGIiIDogIi92YXIvbGliL2hla2V0aS9oZWtldGkuZGIiLAoKCQkia3ViZWV4ZWMiIDogewoJCQkicmViYWxhbmNlX29uX2V4cGFuc2lvbiI6IHRydWUKCQl9LAoKCQkic3NoZXhlYyIgOiB7CgkJCSJyZWJhbGFuY2Vfb25fZXhwYW5zaW9uIjogdHJ1ZSwKCQkJImtleWZpbGUiIDogIi9ldGMvaGVrZXRpL3ByaXZhdGVfa2V5IiwKCQkJInBvcnQiIDogIjIyIiwKCQkJInVzZXIiIDogInJvb3QiLAoJCQkic3VkbyIgOiBmYWxzZQoJCX0sCgoJCSJfYXV0b19jcmVhdGVfYmxvY2tfaG9zdGluZ192b2x1bWUiOiAiQ3JlYXRlcyBCbG9jayBIb3N0aW5nIHZvbHVtZXMgYXV0b21hdGljYWxseSBpZiBub3QgZm91bmQgb3IgZXhzaXN0aW5nIHZvbHVtZSBleGhhdXN0ZWQiLAoJCSJhdXRvX2NyZWF0ZV9ibG9ja19ob3N0aW5nX3ZvbHVtZSI6IHRydWUsCgoJCSJfYmxvY2tfaG9zdGluZ192b2x1bWVfc2l6ZSI6ICJOZXcgYmxvY2sgaG9zdGluZyB2b2x1bWUgd2lsbCBiZSBjcmVhdGVkIGluIHNpemUgbWVudGlvbmVkLCBUaGlzIGlzIGNvbnNpZGVyZWQgb25seSBpZiBhdXRvLWNyZWF0ZSBpcyBlbmFibGVkLiIsCgkJImJsb2NrX2hvc3Rpbmdfdm9sdW1lX3NpemUiOiA1MDAKCX0sCgoJImJhY2t1cF9kYl90b19rdWJlX3NlY3JldCI6IGZhbHNlCn0K| base64 --decode
{ “_port_comment”: “Heketi Server Port Number”, “port” : “8080”, “_use_auth”: “Enable JWT authorization. Please enable for deployment”, “use_auth” : false, “_jwt” : “Private keys for access”, “jwt” : { “_admin” : “Admin has access to all APIs”, “admin” : { “key” : “My Secret” }, “_user” : “User only has access to /volumes endpoint”, “user” : { “key” : “My Secret” } }, “_glusterfs_comment”: “GlusterFS Configuration”, “glusterfs” : { “_executor_comment”: “Execute plugin. Possible choices: mock, kubernetes, ssh”, “executor” : “kubernetes”, “_db_comment”: “Database file name”, “db” : “/var/lib/heketi/heketi.db”, “kubeexec” : { “rebalance_on_expansion”: true }, “sshexec” : { “rebalance_on_expansion”: true, “keyfile” : “/etc/heketi/private_key”, “port” : “22”, “user” : “root”, “sudo” : false }, “_auto_create_block_hosting_volume”: “Creates Block Hosting volumes automatically if not found or exsisting volume exhausted”, “auto_create_block_hosting_volume”: true, “_block_hosting_volume_size”: “New block hosting volume will be created in size mentioned, This is considered only if auto-create is enabled.”, “block_hosting_volume_size”: 500 }, “backup_db_to_kube_secret”: false }
Is this what you wanted?