I have a requirement to find out where an object that is uploaded to a swift cluster is being stored. I finally found the following info.
If you change to the /etc/swift directory on a node, then you can run swift-get-nodes against the ring files to find where the data is located.
If you run swift-get-nodes container.ring.gz <account>, you can find info about the account. The account must be the full account name, including the auth method (AUTH_, LDAP_, KEY_, etc):
# swift-get-nodes account.ring.gz LDAP_ldillon
Account LDAP_ldillon
Container None
Object None
Partition 28748
Hash 704c1d3793cb0cba7a6e24b8fbf962c8
Server:Port Device 172.17.1.11:6002 d6
Server:Port Device 172.17.1.10:6002 d3
Server:Port Device 172.17.1.10:6002 d2
Server:Port Device 172.17.1.11:6002 d5 [Handoff]
Server:Port Device 172.17.1.10:6002 d0 [Handoff]
Server:Port Device 172.17.1.10:6002 d1 [Handoff]
curl -I -XHEAD "http://172.17.1.11:6002/d6/28748/LDAP_ldillon"
curl -I -XHEAD "http://172.17.1.10:6002/d3/28748/LDAP_ldillon"
curl -I -XHEAD "http://172.17.1.10:6002/d2/28748/LDAP_ldillon"
curl -I -XHEAD "http://172.17.1.11:6002/d5/28748/LDAP_ldillon" # [Handoff]
curl -I -XHEAD "http://172.17.1.10:6002/d0/28748/LDAP_ldillon" # [Handoff]
curl -I -XHEAD "http://172.17.1.10:6002/d1/28748/LDAP_ldillon" # [Handoff]
Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 172.17.1.11 "ls -lah ${DEVICE:-/srv/node*}/d6/accounts/28748/2c8/704c1d3793cb0cba7a6e24b8fbf962c8"
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d3/accounts/28748/2c8/704c1d3793cb0cba7a6e24b8fbf962c8"
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d2/accounts/28748/2c8/704c1d3793cb0cba7a6e24b8fbf962c8"
ssh 172.17.1.11 "ls -lah ${DEVICE:-/srv/node*}/d5/accounts/28748/2c8/704c1d3793cb0cba7a6e24b8fbf962c8" # [Handoff]
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d0/accounts/28748/2c8/704c1d3793cb0cba7a6e24b8fbf962c8" # [Handoff]
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d1/accounts/28748/2c8/704c1d3793cb0cba7a6e24b8fbf962c8" # [Handoff]
note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.
This shows that the sqlite db file for the account LDAP_ldillon is stored on node 172.17.1.11 (cluster node replication address), directory /srv/node/d6/accounts/28748/2c8/704c1d3793cb0cba7a6e24b8fbf962c8.
28748 is the partition number, 2c8 is the last three hex characters of the hash of the account, and the rest is, of course, the full hash. That is the directory the account db is located in. The actual file is:
# ll /srv/node/d3/accounts/28748/2c8/704c1d3793cb0cba7a6e24b8fbf962c8/
total 1024
-rw-------. 1 swift swift 17408 Feb 6 11:25 704c1d3793cb0cba7a6e24b8fbf962c8.db
-rw-r--r--. 1 swift swift 0 Feb 6 11:25 704c1d3793cb0cba7a6e24b8fbf962c8.db.pending
That is the sqlite file with the account information. You can look at the file with sqlite3. The container table has interesting information:
# sqlite3 /srv/node/d3/accounts/28748/2c8/704c1dba7a6e24b8fbf962c8/704c1d3793cb0cba7a6e24b8fbf962c8.db
SQLite version 3.6.20
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from container;
84|my_test|1421086860.07597|0|22|1825382740|0|1
88|books|1423055299.11577|0|12|89793423|0|1
90|media|1423055424.51607|0|14|6587548715|0|1
106|.swiftstack-web-console-metadata|1423239655.15962|0|1|42|0|1
sqlite>
This shows three containers associated with this account, my_test, books, and media. You can now query the container information:
# swift-get-nodes container.ring.gz LDAP_ldillon books
Account LDAP_ldillon
Container books
Object None
Partition 54608
Hash d55082e326b1c6adee1f9cca82239fb9
Server:Port Device 172.17.1.11:6001 d6
Server:Port Device 172.17.1.11:6001 d7
Server:Port Device 172.17.1.10:6001 d2
Server:Port Device 172.17.1.10:6001 d0 [Handoff]
Server:Port Device 172.17.1.10:6001 d3 [Handoff]
Server:Port Device 172.17.1.11:6001 d4 [Handoff]
curl -I -XHEAD "http://172.17.1.11:6001/d6/54608/LDAP_ldillon/books"
curl -I -XHEAD "http://172.17.1.11:6001/d7/54608/LDAP_ldillon/books"
curl -I -XHEAD "http://172.17.1.10:6001/d2/54608/LDAP_ldillon/books"
curl -I -XHEAD "http://172.17.1.10:6001/d0/54608/LDAP_ldillon/books" # [Handoff]
curl -I -XHEAD "http://172.17.1.10:6001/d3/54608/LDAP_ldillon/books" # [Handoff]
curl -I -XHEAD "http://172.17.1.11:6001/d4/54608/LDAP_ldillon/books" # [Handoff]
Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 172.17.1.11 "ls -lah ${DEVICE:-/srv/node*}/d6/containers/54608/fb9/d55082e326b1c6adee1f9cca82239fb9"
ssh 172.17.1.11 "ls -lah ${DEVICE:-/srv/node*}/d7/containers/54608/fb9/d55082e326b1c6adee1f9cca82239fb9"
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d2/containers/54608/fb9/d55082e326b1c6adee1f9cca82239fb9"
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d0/containers/54608/fb9/d55082e326b1c6adee1f9cca82239fb9" # [Handoff]
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d3/containers/54608/fb9/d55082e326b1c6adee1f9cca82239fb9" # [Handoff]
ssh 172.17.1.11 "ls -lah ${DEVICE:-/srv/node*}/d4/containers/54608/fb9/d55082e326b1c6adee1f9cca82239fb9" # [Handoff]
note: `/srv/node*` is used as default value of `devices`, the real value is set in the config file on each storage node.
This will show the location of the containers. You can repeat with objects doing:
# swift-get-nodes object-1.ring.gz LDAP_ldillon books EloquentRuby.epub
Account LDAP_ldillon
Container books
Object EloquentRuby.epub
Partition 4507
Hash 119b3e7e9340eb4c2820e2c56a7c5855
Server:Port Device 172.17.1.11:6000 d5
Server:Port Device 172.17.1.10:6000 d1
Server:Port Device 172.17.1.11:6000 d6 [Handoff]
Server:Port Device 172.17.1.10:6000 d2 [Handoff]
curl -I -XHEAD "http://172.17.1.11:6000/d5/4507/LDAP_ldillon/books/EloquentRuby.epub" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://172.17.1.10:6000/d1/4507/LDAP_ldillon/books/EloquentRuby.epub" -H "X-Backend-Storage-Policy-Index: 1"
curl -I -XHEAD "http://172.17.1.11:6000/d6/4507/LDAP_ldillon/books/EloquentRuby.epub" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
curl -I -XHEAD "http://172.17.1.10:6000/d2/4507/LDAP_ldillon/books/EloquentRuby.epub" -H "X-Backend-Storage-Policy-Index: 1" # [Handoff]
Use your own device location of servers:
such as "export DEVICE=/srv/node"
ssh 172.17.1.11 "ls -lah ${DEVICE:-/srv/node*}/d5/objects-1/4507/855/119b3e7e9340eb4c2820e2c56a7c5855"
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d1/objects-1/4507/855/119b3e7e9340eb4c2820e2c56a7c5855"
ssh 172.17.1.11 "ls -lah ${DEVICE:-/srv/node*}/d6/objects-1/4507/855/119b3e7e9340eb4c2820e2c56a7c5855" # [Handoff]
ssh 172.17.1.10 "ls -lah ${DEVICE:-/srv/node*}/d2/objects-1/4507/855/119b3e7e9340eb4c2820e2c56a7c5855" # [Handoff]
(I have another storage policy 1, so instead of my objects being placed in the default of objects, it is placed in objects-1).
You can now get the location of the object. You can get the metadata and stuff about the object using swift-object-info , which gets the metadata from the extended attributes of the filesystem.
Note, this is where the file would be located. All this does is calculate where the file would be placed given the account, container, and object. Remember that the hash is generated from the account, container, object, and a secret swift_hash_path_suffix. This program takes all that information and lists where the file is if it exists, or where a file of that name, being updated to that account and container, would be placed. You can’t run swift-object-info on a file that doesn’t exist, of course, but you can find where a file is by doing the above steps. You really only need to do the last swift-get-nodes step to find the object. The rest is only to find the account info or the container info.
The majority of this information I got from this post: http://blog.chmouel.com/2012/02/01/audit-a-swift-cluster/