Clojure and DynamoDB with Faraday, Part 3

, , , , ,

Welcome back

In part 1 we went over the basic table creation operations, as well as writing data, and on part 2 we looked at getting, querying and scanning data.

We’ve only been working with the primary key so far, but often we’ll want to get data using a secondary index. On this part we’ll follow step 6 of Amazon’s Getting Started. We’ll see how to create, query and scan them, as well as how Faraday handles these asynchronous calls.

Create a global secondary index

I expect you’ve already gone over the two previous parts, so you’ll have a table :music with a hash key on the artist name and song title.

DynamoDB supports global and local secondary indices. If we want to create a new global secondary index, we pass its details to update-table. 

1
2
3
4
5
6
7
8
9
(def update-result 
  (far/update-table 
    client-opts :music
    {:gsindexes {:operation    :create
                 :name         "genre-and-price-index"
                 :hash-keydef  [:genre :s]
                 :range-keydef [:price :n]
                 :throughput   {:read 1 :write 1}
                 }}))

If you’re comparing it against the original example, in Faraday:

  • :projection defaults to :all, so we don’t need to specify it, and
  • much like when we created the table, we don’t need to send the attribute definitions and key schema separately, as Faraday can obtain the attributes from the key schema declaration.

You’ll notice that since :gsindexes is a key on a map. This is because we can only request one global secondary index operation at a time. We will not be able to successfully execute a new one while the first one is executing, which may take a while depending on the size of your table.

As that implies, update-table is an asynchronous call, and the operation may not have completed by the time the function returns. Since this is an async call, update-table will return a future. You can deref this value to get the result once it has completed. Eventually, @update-result will get us the table’s new state.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{:lsindexes nil,
 :gsindexes [{:name :genre-and-price-index,
              :size nil,
              :item-count nil,
              :key-schema [{:name :genre, :type :hash} {:name :price, :type :range}],
              :projection {:projection-type "ALL", :non-key-attributes nil},
              :throughput {:read 1, :write 1, :last-decrease nil, 
                           :last-increase nil, :num-decreases-today nil}}],
 :name :music,
 :throughput {:read 1,
              :write 1,
              :last-decrease #inst"1970-01-01T00:00:00.000-00:00",
              :last-increase #inst"1970-01-01T00:00:00.000-00:00",
              :num-decreases-today 0},
 :prim-keys {:artist {:key-type :hash, :data-type :s},
             :song-title {:key-type :range, :data-type :s},
             :price {:data-type :n},
             :genre {:data-type :s}},
 :size 673,
 :status :active,
 :item-count 5,
 :creation-date #inst"2015-12-19T07:47:03.480-00:00",
 :indexes nil}

Query the index

Now that we have a spanking new secondary index, we can use it to query for all songs in a genre and price range.

This is trivial using same query call we used on part 2 - we merely add the index we want to use to the options map.

1
2
3
4
5
6
7
8
(far/query client-opts :music {:genre [:eq :country]}
           {:proj-expr       "#s, price"
            :index           "genre-and-price-index"
            :expr-attr-names {"#s" "song-title"}})

;; =>
[{:price 1.98M, :song-title "My Dog Spot"} 
 {:price 2.14M, :song-title "Call Me Today"}]

We can also include the range key, if (for instance) we want to query for songs in a genre that are above a certain price.

1
2
3
4
5
6
7
8
(far/query client-opts :music {:genre [:eq "country"]
                               :price [:gt 2]}
           {:proj-expr       "#s, price"
            :index           "genre-and-price-index"
            :expr-attr-names {"#s" "song-title"}})

;; => 
[{:price 2.14M, :song-title "Call Me Today"}]

Scan the index

Finally, we can scan an index providing both the table and index name, and use projection expressions.

1
2
3
4
(far/scan client-opts :music {:index "genre-and-price-index"
                              :proj-expr "genre, price, artist, #s, #at"
                              :expr-attr-names {"#s" "song-title" 
                                                "#at" "album-title"}})

Which returns the values we expected.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[{:artist "No One You Know", 
  :price 1.98M, 
  :genre "country", 
  :song-title "My Dog Spot", 
  :album-title "Hey Now"}
 {:artist "No One You Know",
  :price 2.14M,
  :genre "country",
  :song-title "Call Me Today",
  :album-title "Somewhat Famous"}
 {:artist "The Acme Band",
  :price 0.99M,
  :genre "rock",
  :song-title "Look Out, World",
  :album-title "The Buck Starts Here"}
 {:artist "The Acme Band",
  :price 2.47M,
  :genre "rock",
  :song-title "Still In Love",
  :album-title "The Buck Starts Here"}]

Next steps

On the next and final part 4 we’ll learn about updates, conditional writes and atomic operations.