Sodeso

So now that we have covered some of the theoretical parts, we can start to put our knowledge of the data model and sorting behavior to some use. In this blog post i will start to explain how you can configure your ColumnFamilies, make a connection to the database and how you can store, retrieve, modify and delete data.

Cassandra uses the Apache Thrift framework as its client API, Thrift is a separate Apache project which is to put it simply, a binary communication protocol. It is not only used by Cassandra but by a lot of different projects. To get a nice overview of the various companies / projects that use Thrift take a look at their wiki. Thrift is also available for a large number of development languages, ranging from Java and C++ to Python, PHP and many other software development languages.

Thrift also lacks some of the features that are quite import if you are going to use Cassandra, as we know Cassandra supports multiple nodes where it basically doesn’t matter to which node you connect. This has a big advantage that when a node is failing you can just connect to a different node and continue your work. However, Thrift doesn’t support this feature out of the box. Another important feature is connection pooling, Thrift doesn’t do this for you so you need to take care of this yourself. This isn’t because the authors of Thrift still need to add this these features, it is just not the purpose of Thrift. Thrift is just for the communication between software, not every software product that uses Thrift needs these capabilities and it would just create overhead if they were added.

However, there are a few Cassandra clients around that do add this capability along with some other nice features to make your life easier. Personally, i think Hector is one of the best at the moment, it add’s the features i mentioned and a lot more. It is an open source project with the MIT license and it’s hosted on GitHub.

For now, we will focus on only using the Thrift client, learning the basics makes it easier to determine which client supports your needs the best, and if things cannot be done using a client you can always fallback on using the Thrift API directly.

Before we can start with the client we first need to configure the Cassandra database so that it has the appropriate ColumnFamilies that we are going to use. In the CASSANDRA_HOME/conf folder is a configuration file named storage-conf.xml. Open this file in an editor and find the Keyspaces element, by default there are already two keyspaces, one is the system keyspace which is not defined in the XML file but is present within Cassandra, that one is needed for the internals of Cassandra. The other one is the Keyspace1 keyspace, this one is not used by Cassandra and only serves for demo purposes.

Between the existing Keyspaces element place a new Keyspace element:

Now that we have defined our keyspace we can start to add some ColumnFamilies:

So now we have three ColumnFamilies, one to store the information about the Authors, one to store the Posts that the Authors have created, and finally one that will contain the information about the Tags. The reason for the Tags ColumnFamily will get clearer in the following part, for now we only need to know that the Tags ColumnFamily will keep the associations between the Tags and the Posts. A post can have a number of tags and a tag can have a number of posts.

The last three lines are necessary for Cassandra to operate, these are set to the default values and we will keep them like this.

The Authors ColumnFamily will organize the data using Columns, the following is a representation of the ColumnFamily in a sort of hierarchical way:

As you can see it is quite simple, the key is set to the name of the author and the value is de information about the author. The sorting rule we applied to the ColumnFamily is the UTF8Type, so the sorting is done on the name of the author in a alphabetically order.

The Posts SuperColumnFamily will organize the data using SuperColumns which contain Columns:

Every row in the family consists of two SuperColumns, one for the posting itself and one for retaining a list of tags. The key of the rows is created like this on purpose for simple access, if we want to express a URL we can just add this part to the URL like:

http://myblog.com/i-am-allergic-to-animals

Finally we have the Tags ColumnFamily for associating the Posts with the Tags:

As you can see, every tag that is mentioned in a post is used here as a key, for every key we keep track of which posts have that tag. If we now display a post on the screen and show a listing of all the tags associated with it, the user could click on a tag and see all the posts that have a similar tag. This is a sort of indexing that would otherwise not be possible since we cannot search through all the posts in the Posts ColumnFamily by tag. So now we have a clearer understanding of how the data is arranged in the database we can start working on our client.

This posting also includes a Sample code (947) with all the source code in a Eclipse project, the project also contains some extra examples that you can have a look at. When you have the project loaded into your Eclipse you will probably see some error messages, this has to do with a variable in your project libraries. Change the target folder of the variable in the following way:

Right click on the project and choose Properties.
Click on the Java Build Path in the left part and then click on the Libraries tab. Now click on one of the lines that mention CASSANDRA_HOME and choose the Edit… button. Now you can click on the Variable… button to create a new variable with the name CASSANDRA_HOME, make sure it points to your Cassandra home folder ( for example C:/dev/apache-cassandra-0.6.0-beta2 ).

A client needs a number of libraries on it’s classpath to be able to communicate with Cassandra, all these libraries already exist within the CASSANDRA_HOME/lib folder of your Cassandra installation:

• apache-cassandra-x.x.x-betax.jar (The Cassandra libraries which also contains the customization of the Thrift communications protocol)
• slf4j-log4jxx-x.x.xx.jar (Plugin for SLF4J to add support for Log4j)
• slf4j-api-x.x.xx.jar (Wrapper for various logging frameworks)
• log4j-x.x.xx.jar (Log4J logging framework)
• libthrift-xxxx.jar (The Thrift communication protocol)

In Installing and using Apache Cassandra With Java Part 1 you can find the download locations and what versions of the libraries i am using.

First of all we need to be able to make a connection to the database, to do that we are going to open a connection on port 9160, which is the default port for Cassandra, and hand it over to the Cassandra client, the client will take care of the communication with the server:

So now that we know how to open a connection we also need to close the connection in a descent way, the flush that is done before the close takes care of any data that is still residing in the transport buffer:

First we will have a look at how to store a new Author into the database:

First thing you will probably notice is the amount of code needed to get this done, i will provide some solutions to this in a later posting because we still want to focus on using Thrift. So what is happening here, first we create the timestamp. The timestamp is used by the columns and is always needed, never ever try to put just 0 in there, this will get you into trouble when you are using multiple nodes which rely on this timestamp to see if the data you are receiving is the correct data.

Then we create a Map which will hold the rows that we want to insert. The Map contains a String, which is the name of the ColumnFamily that we defined in the configuration file.

The List contains ColumnOrSuperColumn objects. Cassandra supplies us with an aggregate which contains either a Column or a SuperColumn, it is not possible to set both of them (in code it would be no problem, but you will notice when you execute the code since you will receive exceptions).

Then we create the List instance which will contain our Columns that we want to store, in this case the columns for the email, country and registeredSince. After this is done we add the list of columns to the map.

Finally we use the client.batch_insert to store everything at once. It is possible to just store a single Column or SuperColumn, in that case you would use the client.insert method. But in our case we want to store the complete Author at once. As you may have noticed we provide the key of the Author in the batch_insert method and the ColumnFamily in the outer most Map. This allows us to actually create rows in multiple ColumnFamilies at once which have the same row Key.

All the methods that are provided by the client of Cassandra have a ConsistencyLevel parameter. This parameter is used for both read and write actions and it will determine when the request made by the client is successful, the Cassandra Wiki has an excellent explanation of what the various options are. For now it is sufficient to know that the option ANY means that a write action is successful when it has been written to at least one node. For read action we will use ONE which means that the first node will respond with the data (and more is done but for the moment that is not important).

Suppose we have created a number of authors it would also be nice if we can retrieve them again. So lets retrieve a single author back:

First we create a SlicePredicate, the predicate is used to tell Cassandra what data you want to fetch. Here we say that we want to get a range of columns. Remember the sorting rules? they are important in this case, the start and finish use the name attribute of the column, so it will use the sorting mechanism to know if the column should be retrieved or not. Here we just pass in new byte[] {} which is just an empty byte array. For Cassandra it means that we want to fetch every column that belongs to the Author.

The result will contain a list of ColumnOrSuperColumn objects that in turn will contain the Column objects containing our data. Now when we would want to fetch all or a number of authors then we would use the following:

It is not much different then before, the method has changed to get_range_slices and instead of only having a SlicePredicate we also pass in a KeyRange. The KeyRange is used to specify which keys fall into you search criteria. In our case we actually don’t care what the keys itself are, what we do care about is the number of rows, we only want the first three rows.
The list of KeySlice objects that are fetched contain the row key and the columns belonging to that row.

So far we have only used the SlicePredicate to retrieve all columns, now here is an example where we only retrieve the columns we specify:

So now that we have seen a few of the search methods we would also like to change the data. For this we will use the batch_update method. Suppose that we want to change the e-mail address:

First we create the Column that contains the actual mutation that we want to do, we supply it with the correct key and the new email address. Then we need to add the Column to the ColumnOrSuperColumn aggregate and the aggregate into a Mutation object. The Mutation object can also be used to create new Columns or to delete a Column, if you supply it with a key that doesn’t exist in the database it will create it, if it detects that the key already exists it will overwrite the value and the timestamp. Finally you can also put a Deletion object in a Mutation to tell Cassandra that you actually want to remove a Column, we will see an example of that later on. We then add it to the list and into the Map ColumnFamily. Finally we put it in the most outer Map with the key of the row.

So now we have covered the insertion, selection and mutating of the data so that only leaves deleting data. Suppose we want to remove an Author completely, this is quite simple:

We use the remove method and supply it with the keyspace, the key of the author and the Author ColumnFamily. That is. Now if we wanted to remove a single column from an Author we have to do a bit more:

It is basically the same as what we were doing when we were modifying data, except that we don’t create any Columns but we create the Deletion object which contains a predicate to define what we want to delete. In this sample we use the SlicePredicate to store a list of column names that we want to remove.

It is allowed to combine the actions, so a batch can contain the removal of two columns, creation of one column and modification of four existing columns, they are all separate Mutation objects. The sample project attached to this post contains a method that does this.

We have now covered all the basics for a standard ColumnFamily with normal Column objects. For a SuperColumnFamily it looks quite the same except that you have an extra level within the data structure, we will cover that the next time and we will talk about some other things.

Hope you have enjoyed this posting, if you have any remarks or suggestions then please let me know, if you want to know when the next posting is live then just subscribe to my twitter account..

The following download contains an Eclipse project with all the source code (and some more) that we talked about in the posting:

When you have the project loaded into your Eclipse you will probably see some error messages, this has to do with a variable in your project libraries. Change the target folder of the variable in the following way:

Right click on the project and choose Properties.
Click on the Java Build Path in the left part and then click on the Libraries tab. Now click on one of the lines that mention CASSANDRA_HOME and choose the Edit… button. Now you can click on the Variable… button to create a new variable with the name CASSANDRA_HOME, make sure it points to your Cassandra home folder ( for example C:/dev/apache-cassandra-0.6.0-beta2 ).