Cassandra - learn #2 multinode :)

Last post, I tested Cassandra Installation. However, It was only standalone. I thought how to use it as Multinode. With last configuration. I set them up as: (I was lazy to change hostname)

centos 192.168.111.80
centostest1 192.168.111.81

On "centos": (first node)
Note: Don't forgot to check iptables rule :)
- Checked hosts file.

[surachart@centos apache-cassandra-1.2.0]$ cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.111.80  centos  centos.surachartopun.com
192.168.111.81  centostest1 centostest1.surachartopun.com

- Changed from "localhost to hostname (centos) in "conf/cassandra.yaml" file.

[surachart@centos apache-cassandra-1.2.0]$ grep centos conf/cassandra.yaml
          - seeds: "centos"
listen_address: centos
rpc_address: centos  

- Changed configure in  "conf/cassandra-rackdc.properties" and "conf/cassandra-topology.properties" files.

[surachart@centos apache-cassandra-1.2.0]$ grep \= conf/cassandra-rackdc.properties
dc=DC1
rack=RAC1
[surachart@centos apache-cassandra-1.2.0]$ grep \= conf/cassandra-topology.properties |grep -v \#
192.168.111.80=DC1:RAC1
192.168.111.81=DC1:RAC2
default=DC1:r1

- Started it.

[surachart@centos apache-cassandra-1.2.0]$ bin/cassandra -f &
[1] 3844
[surachart@centos apache-cassandra-1.2.0]$ xss =  -ea -javaagent:bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms938M -Xmx938M -Xmn100M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
 INFO 00:31:44,305 Logging initialized
 INFO 00:31:44,331 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_38
 INFO 00:31:44,333 Heap size: 973078528/973078528
 INFO 00:31:44,335 Classpath: bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/apache-cassandra-1.2.0.jar:bin/../lib/apache-cassandra-clientutil-1.2.0.jar:bin/../lib/apache-cassandra-thrift-1.2.0.jar:bin/../lib/avro-1.4.0-fixes.jar:bin/../lib/avro-1.4.0-sources-fixes.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang-2.6.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:bin/../lib/guava-13.0.1.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jline-1.0.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.7.0.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/metrics-core-2.0.3.jar:bin/../lib/netty-3.5.9.Final.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.7.2.jar:bin/../lib/slf4j-log4j12-1.7.2.jar:bin/../lib/snakeyaml-1.6.jar:bin/../lib/snappy-java-1.0.4.1.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/jamm-0.2.5.jar
 INFO 00:31:44,339 JNA not found. Native methods will be disabled.
 INFO 00:31:44,356 Loading settings from file:/home/surachart/apache-cassandra-1.2.0/conf/cassandra.yaml
 INFO 00:31:44,989 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
 INFO 00:31:44,989 disk_failure_policy is stop
 INFO 00:31:44,994 Global memtable threshold is enabled at 309MB
 INFO 00:31:45,970 Initializing key cache with capacity of 46 MBs.
 INFO 00:31:45,983 Scheduling key cache save to each 14400 seconds (going to save all keys).
 INFO 00:31:45,985 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO 00:31:45,993 Scheduling row cache save to each 0 seconds (going to save all keys).
 INFO 00:31:46,186 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-17 (383 bytes)
 INFO 00:31:46,261 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-18 (169 bytes)
 INFO 00:31:46,325 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ia-22 (4841 bytes)
 INFO 00:31:46,372 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-20 (3898 bytes)
 INFO 00:31:46,377 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-21 (300 bytes)
 INFO 00:31:46,388 Opening /var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-ia-1 (65 bytes)
 INFO 00:31:46,398 Opening /var/lib/cassandra/data/system/peers/system-peers-ia-6 (226 bytes)
 INFO 00:31:46,420 Opening /var/lib/cassandra/data/system/local/system-local-ia-98 (475 bytes)
 INFO 00:31:47,041 Opening /var/lib/cassandra/data/Twissandra/User/Twissandra-User-ia-1 (90 bytes)
 INFO 00:31:47,139 Opening /var/lib/cassandra/data/Twissandra/User/Twissandra-User.User_age_idx-ia-1 (47 bytes)
 INFO 00:31:47,194 Opening /var/lib/cassandra/data/system_auth/users/system_auth-users-ia-1 (72 bytes)
 INFO 00:31:47,247 completed pre-loading (7 keys) key cache.
 INFO 00:31:47,307 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522838.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522839.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522840.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522841.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522842.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522843.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522844.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522845.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522846.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522847.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522848.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522849.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522850.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522851.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522852.log, /var/lib/cassandra/commitlog/CommitLog-2-1357232522853.log
 INFO 00:31:47,348 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522838.log
 INFO 00:31:47,541 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522838.log
 INFO 00:31:47,542 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522839.log
 INFO 00:31:47,542 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522839.log
 INFO 00:31:47,543 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522840.log
 INFO 00:31:47,549 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522840.log
 INFO 00:31:47,552 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522841.log
 INFO 00:31:47,555 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522841.log
 INFO 00:31:47,555 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522842.log
 INFO 00:31:47,562 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522842.log
 INFO 00:31:47,562 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522843.log
 INFO 00:31:47,562 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522843.log
 INFO 00:31:47,569 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522844.log
 INFO 00:31:47,570 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522844.log
 INFO 00:31:47,571 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522845.log
 INFO 00:31:47,571 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522845.log
 INFO 00:31:47,573 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522846.log
 INFO 00:31:47,574 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522846.log
 INFO 00:31:47,574 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522847.log
 INFO 00:31:47,581 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522847.log
 INFO 00:31:47,581 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522848.log
 INFO 00:31:47,590 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522848.log
 INFO 00:31:47,590 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522849.log
 INFO 00:31:47,590 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522849.log
 INFO 00:31:47,595 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522850.log
 INFO 00:31:47,596 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522850.log
 INFO 00:31:47,596 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522851.log
 INFO 00:31:47,601 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522851.log
 INFO 00:31:47,602 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522852.log
 INFO 00:31:47,610 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522852.log
 INFO 00:31:47,611 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357232522853.log
 INFO 00:31:47,614 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357232522853.log
 INFO 00:31:47,635 Enqueuing flush of Memtable-local@1159139518(52/52 serialized/live bytes, 2 ops)
 INFO 00:31:47,646 Enqueuing flush of Memtable-peers@946144838(155/155 serialized/live bytes, 10 ops)
 INFO 00:31:47,653 Writing Memtable-local@1159139518(52/52 serialized/live bytes, 2 ops)
 INFO 00:31:47,676 Enqueuing flush of Memtable-schema_keyspaces@833041663(389/389 serialized/live bytes, 12 ops)
 INFO 00:31:47,677 Enqueuing flush of Memtable-schema_columns@1485335248(21317/21317 serialized/live bytes, 328 ops)
 INFO 00:31:47,682 Enqueuing flush of Memtable-schema_columnfamilies@1520159108(20754/20754 serialized/live bytes, 344 ops)
 INFO 00:31:47,723 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-99-Data.db (85 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=142)
 INFO 00:31:47,740 Writing Memtable-peers@946144838(155/155 serialized/live bytes, 10 ops)
 INFO 00:31:47,771 Completed flushing /var/lib/cassandra/data/system/peers/system-peers-ia-7-Data.db (191 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=142)
 INFO 00:31:47,773 Writing Memtable-schema_keyspaces@833041663(389/389 serialized/live bytes, 12 ops)
 INFO 00:31:47,805 Completed flushing /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-19-Data.db (259 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=142)
 INFO 00:31:47,808 Writing Memtable-schema_columns@1485335248(21317/21317 serialized/live bytes, 328 ops)
 INFO 00:31:47,856 Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-22-Data.db (3749 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=142)
 INFO 00:31:47,859 Writing Memtable-schema_columnfamilies@1520159108(20754/20754 serialized/live bytes, 344 ops)
 INFO 00:31:47,903 Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ia-23-Data.db (4424 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=142)
 INFO 00:31:47,905 Log replay complete, 23 replayed mutations
 INFO 00:31:48,262 Cassandra version: 1.2.0
 INFO 00:31:48,263 Thrift API version: 19.35.0
 INFO 00:31:48,263 CQL supported versions: 2.0.0,3.0.0 (default: 3.0.0)
 INFO 00:31:48,334 Loading persisted ring state
 INFO 00:31:48,342 Starting up server gossip
 INFO 00:31:48,372 Enqueuing flush of Memtable-local@1854861687(251/251 serialized/live bytes, 9 ops)
 INFO 00:31:48,373 Writing Memtable-local@1854861687(251/251 serialized/live bytes, 9 ops)
 INFO 00:31:48,451 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-100-Data.db (241 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=48998)
 INFO 00:31:48,572 Starting Messaging Service on port 7000
 INFO 00:31:48,615 Using saved token [3339682839163339854]
 INFO 00:31:48,618 Enqueuing flush of Memtable-local@1337300467(84/84 serialized/live bytes, 4 ops)
 INFO 00:31:48,620 Writing Memtable-local@1337300467(84/84 serialized/live bytes, 4 ops)
 INFO 00:31:48,652 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-101-Data.db (121 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=49273)
 INFO 00:31:48,663 Enqueuing flush of Memtable-local@2005580923(50/50 serialized/live bytes, 2 ops)
 INFO 00:31:48,664 Writing Memtable-local@2005580923(50/50 serialized/live bytes, 2 ops)
 INFO 00:31:48,675 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-101-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-100-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-99-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-98-Data.db')]
 INFO 00:31:48,701 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-102-Data.db (110 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=49447)
 INFO 00:31:48,723 Node centos/192.168.111.80 state jump to normal
 INFO 00:31:48,759 Enqueuing flush of Memtable-local@960861909(50/50 serialized/live bytes, 2 ops)
 INFO 00:31:48,760 Writing Memtable-local@960861909(50/50 serialized/live bytes, 2 ops)
 INFO 00:31:48,791 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-103-Data.db (110 bytes) for commitlog position ReplayPosition(segmentId=1357234306996, position=49621)
 INFO 00:31:48,801 Startup completed! Now serving reads.
 INFO 00:31:48,875 Not starting native transport as requested. Use JMX (StorageService->startNativeTransport()) to start it
 INFO 00:31:48,879 Binding thrift service to centos/192.168.111.80:9160
 INFO 00:31:48,947 Compacted to [/var/lib/cassandra/data/system/local/system-local-ia-104-Data.db,].  922 to 427 (~46% of original) bytes for 1 keys at 0.001649MB/s.  Time: 247ms.
 INFO 00:31:48,948 Using TFramedTransport with a max frame size of 15728640 bytes.
 INFO 00:31:48,957 Using synchronous/threadpool thrift server on centos : 9160
 INFO 00:31:48,958 Listening for thrift clients...

[surachart@centos apache-cassandra-1.2.0]$

On "centostest1": (the rest of the node)
- Checked hosts file.

[surachart@centostest1 apache-cassandra-1.2.0]$
[surachart@centostest1 apache-cassandra-1.2.0]$  cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.111.81  centostest1 centostest1.surachartopun.com
192.168.111.80  centos centos.surachartopun.com

- Changed from "localhost to hostname (centostest1) in "conf/cassandra.yaml" file.
Note:  Seeds section of the configuration... use "centos" (firstnode)

[surachart@centostest1 apache-cassandra-1.2.0]$  grep centos conf/cassandra.yaml
          - seeds: "centos"
listen_address: centostest1
rpc_address: centostest1

- Changed configure in  "conf/cassandra-rackdc.properties" and "conf/cassandra-topology.properties" files.

[surachart@centostest1 apache-cassandra-1.2.0]$ grep \= conf/cassandra-rackdc.properties
dc=DC1
rack=RAC2
[surachart@centostest1 apache-cassandra-1.2.0]$ grep \= conf/cassandra-topology.properties |grep -v \#
192.168.111.80=DC1:RAC1
192.168.111.81=DC1:RAC2
default=DC1:r1

- Started it.

[surachart@centostest1 apache-cassandra-1.2.0]$  bin/cassandra -f &
[1] 12118
[surachart@centostest1 apache-cassandra-1.2.0]$ xss =  -ea -javaagent:bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms938M -Xmx938M -Xmn100M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
 INFO 00:39:35,425 Logging initialized
 INFO 00:39:35,449 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_38
 INFO 00:39:35,449 Heap size: 973078528/973078528
 INFO 00:39:35,450 Classpath: bin/../conf:bin/../build/classes/main:bin/../build/classes/thrift:bin/../lib/antlr-3.2.jar:bin/../lib/apache-cassandra-1.2.0.jar:bin/../lib/apache-cassandra-clientutil-1.2.0.jar:bin/../lib/apache-cassandra-thrift-1.2.0.jar:bin/../lib/avro-1.4.0-fixes.jar:bin/../lib/avro-1.4.0-sources-fixes.jar:bin/../lib/commons-cli-1.1.jar:bin/../lib/commons-codec-1.2.jar:bin/../lib/commons-lang-2.6.jar:bin/../lib/compress-lzf-0.8.4.jar:bin/../lib/concurrentlinkedhashmap-lru-1.3.jar:bin/../lib/guava-13.0.1.jar:bin/../lib/high-scale-lib-1.1.2.jar:bin/../lib/jackson-core-asl-1.9.2.jar:bin/../lib/jackson-mapper-asl-1.9.2.jar:bin/../lib/jamm-0.2.5.jar:bin/../lib/jline-1.0.jar:bin/../lib/json-simple-1.1.jar:bin/../lib/libthrift-0.7.0.jar:bin/../lib/log4j-1.2.16.jar:bin/../lib/metrics-core-2.0.3.jar:bin/../lib/netty-3.5.9.Final.jar:bin/../lib/servlet-api-2.5-20081211.jar:bin/../lib/slf4j-api-1.7.2.jar:bin/../lib/slf4j-log4j12-1.7.2.jar:bin/../lib/snakeyaml-1.6.jar:bin/../lib/snappy-java-1.0.4.1.jar:bin/../lib/snaptree-0.1.jar:bin/../lib/jamm-0.2.5.jar
 INFO 00:39:35,452 JNA not found. Native methods will be disabled.
 INFO 00:39:35,470 Loading settings from file:/home/surachart/apache-cassandra-1.2.0/conf/cassandra.yaml
 INFO 00:39:36,083 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap
 INFO 00:39:36,084 disk_failure_policy is stop
 INFO 00:39:36,089 Global memtable threshold is enabled at 309MB
 INFO 00:39:37,047 Initializing key cache with capacity of 46 MBs.
 INFO 00:39:37,059 Scheduling key cache save to each 14400 seconds (going to save all keys).
 INFO 00:39:37,060 Initializing row cache with capacity of 0 MBs and provider org.apache.cassandra.cache.SerializingCacheProvider
 INFO 00:39:37,069 Scheduling row cache save to each 0 seconds (going to save all keys).
 INFO 00:39:37,305 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-14 (382 bytes)
 INFO 00:39:37,377 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-16 (169 bytes)
 INFO 00:39:37,396 Opening /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-15 (293 bytes)
 INFO 00:39:37,465 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ia-15 (4705 bytes)
 INFO 00:39:37,471 Opening /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ia-16 (658 bytes)
 INFO 00:39:37,496 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-16 (300 bytes)
 INFO 00:39:37,499 Opening /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-15 (3902 bytes)
 INFO 00:39:37,511 Opening /var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-ia-1 (65 bytes)
 INFO 00:39:37,536 Opening /var/lib/cassandra/data/system/peers/system-peers-ia-5 (252 bytes)
 INFO 00:39:37,560 Opening /var/lib/cassandra/data/system/local/system-local-ia-89 (430 bytes)
 INFO 00:39:37,565 Opening /var/lib/cassandra/data/system/local/system-local-ia-87 (112 bytes)
 INFO 00:39:37,571 Opening /var/lib/cassandra/data/system/local/system-local-ia-88 (112 bytes)
 INFO 00:39:38,101 Opening /var/lib/cassandra/data/schema1/users/schema1-users-ia-1 (148 bytes)
 INFO 00:39:38,150 Opening /var/lib/cassandra/data/system_auth/users/system_auth-users-ia-1 (72 bytes)
 INFO 00:39:38,395 completed pre-loading (10 keys) key cache.
 INFO 00:39:38,450 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545984.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545985.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545986.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545987.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545988.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545989.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545990.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545991.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545992.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545993.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545994.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545995.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545996.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545997.log, /var/lib/cassandra/commitlog/CommitLog-2-1357234545998.log
 INFO 00:39:38,480 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545984.log
 INFO 00:39:38,646 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545984.log
 INFO 00:39:38,647 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545985.log
 INFO 00:39:38,651 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545985.log
 INFO 00:39:38,651 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545986.log
 INFO 00:39:38,657 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545986.log
 INFO 00:39:38,657 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545987.log
 INFO 00:39:38,660 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545987.log
 INFO 00:39:38,660 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545988.log
 INFO 00:39:38,661 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545988.log
 INFO 00:39:38,661 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545989.log
 INFO 00:39:38,666 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545989.log
 INFO 00:39:38,666 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545990.log
 INFO 00:39:38,667 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545990.log
 INFO 00:39:38,667 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545991.log
 INFO 00:39:38,668 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545991.log
 INFO 00:39:38,673 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545992.log
 INFO 00:39:38,677 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545992.log
 INFO 00:39:38,677 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545993.log
 INFO 00:39:38,680 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545993.log
 INFO 00:39:38,681 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545994.log
 INFO 00:39:38,686 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545994.log
 INFO 00:39:38,687 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545995.log
 INFO 00:39:38,689 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545995.log
 INFO 00:39:38,690 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545996.log
 INFO 00:39:38,693 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545996.log
 INFO 00:39:38,694 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545997.log
 INFO 00:39:38,697 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545997.log
 INFO 00:39:38,697 Replaying /var/lib/cassandra/commitlog/CommitLog-2-1357234545998.log
 INFO 00:39:38,698 Finished reading /var/lib/cassandra/commitlog/CommitLog-2-1357234545998.log
 INFO 00:39:38,724 Enqueuing flush of Memtable-local@583175351(52/52 serialized/live bytes, 2 ops)
 INFO 00:39:38,737 Writing Memtable-local@583175351(52/52 serialized/live bytes, 2 ops)
 INFO 00:39:38,750 Enqueuing flush of Memtable-schema_keyspaces@72177013(389/389 serialized/live bytes, 12 ops)
 INFO 00:39:38,752 Enqueuing flush of Memtable-schema_columns@1898280583(21317/21317 serialized/live bytes, 328 ops)
 INFO 00:39:38,753 Enqueuing flush of Memtable-schema_columnfamilies@1384698649(20754/20754 serialized/live bytes, 344 ops)
 INFO 00:39:38,838 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-90-Data.db (85 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=142)
 INFO 00:39:38,861 Writing Memtable-schema_keyspaces@72177013(389/389 serialized/live bytes, 12 ops)
 INFO 00:39:38,897 Completed flushing /var/lib/cassandra/data/system/schema_keyspaces/system-schema_keyspaces-ia-17-Data.db (259 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=142)
 INFO 00:39:38,900 Writing Memtable-schema_columns@1898280583(21317/21317 serialized/live bytes, 328 ops)
 INFO 00:39:38,944 Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-ia-17-Data.db (3747 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=142)
 INFO 00:39:38,946 Writing Memtable-schema_columnfamilies@1384698649(20754/20754 serialized/live bytes, 344 ops)
 INFO 00:39:38,987 Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-ia-17-Data.db (4424 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=142)
 INFO 00:39:38,989 Log replay complete, 18 replayed mutations
 INFO 00:39:39,362 Cassandra version: 1.2.0
 INFO 00:39:39,363 Thrift API version: 19.35.0
 INFO 00:39:39,363 CQL supported versions: 2.0.0,3.0.0 (default: 3.0.0)
 INFO 00:39:39,426 Loading persisted ring state
 INFO 00:39:39,433 Starting up server gossip
 INFO 00:39:39,457 Enqueuing flush of Memtable-local@801042860(251/251 serialized/live bytes, 9 ops)
 INFO 00:39:39,460 Writing Memtable-local@801042860(251/251 serialized/live bytes, 9 ops)
 INFO 00:39:39,546 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-91-Data.db (237 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=48998)
 INFO 00:39:39,586 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-91-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-87-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-90-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-89-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-88-Data.db')]
 INFO 00:39:39,684 Starting Messaging Service on port 7000
 INFO 00:39:39,714 Using saved token [-4649311623768988171]
 INFO 00:39:39,719 Enqueuing flush of Memtable-local@434033865(84/84 serialized/live bytes, 4 ops)
 INFO 00:39:39,720 Writing Memtable-local@434033865(84/84 serialized/live bytes, 4 ops)
 INFO 00:39:39,759 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-92-Data.db (120 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=49273)
 INFO 00:39:39,765 Enqueuing flush of Memtable-local@1294965028(51/51 serialized/live bytes, 2 ops)
 INFO 00:39:39,766 Writing Memtable-local@1294965028(51/51 serialized/live bytes, 2 ops)
 INFO 00:39:39,802 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-93-Data.db (112 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=49448)
 INFO 00:39:39,829 Node centostest1/192.168.111.81 state jump to normal
 INFO 00:39:39,847 Enqueuing flush of Memtable-local@1132676771(51/51 serialized/live bytes, 2 ops)
 INFO 00:39:39,849 Writing Memtable-local@1132676771(51/51 serialized/live bytes, 2 ops)
 INFO 00:39:39,885 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-94-Data.db (112 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=49623)
 INFO 00:39:39,892 Startup completed! Now serving reads.
 INFO 00:39:39,987 Not starting native transport as requested. Use JMX (StorageService->startNativeTransport()) to start it
 INFO 00:39:39,996 Binding thrift service to centostest1/192.168.111.81:9160
 INFO 00:39:40,076 Using TFramedTransport with a max frame size of 15728640 bytes.
 INFO 00:39:40,077 Compacted to [/var/lib/cassandra/data/system/local/system-local-ia-95-Data.db,].  976 to 436 (~44% of original) bytes for 1 keys at 0.000904MB/s.  Time: 460ms.
 INFO 00:39:40,087 Using synchronous/threadpool thrift server on centostest1 : 9160
 INFO 00:39:40,088 Listening for thrift clients...
 INFO 00:39:40,749 Node /192.168.111.80 is now part of the cluster
 INFO 00:39:40,760 InetAddress /192.168.111.80 is now UP
 INFO 00:39:40,823 Enqueuing flush of Memtable-peers@2104483009(81/81 serialized/live bytes, 4 ops)
 INFO 00:39:40,827 Writing Memtable-peers@2104483009(81/81 serialized/live bytes, 4 ops)
 INFO 00:39:40,865 Completed flushing /var/lib/cassandra/data/system/peers/system-peers-ia-6-Data.db (145 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=49927)
 INFO 00:39:40,885 Enqueuing flush of Memtable-local@1349072889(51/51 serialized/live bytes, 2 ops)
 INFO 00:39:40,890 Writing Memtable-local@1349072889(51/51 serialized/live bytes, 2 ops)
 INFO 00:39:40,924 Completed flushing /var/lib/cassandra/data/system/local/system-local-ia-96-Data.db (112 bytes) for commitlog position ReplayPosition(segmentId=1357234778161, position=50102)
 INFO 00:39:40,928 Compacting [SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-95-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-96-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-92-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-93-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/local/system-local-ia-94-Data.db')]
 INFO 00:39:40,981 Compacted to [/var/lib/cassandra/data/system/local/system-local-ia-97-Data.db,].  892 to 457 (~51% of original) bytes for 1 keys at 0.009685MB/s.  Time: 45ms.

[surachart@centostest1 apache-cassandra-1.2.0]$

Test - Created schema & table by using "cqlsh".
On "centos" (first node):

[surachart@centos apache-cassandra-1.2.0]$ cp conf/cqlshrc.sample  ~/.cqlshrc
[surachart@centos apache-cassandra-1.2.0]$ vi ~/.cqlshrc
[surachart@centos apache-cassandra-1.2.0]$ diff conf/cqlshrc.sample  ~/.cqlshrc
32c32
< hostname = 127.0.0.1
---
> hostname = centos
[surachart@centos apache-cassandra-1.2.0]$ bin/cqlsh
Connected to Test Cluster at centos:9160.
[cqlsh 2.3.0 | Cassandra 1.2.0 | CQL spec 3.0.0 | Thrift protocol 19.35.0]
Use HELP for help.
cqlsh>
cqlsh> CREATE SCHEMA schema1
   ...          WITH replication = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
cqlsh> USE schema1;
cqlsh:schema1> show host;
Connected to Test Cluster at centos:9160.
cqlsh:schema1> CREATE TABLE users (
           ...                    user_id varchar PRIMARY KEY,
           ...                    first varchar,
           ...                    last varchar,
           ...                    age int
           ...                  );
cqlsh:schema1> INSERT INTO users (user_id, first, last, age)
           ...                  VALUES ('jsmith', 'John', 'Smith', 42);
cqlsh:schema1> SELECT * FROM users;

 user_id | age | first | last
---------+-----+-------+-------
  jsmith |  42 |  John | Smith

cqlsh:schema1>

On "centostest1" (the rest of the node):

[surachart@centostest1 apache-cassandra-1.2.0]$ cp conf/cqlshrc.sample  ~/.cqlshrc
[surachart@centostest1 apache-cassandra-1.2.0]$ vi ~/.cqlshrc
[surachart@centostest1 apache-cassandra-1.2.0]$ diff conf/cqlshrc.sample ~/.cqlshrc
32c32
< hostname = 127.0.0.1
---
> hostname = centostest1
[surachart@centostest1 apache-cassandra-1.2.0]$ bin/nodetool -host centos ring
Note: Ownership information does not include topology; for complete information, specify a keyspace

Datacenter: datacenter1
==========
Address         Rack        Status State   Load            Owns                Token
                                                                               3339682839163339854
192.168.111.81  rack1       Up     Normal  67.57 KB        56.69%              -4649311623768988171
192.168.111.80  rack1       Up     Normal  85.71 KB        43.31%              3339682839163339854

[surachart@centostest1 apache-cassandra-1.2.0]$  bin/cqlsh
Connected to Test Cluster at centostest1:9160.
[cqlsh 2.3.0 | Cassandra 1.2.0 | CQL spec 3.0.0 | Thrift protocol 19.35.0]
Use HELP for help.
cqlsh> use schema1
   ... ;
cqlsh:schema1> show host;
Connected to Test Cluster at centos:9160.

cqlsh:schema1> select * from users;

 user_id | age | first | last
---------+-----+-------+-------
  jsmith |  42 |  John | Smith

It looks like ... Cool!