Learned a bit - ACCUMULO

Today. I have installed Hadoop distribution from Bigtop 0.6.0 on CentOS 6. It's very cool and easy for installation. After installation, I had a chance to read a bit about Apache Accumulo, it's a highly scalable structured store based on Google's Big Table. Accumulo is written in Java and operates over the HDFS and it provides a richer data model than simple key-value stores, but not a fully relational database... Read More.

So, I downloaded rpm package and installed it.

Note: I tested it on standalone and got some warning -Forgot it,  I just wanted to learn command and etc... for now.

I set profile on CentOS system.

export ACCUMULO_HOME=/opt/accumulo/accumulo-1.5.0
export HADOOP_HOME=/usr
export JAVA_HOME=/usr
export ZOOKEEPER_HOME=/usr
export HADOOP_CONF_DIR=/etc/hadoop/conf

Then, created  /accumulo in HDFS by command: "hadoop fs -mkdir /accumulo" and "hadoop fs -chmod 777 /accumulo". After that, performed by executing $ACCUMULO_HOME/bin/accumulo init and Started Accumulo.

[root@centos01 bin]# pwd
/opt/accumulo/accumulo-1.5.0/bin
[root@centos01 bin]# cp accumulo accumulo.orig
[root@centos01 bin]# diff accumulo.orig accumulo
85c85
< CLASSPATH=${XML_FILES}:${START_JAR}:${LOG4J_JAR}
---
> CLASSPATH=${XML_FILES}:${START_JAR}:${LOG4J_JAR}:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/client/*

[root@centos01 bin]# ./accumulo init
2013-11-08 20:50:10,299 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-11-08 20:50:11,700 [util.Initialize] INFO : Hadoop Filesystem is hdfs://localhost:8020
2013-11-08 20:50:11,700 [util.Initialize] INFO : Accumulo data dir is /accumulo
2013-11-08 20:50:11,727 [util.Initialize] INFO : Zookeeper server is localhost:2181
2013-11-08 20:50:11,727 [util.Initialize] INFO : Checking if Zookeeper is available. If this hangs, then you need to make sure zookeeper is running


Warning!!! Your instance secret is still set to the default, this is not secure. We highly recommend you change it.


You can change the instance secret in accumulo by using:
   bin/accumulo org.apache.accumulo.server.util.ChangeSecret oldPassword newPassword.
You will also need to edit your secret in your configuration file by adding the property instance.secret to your conf/accumulo-site.xml. Without this accumulo will not operate correctly
Instance name : opun
Instance name "opun" exists. Delete existing entry from zookeeper? [Y/N] : y
Enter initial password for root (this may not be applicable for your security setup): ********
Confirm initial password for root: ********
2013-11-08 20:50:22,967 [conf.Configuration] WARN : dfs.replication.min is deprecated. Instead, use dfs.namenode.replication.min
2013-11-08 20:50:23,146 [conf.Configuration] WARN : dfs.block.size is deprecated. Instead, use dfs.blocksize
2013-11-08 20:50:23,692 [master.Master] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKAuthorizor
2013-11-08 20:50:23,731 [master.Master] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKAuthenticator
2013-11-08 20:50:23,737 [master.Master] INFO : Loaded class : org.apache.accumulo.server.security.handler.ZKPermHandler
2013-11-08 20:50:23,921 [security.AuditedSecurityOperation] INFO : Initialized root user with username: root at the request of user !SYSTEM
[root@centos01 bin]#
[root@centos01 bin]#
[root@centos01 bin]#
[root@centos01 bin]# ./start-all.sh
ls: cannot access zookeeper-[0-9]*.jar: No such file or directory
WARN : Using Zookeeper .  Use version 3.3.0 or greater to avoid zookeeper deadlock bug.
Starting monitor on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting tablet servers .... done
Starting tablet server on localhost
WARN : Max files open on localhost is 1024, recommend 65536
2013-11-08 20:50:38,464 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2013-11-08 20:50:43,210 [server.Accumulo] INFO : Attempting to talk to zookeeper
2013-11-08 20:50:43,528 [server.Accumulo] INFO : Zookeeper connected and initialized, attemping to talk to HDFS
2013-11-08 20:50:44,026 [server.Accumulo] INFO : Connected to HDFS
Starting master on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting garbage collector on localhost
WARN : Max files open on localhost is 1024, recommend 65536
Starting tracer on localhost
WARN : Max files open on localhost is 1024, recommend 65536
[root@centos01 bin]#
[root@centos01 bin]# ./accumulo monitor
2013-11-08 20:52:19,352 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

 In HDFS,

-bash-4.1$ hadoop fs -ls /
Found 6 items
drwxrwxrwx   - hdfs  supergroup          0 2013-11-08 20:51 /accumulo
drwxrwxrwx   - hdfs  supergroup          0 2013-11-08 19:55 /benchmarks
drwxr-xr-x   - hbase hbase               0 2013-11-08 19:55 /hbase
drwxrwxrwt   - hdfs  supergroup          0 2013-11-08 19:54 /tmp
drwxr-xr-x   - hdfs  supergroup          0 2013-11-08 19:59 /user
drwxr-xr-x   - hdfs  supergroup          0 2013-11-08 19:54 /var
-bash-4.1$ hadoop fs -ls /accumulo
Found 4 items
drwxr-xr-x   - root supergroup          0 2013-11-08 20:50 /accumulo/instance_id
drwxr-xr-x   - root supergroup          0 2013-11-08 21:47 /accumulo/tables
drwxr-xr-x   - root supergroup          0 2013-11-08 20:50 /accumulo/version
drwxrwxrwx   - root supergroup          0 2013-11-08 20:51 /accumulo/wal
-bash-4.1$

After Accumulo started, then tested to use some command.

[root@centos01 bin]# ./accumulo shell -u root
2013-11-08 20:51:53,536 [util.NativeCodeLoader] WARN : Unable to load native-had                                                                oop library for your platform... using builtin-java classes where applicable
Password: ********

Shell - Apache Accumulo Interactive Shell
-
- version: 1.5.0
- instance name: opun
- instance id: e799232d-5638-4122-b4aa-bdfe81eff3f7
-
- type 'help' for a list of available commands
-
root@opun> tables
!METADATA
root@opun> createtable test
root@opun test> tables
!METADATA
test
root@opun test> getauths

root@opun test> setauths -s billing,inventory
root@opun test> getauths
billing,inventory
root@opun test> insert -l billing "bob jones" contact phone 555-1212
root@opun test> insert -l billing "bob jones" contact address "123 anystreet"
root@opun test> insert -l billing "bob jones" contact city "anytown"
root@opun test> insert -l billing|inventory "bob jones" purchases sneakers "$60"
root@opun test> insert -l billing "fred smith" contact address "444 main street"
root@opun test> insert -l billing "fred smith" contact city "othertown"
root@opun test> insert -l billing|inventory "fred smith" purchases glasses "$30"
root@opun test> insert -l billing|inventory "fred smith" purchases hat "$20"
root@opun test> scan
bob jones contact:address [billing]    123 anystreet
bob jones contact:city [billing]    anytown
bob jones contact:phone [billing]    555-1212
bob jones purchases:sneakers [billing|inventory]    $60
fred smith contact:address [billing]    444 main street
fred smith contact:city [billing]    othertown
fred smith purchases:glasses [billing|inventory]    $30
fred smith purchases:hat [billing|inventory]    $20
root@opun test> scan -s billing
bob jones contact:address [billing]    123 anystreet
bob jones contact:city [billing]    anytown
bob jones contact:phone [billing]    555-1212
bob jones purchases:sneakers [billing|inventory]    $60
fred smith contact:address [billing]    444 main street
fred smith contact:city [billing]    othertown
fred smith purchases:glasses [billing|inventory]    $30
fred smith purchases:hat [billing|inventory]    $20
root@opun test> scan -b "fred smith"
fred smith contact:address [billing]    444 main street
fred smith contact:city [billing]    othertown
fred smith purchases:glasses [billing|inventory]    $30
fred smith purchases:hat [billing|inventory]    $20
root@opun test> insert -l billing|inventory "fred smith" purchases hat "$25"
root@opun test> scan -b "fred smith"
fred smith contact:address [billing]    444 main street
fred smith contact:city [billing]    othertown
fred smith purchases:glasses [billing|inventory]    $30
fred smith purchases:hat [billing|inventory]    $25
root@opun test> scan -b "fred smith" -c purchases
fred smith purchases:glasses [billing|inventory]    $30
fred smith purchases:hat [billing|inventory]    $25
root@opun test> scan -b "fred smith" -c purchases:glasses
fred smith purchases:glasses [billing|inventory]    $30
root@opun test> scan -c purchases
bob jones purchases:sneakers [billing|inventory]    $60
fred smith purchases:glasses [billing|inventory]    $30
fred smith purchases:hat [billing|inventory]    $25

root@opun test>  config -t test
---------+---------------------------------------------+----------------------------------------------------------------------------------------
SCOPE    | NAME                                        | VALUE
---------+---------------------------------------------+----------------------------------------------------------------------------------------
default  | table.balancer ............................ | org.apache.accumulo.server.master.balancer.DefaultLoadBalancer
default  | table.bloom.enabled ....................... | false
default  | table.bloom.error.rate .................... | 0.5%
default  | table.bloom.hash.type ..................... | murmur
default  | table.bloom.key.functor ................... | org.apache.accumulo.core.file.keyfunctor.RowFunctor
default  | table.bloom.load.threshold ................ | 1
default  | table.bloom.size .......................... | 1048576
default  | table.cache.block.enable .................. | false
default  | table.cache.index.enable .................. | true
default  | table.classpath.context ................... |
default  | table.compaction.major.everything.idle .... | 1h
default  | table.compaction.major.ratio .............. | 3
default  | table.compaction.minor.idle ............... | 5m
default  | table.compaction.minor.logs.threshold ..... | 3
default  | table.failures.ignore ..................... | false
default  | table.file.blocksize ...................... | 0B
default  | table.file.compress.blocksize ............. | 100K
default  | table.file.compress.blocksize.index ....... | 128K
default  | table.file.compress.type .................. | gz
default  | table.file.max ............................ | 15
default  | table.file.replication .................... | 0
default  | table.file.type ........................... | rf
default  | table.formatter ........................... | org.apache.accumulo.core.util.format.DefaultFormatter
default  | table.groups.enabled ...................... |
default  | table.interepreter ........................ | org.apache.accumulo.core.util.interpret.DefaultScanInterpreter
table    | table.iterator.majc.vers .................. | 20,org.apache.accumulo.core.iterators.user.VersioningIterator
table    | table.iterator.majc.vers.opt.maxVersions .. | 1
table    | table.iterator.minc.vers .................. | 20,org.apache.accumulo.core.iterators.user.VersioningIterator
table    | table.iterator.minc.vers.opt.maxVersions .. | 1
table    | table.iterator.scan.vers .................. | 20,org.apache.accumulo.core.iterators.user.VersioningIterator
table    | table.iterator.scan.vers.opt.maxVersions .. | 1
default  | table.scan.max.memory ..................... | 512K
default  | table.security.scan.visibility.default .... |
default  | table.split.threshold ..................... | 1G
---------------------------------------------------- hit any key to continue or 'q' to quit ----------------------------------------------------
default  | table.walog.enabled ....................... | true
---------+---------------------------------------------+----------------------------------------------------------------------------------------
root@opun test>

Connect http://localhost:50095/

Great! Tested to create user and grant some permission.

root@opun> createuser surachart
Enter new password for 'surachart': *********
Please confirm new password for 'surachart': *********
root@opun> systempermissions
System.GRANT
System.CREATE_TABLE
System.DROP_TABLE
System.ALTER_TABLE
System.CREATE_USER
System.DROP_USER
System.ALTER_USER
System.SYSTEM
root@opun>
root@opun>
root@opun> tablepermissions
Table.READ
Table.WRITE
Table.BULK_IMPORT
Table.ALTER_TABLE
Table.GRANT
Table.DROP_TABLE
root@opun> grant System.CREATE_TABLE -s -u surachart
root@opun>
root@opun>
[root@centos01 bin]# ./accumulo shell -u surachart
2013-11-08 22:32:07,960 [util.NativeCodeLoader] WARN : Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Password: *********

Shell - Apache Accumulo Interactive Shell
-
- version: 1.5.0
- instance name: opun
- instance id: e799232d-5638-4122-b4aa-bdfe81eff3f7
-
- type 'help' for a list of available commands
-
surachart@opun> createtable mytest
surachart@opun mytest> table mytest
surachart@opun mytest> config -t  mytest
---------+---------------------------------------------+----------------------------------------------------------------------------------------
SCOPE    | NAME                                        | VALUE
---------+---------------------------------------------+----------------------------------------------------------------------------------------
default  | table.balancer ............................ | org.apache.accumulo.server.master.balancer.DefaultLoadBalancer
default  | table.bloom.enabled ....................... | false
default  | table.bloom.error.rate .................... | 0.5%
default  | table.bloom.hash.type ..................... | murmur
default  | table.bloom.key.functor ................... | org.apache.accumulo.core.file.keyfunctor.RowFunctor
default  | table.bloom.load.threshold ................ | 1
default  | table.bloom.size .......................... | 1048576
default  | table.cache.block.enable .................. | false
default  | table.cache.index.enable .................. | true
default  | table.classpath.context ................... |
default  | table.compaction.major.everything.idle .... | 1h
default  | table.compaction.major.ratio .............. | 3
default  | table.compaction.minor.idle ............... | 5m
default  | table.compaction.minor.logs.threshold ..... | 3
default  | table.failures.ignore ..................... | false
default  | table.file.blocksize ...................... | 0B
default  | table.file.compress.blocksize ............. | 100K
default  | table.file.compress.blocksize.index ....... | 128K
default  | table.file.compress.type .................. | gz
default  | table.file.max ............................ | 15
default  | table.file.replication .................... | 0
default  | table.file.type ........................... | rf
default  | table.formatter ........................... | org.apache.accumulo.core.util.format.DefaultFormatter
default  | table.groups.enabled ...................... |
default  | table.interepreter ........................ | org.apache.accumulo.core.util.interpret.DefaultScanInterpreter
table    | table.iterator.majc.vers .................. | 20,org.apache.accumulo.core.iterators.user.VersioningIterator
table    | table.iterator.majc.vers.opt.maxVersions .. | 1
table    | table.iterator.minc.vers .................. | 20,org.apache.accumulo.core.iterators.user.VersioningIterator
table    | table.iterator.minc.vers.opt.maxVersions .. | 1
table    | table.iterator.scan.vers .................. | 20,org.apache.accumulo.core.iterators.user.VersioningIterator
table    | table.iterator.scan.vers.opt.maxVersions .. | 1
default  | table.scan.max.memory ..................... | 512K
default  | table.security.scan.visibility.default .... |
default  | table.split.threshold ..................... | 1G
---------------------------------------------------- hit any key to continue or 'q' to quit ----------------------------------------------------
default  | table.walog.enabled ....................... | true
---------+---------------------------------------------+----------------------------------------------------------------------------------------
surachart@opun mytest>

Referenced Links:
http://accumulo.apache.org/
http://shop.oreilly.com/product/0636920032304.do