rsyslog: Send logs to Flume

Good day for learning something new. After read Flume book, that something popped up in my head. Wanted to test "rsyslog" => Flume => HDFS. As we know, forwarding log to other systems. We can set rsyslog:

*.* @YOURSERVERADDRESS:YOURSERVERPORT ## for UDP
*.* @@YOURSERVERADDRESS:YOURSERVERPORT ## for TCP

For rsyslog:

[root@centos01 ~]# grep centos /etc/rsyslog.conf
*.* @centos01:7777

Came back to Flume, I used Simple Example for reference and changed a bit. Because I wanted it write to HDFS.

[root@centos01 ~]# grep "^FLUME_AGENT_NAME\="  /etc/default/flume-agent
FLUME_AGENT_NAME=a1
[root@centos01 ~]# cat /etc/flume/conf/flume.conf
# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
#a1.sources.r1.type = netcat
a1.sources.r1.type = syslogudp
a1.sources.r1.bind = 0.0.0.0
a1.sources.r1.port = 7777
# Describe the sink
#a1.sinks.k1.type = logger
a1.sinks.k1.type = hdfs
a1.sinks.k1.hdfs.path = hdfs://localhost:8020/user/flume/syslog/%Y/%m/%d/%H/
a1.sinks.k1.hdfs.fileType = DataStream
a1.sinks.k1.hdfs.writeFormat = Text
a1.sinks.k1.hdfs.batchSize = 10000
a1.sinks.k1.hdfs.rollSize = 0
a1.sinks.k1.hdfs.rollCount = 10000
a1.sinks.k1.hdfs.filePrefix = syslog
a1.sinks.k1.hdfs.round = true


# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
[root@centos01 ~]# /etc/init.d/flume-agent start
Flume NG agent is not running                              [FAILED]
Starting Flume NG agent daemon (flume-agent):              [  OK  ]

Tested to login by ssh.

[root@centos01 ~]#  tail -0f  /var/log/flume/flume.log
06 Oct 2014 16:35:40,601 INFO  [hdfs-k1-call-runner-0] (org.apache.flume.sink.hdfs.BucketWriter.doOpen:208)  - Creating hdfs://localhost:8020/user/flume/syslog/2014/10/06/16//syslog.1412588139067.tmp
06 Oct 2014 16:36:10,957 INFO  [hdfs-k1-roll-timer-0] (org.apache.flume.sink.hdfs.BucketWriter.renameBucket:427)  - Renaming hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067.tmp to hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067
[root@centos01 ~]# hadoop fs -ls hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067
14/10/06 16:37:31 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   1 flume supergroup        299 2014-10-06 16:36 hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067
[root@centos01 ~]#
[root@centos01 ~]#
[root@centos01 ~]# hadoop fs -cat hdfs://localhost:8020/user/flume/syslog/2014/10/06/16/syslog.1412588139067
14/10/06 16:37:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
sshd[20235]: Accepted password for surachart from 192.168.111.16 port 65068 ssh2
sshd[20235]: pam_unix(sshd:session): session opened for user surachart by (uid=0)
su: pam_unix(su-l:session): session opened for user root by surachart(uid=500)
su: pam_unix(su-l:session): session closed for user root

Look good... Anyway, It needs to adapt more...

Life Learning - Pivotal HD

I was unable to learn much as much people who have spent life with products, but I was interested in something new, for example: Pivotal HD and it was easy for download it as Pivotal HD Single Node VM.

Pivotal HD is a 100% Apache-compatible Hadoop distribution featuring a fully SQL compliant query engine for processing data stored in Hadoop. By adding rich, mature SQL processing, Pivotal HD allows enterprises to simplify development, expand Hadoop’s capabilities, increase productivity, and cut costs. It has been tested for scale on the 1000 node Pivotal Analytics Workbench to ensure that the stack works flawlessly in large enterprise deployments.
Wow! I thought it's not bad, if I will learn a little bit about it.
First of all, I downloaded Pivotal HD Single Node VM and added it in my virtualbox, then started vm and check. (ssh by using "gpadmin"/"password")

[gpadmin@pivhdsne ~]$ pwd
/home/gpadmin
[gpadmin@pivhdsne ~]$ ls
Attic  Desktop  Documents  Downloads  gpAdminLogs  pivotal-samples  workspace
[gpadmin@pivhdsne ~]$ cd Desktop/
[gpadmin@pivhdsne Desktop]$ ls
eclipse  Pivotal_Community_Edition_VM_EULA_20130712_final.pdf  Pivotal_Docs  README  README~  start_piv_hd.sh  stop_piv_hd.sh
[gpadmin@pivhdsne Desktop]$ cat README
Pivotal HD 1.0.1 Single Node (VM)
Version 1

How to use this VM:
1. Start the Hadoop services using start_all.sh on the desktop
2. Follow the tutorials at http://pivotalhd.cfapps.io/getting-started/pivotalhd-vm.html
3. Leverage the Pivotal HD community at http://gopivotal.com/community for support
4. root and gpadmin accounts have password password
5. Command Center login is gpadmin, with password gpadmin
6. gpadmin account has sudo privileges

What is included:
1. Pivotal HD - Hadoop 2.x, Zookeeper, HBase, Hive, Pig, Mahout
2. Pivotal HAWQ
3. Pivotal Extension Framework (PXF)
4. Pivotal DataLoader
5. Product usage documentation

Other installed packages:
1. JDK 6
2. Ant
3. Maven
4. Eclipse

[gpadmin@pivhdsne Desktop]$ ./start_piv_hd.sh
Starting services
SUCCESS: Start complete
Using JAVA_HOME: /usr/java/jdk1.6.0_26
Starting dataloader in standalone mode...
Starting Embedded Zookeeper Server...
Sending output to /var/log/gphd/dataloader/dataloader-embedded-zk.log
Embedded Zookeeper Server started!
Starting dataloader scheduler...
Sending output to /var/log/gphd/dataloader/dataloader-scheduler.log
Dataloader Scheduler Started!
Starting dataloader manager...
Sending output to /var/log/gphd/dataloader/dataloader-manager.log
Dataloader Manager Started!
Dataloader started!
20130725:01:16:18:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Starting gpstart with args: -a
20130725:01:16:19:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Gathering information and validating the environment...
20130725:01:16:32:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (HAWQ) 4.2.0 build 1'
20130725:01:16:41:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Greenplum Catalog Version: '201306170'
20130725:01:16:41:005738 gpstart:pivhdsne:gpadmin-[WARNING]:-postmaster.pid file exists on Master, checking if recovery startup required
20130725:01:16:41:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Commencing recovery startup checks
20130725:01:16:43:005738 gpstart:pivhdsne:gpadmin-[INFO]:-No socket connection or lock file in /tmp found for port=5432
20130725:01:16:43:005738 gpstart:pivhdsne:gpadmin-[INFO]:-No Master instance process, entering recovery startup mode
20130725:01:16:43:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Clearing Master instance pid file
20130725:01:16:43:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Starting Master instance in admin mode
20130725:01:17:18:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20130725:01:17:18:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Obtaining Segment details from master...
20130725:01:17:20:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Setting new master era
20130725:01:17:20:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Commencing forced instance shutdown
20130725:01:17:33:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Starting Master instance in admin mode
20130725:01:17:46:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20130725:01:17:46:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Obtaining Segment details from master...
20130725:01:17:47:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Setting new master era
20130725:01:17:47:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Master Started...
20130725:01:17:47:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Shutting down master
20130725:01:17:59:005738 gpstart:pivhdsne:gpadmin-[INFO]:-No standby master configured.  skipping...
20130725:01:18:00:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
...............
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Process results...
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-----------------------------------------------------
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-   Successful segment starts                                            = 2
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-   Failed segment starts                                                = 0
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-----------------------------------------------------
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Successfully started 2 of 2 segment instances
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-----------------------------------------------------
20130725:01:18:15:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Starting Master instance pivhdsne.localdomain directory /data/1/hawq_master/gpseg-1
20130725:01:18:23:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Command pg_ctl reports Master pivhdsne.localdomain instance active
20130725:01:18:23:005738 gpstart:pivhdsne:gpadmin-[INFO]:-Database successfully started

This was first step. Then Read More..
After I read a bit, I thought I should test some sample. I chose to begin with Setting up the Development Environment. ...some step from link.

[gpadmin@pivhdsne customer_first_and_last_order_dates]$ hadoop jar target/customer_first_and_last_order_dates-1.0.jar com.pivotal.hadoop.CustomerFirstLastOrderDateDriver /retail_demo/orders/orders.tsv.gz /output-mr2
13/07/25 04:28:22 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
13/07/25 04:28:24 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
13/07/25 04:28:25 INFO input.FileInputFormat: Total input paths to process : 1
13/07/25 04:28:26 WARN snappy.LoadSnappy: Snappy native library is available
13/07/25 04:28:26 INFO snappy.LoadSnappy: Snappy native library loaded
13/07/25 04:28:27 INFO mapreduce.JobSubmitter: number of splits:1
13/07/25 04:28:27 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
13/07/25 04:28:27 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
13/07/25 04:28:27 WARN conf.Configuration: mapred.mapoutput.value.class is deprecated. Instead, use mapreduce.map.output.value.class
13/07/25 04:28:27 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
13/07/25 04:28:27 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
13/07/25 04:28:27 WARN conf.Configuration: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
13/07/25 04:28:27 WARN conf.Configuration: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
13/07/25 04:28:27 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
13/07/25 04:28:27 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
13/07/25 04:28:27 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
13/07/25 04:28:27 WARN conf.Configuration: mapred.mapoutput.key.class is deprecated. Instead, use mapreduce.map.output.key.class
13/07/25 04:28:27 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
13/07/25 04:28:28 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1374729279703_0002
13/07/25 04:28:30 INFO client.YarnClientImpl: Submitted application application_1374729279703_0002 to ResourceManager at pivhdsne/127.0.0.2:8032
13/07/25 04:28:30 INFO mapreduce.Job: The url to track the job: http://pivhdsne:8088/proxy/application_1374729279703_0002/
13/07/25 04:28:30 INFO mapreduce.Job: Running job: job_1374729279703_0002
13/07/25 04:29:01 INFO mapreduce.Job: Job job_1374729279703_0002 running in uber mode : false
13/07/25 04:29:01 INFO mapreduce.Job:  map 0% reduce 0%
13/07/25 04:29:59 INFO mapreduce.Job:  map 3% reduce 0%
13/07/25 04:30:03 INFO mapreduce.Job:  map 10% reduce 0%
13/07/25 04:30:06 INFO mapreduce.Job:  map 22% reduce 0%
13/07/25 04:30:10 INFO mapreduce.Job:  map 33% reduce 0%
13/07/25 04:30:13 INFO mapreduce.Job:  map 45% reduce 0%
13/07/25 04:30:16 INFO mapreduce.Job:  map 52% reduce 0%
13/07/25 04:30:20 INFO mapreduce.Job:  map 59% reduce 0%
13/07/25 04:30:23 INFO mapreduce.Job:  map 66% reduce 0%
13/07/25 04:30:32 INFO mapreduce.Job:  map 100% reduce 0%
13/07/25 04:31:08 INFO mapreduce.Job:  map 100% reduce 33%
13/07/25 04:31:11 INFO mapreduce.Job:  map 100% reduce 66%
13/07/25 04:31:26 INFO mapreduce.Job:  map 100% reduce 67%
13/07/25 04:31:33 INFO mapreduce.Job:  map 100% reduce 68%
13/07/25 04:31:36 INFO mapreduce.Job:  map 100% reduce 69%
13/07/25 04:31:39 INFO mapreduce.Job:  map 100% reduce 74%
13/07/25 04:31:43 INFO mapreduce.Job:  map 100% reduce 78%
13/07/25 04:31:46 INFO mapreduce.Job:  map 100% reduce 82%
13/07/25 04:31:49 INFO mapreduce.Job:  map 100% reduce 87%
13/07/25 04:31:53 INFO mapreduce.Job:  map 100% reduce 91%
13/07/25 04:31:56 INFO mapreduce.Job:  map 100% reduce 96%
13/07/25 04:31:59 INFO mapreduce.Job:  map 100% reduce 98%
13/07/25 04:32:02 INFO mapreduce.Job:  map 100% reduce 100%
13/07/25 04:32:02 INFO mapreduce.Job: Job job_1374729279703_0002 completed successfully
13/07/25 04:32:02 INFO mapreduce.Job: Counters: 43
        File System Counters
                FILE: Number of bytes read=18946633
                FILE: Number of bytes written=38031433
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=72797182
                HDFS: Number of bytes written=11891611
                HDFS: Number of read operations=6
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=2
        Job Counters
                Launched map tasks=1
                Launched reduce tasks=1
                Rack-local map tasks=1
                Total time spent by all maps in occupied slots (ms)=184466
                Total time spent by all reduces in occupied slots (ms)=257625
        Map-Reduce Framework
                Map input records=512071
                Map output records=512071
                Map output bytes=17922485
                Map output materialized bytes=18946633
                Input split bytes=118
                Combine input records=0
                Combine output records=0
                Reduce input groups=167966
                Reduce shuffle bytes=18946633
                Reduce input records=512071
                Reduce output records=167966
                Spilled Records=1024142
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=2486
                CPU time spent (ms)=67950
                Physical memory (bytes) snapshot=1088774144
                Virtual memory (bytes) snapshot=5095378944
                Total committed heap usage (bytes)=658378752
        Shuffle Errors
                BAD_ID=0
                CONNECTION=0
                IO_ERROR=0
                WRONG_LENGTH=0
                WRONG_MAP=0
                WRONG_REDUCE=0
        File Input Format Counters
                Bytes Read=72797064
        File Output Format Counters
                Bytes Written=11891611
[gpadmin@pivhdsne customer_first_and_last_order_dates]$
[gpadmin@pivhdsne customer_first_and_last_order_dates]$ hadoop fs -cat /output-mr2/part-r-00000 | wc -l
167966
[gpadmin@pivhdsne customer_first_and_last_order_dates]$ hadoop fs -cat /output-mr2/part-r-00000 | tail
54992348        6933068175      2010-10-04 02:21:44     6311149380      2010-10-11 18:21:12
54992896        8136297804      2010-10-02 11:33:38     6310999573      2010-10-11 19:01:48
54993581        6311050522      2010-10-11 21:43:05     8122646976      2010-10-14 16:28:00
54993992        6805481711      2010-10-01 15:32:50     8212538352      2010-10-07 00:20:50
54994403        7708210740      2010-10-08 06:41:29     8122646502      2010-10-14 06:36:05
54994814        8136355210      2010-10-02 15:36:27     7708874714      2010-10-08 19:50:08
54994951        6805748378      2010-10-01 05:38:36     8494440118      2010-10-09 04:44:55
54995088        8136355283      2010-10-02 23:29:08     5007019717      2010-10-13 12:32:23
54995225        6805524075      2010-10-01 08:26:01     6933068564      2010-10-04 23:02:24
54995773        6933024646      2010-10-04 11:57:57     5007019751      2010-10-13 03:36:25
[gpadmin@pivhdsne customer_first_and_last_order_dates]$

I forgot to show url (http://ip:50000)

I thought Pivotal HD is a good product that I can use it to learn about Hadoop and etc (Hadoop 2.x, Zookeeper, HBase, Hive, Pig, Mahout and Pivotal HAWQ).

learn - perl DBD:Oracle bind variable?

I was interested about perl (DBD:Oracle) connect with Oracle Database and bind variable, Everyone who works with Oracle Database know, we should use SQL or PL/SQL with binding variable. So, This was simple program for learning with DBD:Oracle.
- bind_param 
- bind_param_inout

[oracle@chart01 ~]$ cat add.pl
#!/usr/bin/perl
use strict;
use DBI;
my $num_args = $#ARGV + 1;
if ($num_args != 2) {
  print "Example:\t./add.pl number1 number2\n\n";
  exit;
}
my $i=$ARGV[0];
my $j=$ARGV[1];
my $result;
my $db = DBI->connect( "dbi:Oracle:orcl", "demo", "demo",{AutoCommit=>0, RaiseError=>1} )
or die "Couldn't connect to database: " . DBI->errstr;
my $sth = $db->prepare("BEGIN
                :result := fn_add(:i,:j);
                END; ")
or die "Couldn't prepare statement: " . $db->errstr;
$sth->bind_param( ":i",$i);
$sth->bind_param( ":j",$j);
$sth->bind_param_inout(":result", \$result,38);
$sth->execute()
or die "Couldn't execute statement: " . $sth->errstr;
$sth->finish;
$db->disconnect;
print "Result: $result \n\n"; 

[oracle@chart01 ~]$ ./add.pl 1 2
Result: 3
[oracle@chart01 ~]$ ./add.pl 1 1
Result: 2
[oracle@chart01 ~]$ ./add.pl 1 0
Result: 1
[oracle@chart01 ~]$ ./add.pl 100 50
Result: 150
[oracle@chart01 ~]$ ./add.pl 10 5
Result: 15

Then I checked in V$SQLAREA

SQL> select sql_text, executions  from v$sqlarea where sql_text like '%:result := fn_add(:i,:j)%';
SQL_TEXT                                                                                   EXECUTIONS
------------------------------------------------------------------------------------------ ----------
BEGIN                 :result := fn_add(:i,:j);                 END;                               14

However, If you would like to query by using "SELECT" command. You could bind column (bind_columns) as:

$sth = $db->prepare("select address from accounts where name = :name ")
        or die "Couldn't prepare statement: " . $db->errstr;
$sth->bind_param( ":name",$in_name);
$sth->execute()
or die "Couldn't execute statement: " . $sth->errstr;
$sth->bind_columns(undef, \$address);

Yes, that's something. I learned today. not a lot.

learn - a little bit of RHEL Cluster on OL

I was interested a little bit of Linux Clustering with RHEL. I just thought how to use it controlling httpd service(only). Anyway, I had Oracle Linux only. I just installed it on my virtualbox for 2 guests. After I installed Oracle Linux and Clustering packages. I tested to starting. I found error. FATAL: Module lock_dlm not found. 
I just found out in the internet and etc. Then!

[root@server01 ~]# /etc/init.d/cman start
Starting cluster:
   Loading modules... failed
FATAL: Module lock_dlm not found.
                                                           [FAILED]
[root@server01 ~]# vim /etc/init.d/cman

I checked in "/etc/init.d/cman" file.

load_modules()
{
    errmsg=$( /sbin/modprobe configfs 2>&1 ) || return 1
    errmsg=$( /sbin/modprobe dlm 2>&1 ) || return 1
    errmsg=$( /sbin/modprobe lock_dlm 2>&1 ) || return 1
    return 0
}

I tried to testing and finding out some.

[root@server01 ~]# modprobe lock_dlm
FATAL: Module lock_dlm not found.
[root@server01 ~]# uname -r
2.6.39-200.29.2.el5uek
[root@server01 ~]# find /lib/modules/ -name '*dlm.ko'
/lib/modules/2.6.18-308.11.1.0.1.el5xen/kernel/fs/gfs2/locking/dlm/lock_dlm.ko
/lib/modules/2.6.18-308.11.1.0.1.el5xen/kernel/fs/dlm/dlm.ko
/lib/modules/2.6.32-200.13.1.el5uek/kernel/fs/dlm/dlm.ko
/lib/modules/2.6.32-200.13.1.el5uek/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko
/lib/modules/2.6.18-308.11.1.0.1.el5/kernel/fs/gfs2/locking/dlm/lock_dlm.ko
/lib/modules/2.6.18-308.11.1.0.1.el5/kernel/fs/dlm/dlm.ko
/lib/modules/2.6.18-274.el5/kernel/fs/gfs2/locking/dlm/lock_dlm.ko
/lib/modules/2.6.18-274.el5/kernel/fs/dlm/dlm.ko
/lib/modules/2.6.39-200.29.2.el5uek/kernel/fs/dlm/dlm.ko
/lib/modules/2.6.39-200.29.2.el5uek/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko
[root@server01 ~]# yum list | grep kmod-gfs
This system is not registered with ULN.
ULN support will be disabled.
kmod-gfs.i686                               0.1.34-17.0.1.el5         installed
kmod-gfs-xen.i686                           0.1.34-17.0.1.el5         installed
kmod-gfs-PAE.i686                           0.1.34-17.0.1.el5         el5_latest

I was using "2.6.39-200.29.2.el5uek" kernel. I found out and ... READ!

Does Oracle provide support for Red Hat Global File System (GFS), Red Hat Cluster Suite (RHCS), or RedHat Application Server (RHAS)?Red Hat Global File System (GFS) and Red Hat Cluster Suite (RHCS) are part of RHEL5 and Oracle Linux 5 so they are supported by Oracle under the Linux Support Program. However, since GFS and RHCS are not included with RHEL4, Oracle Linux 4 and earlier versions, they are not supported by Oracle with RHEL4, OL4, and earlier versions. Similarly, Oracle Linux 6 does not include support for features that are delivered as extra cost add-ons in Red Hat Enterprise Linux 6, such as the High Availability Add-On for clustering and the Resilient Storage Add-On for GFS2.

That was something, I read. However, I can rebooted with "2.6.18-308.11.1.0.1.el5xe" kernel and started it. Like

[root@server01 ~]# uname -r
2.6.18-308.11.1.0.1.el5xen
[root@server01 ~]# /etc/init.d/cman start
Starting cluster:
   Loading modules... done
   Mounting configfs... done
   Starting ccsd... done
   Starting cman... done
   Starting daemons... done
   Starting fencing... done
                                                           [  OK  ]
[root@server01 ~]# lsmod | grep lock_dlm
lock_dlm               20705  0
gfs2                  354825  1 lock_dlm
dlm                   114645  5 lock_dlm

However, what is "lock_dlm"? I needed to use it or not?  Read!

lock_dlm — The standard locking module, required for a clustered file system.

In this case, I didn't use the clustered file system. I just need Linux Cluster to control httpd service. So, I could comment it in "/etc/init.d/cman" file. Like

load_modules(){
    errmsg=$( /sbin/modprobe configfs 2>&1 ) || return 1
    errmsg=$( /sbin/modprobe dlm 2>&1 ) || return 1
#    errmsg=$( /sbin/modprobe lock_dlm 2>&1 ) || return 1
    return 0
}

Really? Let me show my stupid with it:) just fun.

[root@server01 ~]# uname -r
2.6.39-200.29.2.el5uek
[root@server01 ~]# grep  lock_dlm /etc/init.d/cman
#    errmsg=$( /sbin/modprobe lock_dlm 2>&1 ) || return 1
[root@server01 ~]# /etc/init.d/cman start
Starting cluster:
   Loading modules... done
   Mounting configfs... done
   Starting ccsd... done
   Starting cman... done
   Starting daemons... done
   Starting fencing... done
                                                           [  OK  ]
[root@server01 ~]# clustat
Cluster Status for new_cluster @ Sat Jul 21 13:39:22 2012
Member Status: Quorate
 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 server01                                                            1 Online, Local
 server02                                                            2 Online
[root@server01 ~]# /etc/init.d/rgmanager start
Starting Cluster Service Manager:                          [  OK  ]
[root@server01 ~]# clustat
Cluster Status for new_cluster @ Sat Jul 21 13:39:32 2012
Member Status: Quorate
 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 server01                                                            1 Online, Local, rgmanager
 server02                                                            2 Online, rgmanager
 Service Name                                                   Owner (Last)                                                   State
 ------- ----                                                   ----- ------                                                   -----
 service:httpd                                                  server02                                                       started
[root@server01 ~]# clustat
Cluster Status for new_cluster @ Sat Jul 21 13:39:48 2012
Member Status: Quorate
 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 server01                                                            1 Online, Local, rgmanager
 server02                                                            2 Online, rgmanager
 Service Name                                                   Owner (Last)                                                   State
 ------- ----                                                   ----- ------                                                   -----
 service:httpd                                                  server01                                                       starting
[root@server01 ~]# clustat
Cluster Status for new_cluster @ Sat Jul 21 13:40:03 2012
Member Status: Quorate
 Member Name                                                     ID   Status
 ------ ----                                                     ---- ------
 server01                                                            1 Online, Local, rgmanager
 server02                                                            2 Online, rgmanager
 Service Name                                                   Owner (Last)                                                   State
 ------- ----                                                   ----- ------                                                   -----
 service:httpd                                                  server01                                                       started

I assumed all ok. So Far, So Good.

learn - disable zeroconf(169.254/16)

When you executed "route" on linux and saw 169.254.0.0/255.255.0.0 in the routing table.

[root@testlinux ~]# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.111.1   0.0.0.0         UG    0      0        0 eth0
169.254.0.0     *               255.255.0.0     U     0      0        0 eth0
192.168.111.0   *               255.255.255.0   U     0      0        0 eth0

What is that? A lot of people call it - zeroconf (Zero configuration networking). It will help making networking easy & easy.
If You read on  zoneconf website, there are four main requirements:
- Allocate addresses without a DHCP server
- Translate between names and IP addresses without a DNS server
- Find services, like printers, without a directory server
- Allocate IP Multicast addresses without a MADCAP server

You can find out more on the internet. Example links:
http://www.zeroconf.org/
http://en.wikipedia.org/wiki/Zero_configuration_networking
http://datatracker.ietf.org/wg/zeroconf/charter/
ETC.

However, Zeroconf protocols might cause harm to the network when a machine is plugged into a large network. In this post, I just wanted to disable 169.254/16 in the routing table, when network started.

[root@testlinux ~]# cp /etc/sysconfig/network /etc/sysconfig/network-orig[root@testlinux ~]# vi /etc/sysconfig/network[root@testlinux ~]# diff  /etc/sysconfig/network-orig /etc/sysconfig/network
4a5
> NOZEROCONF=yes[root@testlinux ~]# service network restartShutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface eth0:                                [  OK  ]
[root@testlinux ~]#
[root@testlinux ~]#
[root@testlinux ~]# routeKernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.168.111.1   0.0.0.0         UG    0      0        0 eth0
192.168.111.0   *               255.255.255.0   U     0      0        0 eth0

It doesn't have 169.254/16 in the routing table now :)

Learn ulogd - set ulogd with mysql

In this post, it looks like nothing. It was something fun for me. Because i was interested in shorewall and ulogd. So, I just thought how I keep log in mysql. This post wasn't value much, but helped me to learning something new.
I installed what packages were necessary (disabled iptables service). It was easy for you to use "yum" and "systemctl". First of all, I just ensured I have all packages.

[root@fedora ~]# yum list shorewall shorewall-core  ulogd ulogd-mysql mysql-server
Loaded plugins: langpacks, presto, refresh-packagekit
Installed Packages
mysql-server.i686                                                    5.5.24-1.fc17                                                      @updates
shorewall.noarch                                                     4.5.4-1.fc17                                                       @updates
shorewall-core.noarch                                                4.5.4-1.fc17                                                       @updates
ulogd.i686                                                           2.0.0-2.beta4.fc17                                                 @fedora
ulogd-mysql.i686                                                     2.0.0-2.beta4.fc17                                                 @fedora

In shorewall configuration. I just set policy for logging to NFLOG. If you are interested about it. You can read on http://www.shorewall.net/  (I did not emphasize how to configure shorewall, I just figured ulogd with mysql).

[root@fedora ~]# cat /etc/shorewall/policy
.
.
.
$FW            net                ACCEPT NFLOG
net            all                ACCEPT  NFLOG
all            all                REJECT  NFLOG

Then I configured ulogd with mysql.  
Note: mysql-ulogd2.sql file, If you downloaded "ulogd" from source. You will see it in compress file. 

[root@fedora ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 4
Server version: 5.5.24 MySQL Community Server (GPL)
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> create database nulog;
Query OK, 1 row affected (0.00 sec)
mysql>mysql> grant all privileges on nulog.* to surachart@localhost identified by 'password';

Query OK, 0 rows affected (0.00 sec)
mysql> commit;
Query OK, 0 rows affected (0.00 sec) 

[root@fedora ~]# mysql -h localhost -u surachart -p nulog
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 11
Server version: 5.5.24 MySQL Community Server (GPL)
Copyright (c) 2000, 2011, Oracle and/or its affiliates. All rights reserved.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> source mysql-ulogd2.sql

After mysql no error, I configured ulogd.conf. 

[root@fedora ~]# cp /etc/ulogd.conf /etc/ulogd.conf-orig
[root@fedora ~]# vi /etc/ulogd.conf
[root@fedora ~]# diff /etc/ulogd.conf-orig /etc/ulogd.conf
45c45
< #plugin="/usr/lib/ulogd/ulogd_output_MYSQL.so"
---
> plugin="/usr/lib/ulogd/ulogd_output_MYSQL.so"
77c77
< #stack=log2:NFLOG,base1:BASE,ifi1:IFINDEX,ip2bin1:IP2BIN,mac2str1:HWHDR,mysql1:MYSQL
---
> stack=log1:NFLOG,base1:BASE,ifi1:IFINDEX,ip2bin1:IP2BIN,mac2str1:HWHDR,mysql1:MYSQL
166c166
< user="nupik"
---
> user="surachart"
168c168
< pass="changeme"
---
> pass="password"

In configuration file, I used  log1 in stack.
After I ensured configuration, that's fine. I just started ulogd and shorewall.

[root@fedora ~]# service ulogd start
Starting ulogd (via systemctl):                            [  OK  ]
[root@fedora ~]# service shorewall start
Redirecting to /bin/systemctl  start shorewall.service

[root@fedora ~]# /bin/systemctl  status  ulogd.service
ulogd.service - LSB: start and stop ulogd
          Loaded: loaded (/etc/rc.d/init.d/ulogd)
          Active: active (running) since Thu, 12 Jul 2012 15:10:28 +0700; 17min ago
         Process: 26609 ExecStop=/etc/rc.d/init.d/ulogd stop (code=exited, status=0/SUCCESS)
         Process: 26632 ExecStart=/etc/rc.d/init.d/ulogd start (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/ulogd.service
                  â”” 26638 /usr/sbin/ulogd -d
Jul 12 15:10:28 fedora.surachartopun.com ulogd[26632]: [33B blob data]
[root@fedora ~]# /bin/systemctl  status  shorewall.service
shorewall.service - Shorewall IPv4 firewall
          Loaded: loaded (/usr/lib/systemd/system/shorewall.service; enabled)
          Active: active (exited) since Thu, 12 Jul 2012 15:10:36 +0700; 17min ago
         Process: 26475 ExecStop=/sbin/shorewall $OPTIONS stop (code=exited, status=0/SUCCESS)
         Process: 26649 ExecStart=/sbin/shorewall $OPTIONS start (code=exited, status=0/SUCCESS)
          CGroup: name=systemd:/system/shorewall.service
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: Setting up Martian Logging...
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: Setting up Proxy ARP...
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: Setting up Traffic Control...
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: Preparing iptables-restore input...
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: Running /sbin/iptables-restore...
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: IPv4 Forwarding Enabled
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: Processing /etc/shorewall/start ...
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: Processing /etc/shorewall/started ...
Jul 12 15:10:36 fedora.surachartopun.com logger[26915]: Shorewall started
Jul 12 15:10:36 fedora.surachartopun.com shorewall[26649]: done.

 [root@fedora ~]# iptables -LChain INPUT (policy DROP)
target     prot opt source               destination
net2fw     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
Reject     all  --  anywhere             anywhere
NFLOG      all  --  anywhere             anywhere             nflog-prefix  --nflog-prefix
reject     all  --  anywhere             anywhere            [goto]
Chain FORWARD (policy DROP)
target     prot opt source               destination
Reject     all  --  anywhere             anywhere
NFLOG      all  --  anywhere             anywhere             nflog-prefix  --nflog-prefix
reject     all  --  anywhere             anywhere            [goto]
Chain OUTPUT (policy DROP)
target     prot opt source               destination
fw2net     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
Reject     all  --  anywhere             anywhere
NFLOG      all  --  anywhere             anywhere             nflog-prefix  --nflog-prefix
reject     all  --  anywhere             anywhere            [goto]
Chain Broadcast (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type MULTICAST
DROP       all  --  anywhere             anywhere             ADDRTYPE match dst-type ANYCAST
DROP       all  --  anywhere             base-address.mcast.net/4
Chain Invalid (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             ctstate INVALID
Chain NotSyn (1 references)
target     prot opt source               destination
DROP       tcp  --  anywhere             anywhere             tcpflags:! FIN,SYN,RST,ACK/SYN
Chain Reject (3 references)
target     prot opt source               destination
           all  --  anywhere             anywhere
reject     tcp  --  anywhere             anywhere             tcp dpt:auth /* --co */
Broadcast  all  --  anywhere             anywhere
ACCEPT     icmp --  anywhere             anywhere             icmp fragmentation-needed /* --comment */
ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded /* --comment */
Invalid    all  --  anywhere             anywhere
reject     udp  --  anywhere             anywhere             multiport dports epmap,microsoft-ds /* --c */
reject     udp  --  anywhere             anywhere             udp dpts:netbios-ns:netbios-ssn /* --c */
reject     udp  --  anywhere             anywhere             udp spt:netbios-ns dpts:1024:65535 /* --c */
reject     tcp  --  anywhere             anywhere             multiport dports epmap,netbios-ssn,microsoft-ds /* --c */
DROP       udp  --  anywhere             anywhere             udp dpt:ssdp /* --co */
NotSyn     tcp  --  anywhere             anywhere
DROP       udp  --  anywhere             anywhere             udp spt:domain /* --comment */
Chain dynamic (1 references)
target     prot opt source               destination
Chain fw2net (1 references)
target     prot opt source               destination
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
NFLOG      all  --  anywhere             anywhere             nflog-prefix  --nflog-prefix
ACCEPT     all  --  anywhere             anywhere
Chain logdrop (0 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
Chain logreject (0 references)
target     prot opt source               destination
reject     all  --  anywhere             anywhere
Chain net2fw (1 references)
target     prot opt source               destination
dynamic    all  --  anywhere             anywhere             ctstate INVALID,NEW
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
NFLOG      all  --  anywhere             anywhere             nflog-prefix  --nflog-prefix
ACCEPT     all  --  anywhere             anywhere
Chain reject (9 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             ADDRTYPE match src-type BROADCAST
DROP       all  --  base-address.mcast.net/4  anywhere
DROP       igmp --  anywhere             anywhere
REJECT     tcp  --  anywhere             anywhere             reject-with tcp-reset
REJECT     udp  --  anywhere             anywhere             reject-with icmp-port-unreachable
REJECT     icmp --  anywhere             anywhere             reject-with icmp-host-unreachable
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited
Chain sfilter (0 references)
target     prot opt source               destination
LOG        all  --  anywhere             anywhere             LOG level info prefix "--log-prefix"
DROP       all  --  anywhere             anywhere
Chain shorewall (0 references)
target     prot opt source               destination

Test & Test. After that, I checked in mysql

mysql> SELECT tcp_dport,count(*) from view_tcp GROUP BY view_tcp.tcp_dport;
+-----------+----------+
| tcp_dport | count(*) |
+-----------+----------+
|        22 |        7 |
|        80 |        3 |
|       443 |        1 |
+-----------+----------+
3 rows in set (0.00 sec)

It looked like work now. I plan to learn more about it.  
so far so good :)

Learn fedora - Network interface 's renamed

I was new.... with fedora. When I installed it (fedora 17) in my virtualbox. I found network interface name (p2p1). So I just needed to change it to be "eth0". I tried to finding a lot in /etc/ path. but nothing what I figured out. Then, I read on link. So, I started to fix it by uncomment in  /lib/udev/rules.d/71-biosdevname.rules  file.

[root@fedora ~]# ifconfig
lo: flags=73  mtu 16436
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 64  bytes 5952 (5.8 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 64  bytes 5952 (5.8 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
p2p1: flags=4163  mtu 1500
        inet 192.168.111.50  netmask 255.255.255.0  broadcast 192.168.111.255
        inet6 fe80::a00:27ff:fead:793d  prefixlen 64  scopeid 0x20
        ether 08:00:27:ad:79:3d  txqueuelen 1000  (Ethernet)
        RX packets 1538  bytes 140302 (137.0 KiB)
        RX errors 0  dropped 3  overruns 0  frame 0
        TX packets 1204  bytes 191575 (187.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
[root@fedora ~]#dmesg | grep p2p1
[   10.731136] udevd[374]: renamed network interface eth0 to p2p1
[root@fedora ~]# cp -a /lib/udev/rules.d/71-biosdevname.rules /tmp/71-biosdevname.rules-orig
[root@fedora ~]# vi /lib/udev/rules.d/71-biosdevname.rules
[root@fedora ~]# diff /tmp/71-biosdevname.rules-orig /lib/udev/rules.d/71-biosdevname.rules
15c15
< # GOTO="netdevicename_end"
---
> GOTO="netdevicename_end"

[root@fedora ~]# reboot 

 Then, I checked it again.

[root@fedora ~]# ifconfig -a
eth0: flags=4163  mtu 1500
        inet 192.168.111.50  netmask 255.255.255.0  broadcast 192.168.111.255
        inet6 fe80::a00:27ff:fead:793d  prefixlen 64  scopeid 0x20
        ether 08:00:27:ad:79:3d  txqueuelen 1000  (Ethernet)
        RX packets 119  bytes 12463 (12.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 111  bytes 16396 (16.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
lo: flags=73  mtu 16436
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10
        loop  txqueuelen 0  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

I knew this's simple for someone who works with it a long time. But it was something new and funny for me.