Thursday, November 01, 2012

Learn - hadoop-fuse-dfs


Read Mounting HDFS. There is something very interesting. As people know HDFS is Hadoop Distributed File System. So, i wanted to test a bit about hadoop-fuse-dfs. Somehow, I had impala for test only. I thought it's easy to use it and learn.
-bash-4.1$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                24607156   4370100  19987060  18% /
devtmpfs               1534680       200   1534480   1% /dev
tmpfs                  1544584       112   1544472   1% /dev/shm
/dev/sda1             24607156   4370100  19987060  18% /
-bash-4.1$ sudo hadoop-fuse-dfs   rw -oserver=127.0.0.1 /mnt/
INFO /data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.1.0/src/hadoop-hdfs-project/hadoop-hdfs/src/main/native/fuse-dfs/fuse_options.c:164 Adding FUSE arg /mnt/
-bash-4.1$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
rootfs                24607156   4370104  19987056  18% /
devtmpfs               1534680       200   1534480   1% /dev
tmpfs                  1544584       112   1544472   1% /dev/shm
/dev/sda1             24607156   4370104  19987056  18% /
fuse_dfs              24576000         0  24576000   0% /mnt
-bash-4.1$
-bash-4.1$ ls /mnt/user/cloudera/
file.out  file.txt
-bash-4.1$ cat  /mnt/user/cloudera/file.txt
1,Surachart
2,John
3,Jame
-bash-4.1$
-bash-4.1$  hadoop fs -cat file.txt
1,Surachart
2,John
3,Jame
-bash-4.1$ cat /mnt/user/cloudera/file.txt
1,Surachart
2,John
3,Jame
-bash-4.1$ vi /mnt/user/cloudera/file.txt
-bash-4.1$
-bash-4.1$ cat /mnt/user/cloudera/file.txt
-bash-4.1$-- no data, used "cat" command again  --
-bash-4.1$ cat /mnt/user/cloudera/file.txt
1,Surachart
2,John
3,Jame
4,Chart
-bash-4.1$  hadoop fs -cat file.txt
1,Surachart
2,John
3,Jame
4,Chart
-bash-4.1$  hadoop fs -ls
Found 4 items
-rw-r--r--   1 root     cloudera          0 2012-11-01 09:46 .file.txt.swp
-rw-r--r--   1 root     cloudera          0 2012-11-01 09:49 .file.txt.swx
drwxr-xr-x   - cloudera cloudera          0 2012-11-01 09:29 file.out
-rw-r--r--   1 cloudera cloudera         34 2012-11-01 09:58 file.txt
From above example, I mount hdfs to /mnt path and modified file. After saved file (wq), I used "cat" command but no data in file. Then tried again, i found data. :)
Then, tested creating file.

-bash-4.1$ hadoop fs -ls file2.txt
ls: `file2.txt': No such file or directory
-bash-4.1$ echo "TEST1" > /mnt/user/cloudera/file2.txt
-bash-4.1$ echo "TEST2" >> /mnt/user/cloudera/file2.txt
-bash-4.1$ cat /mnt/user/cloudera/file2.txt
TEST1
TEST2
-bash-4.1$ hadoop fs -ls file2.txt
Found 1 items
-rw-r--r--   1 cloudera cloudera         12 2012-11-01 11:08 file2.txt
-bash-4.1$ hadoop fs -cat file2.txt
TEST1
TEST2
However, there supports some... operations.
Supports reads, writes, and directory operations (e.g., cp, ls, more, cat, find, less, rm, mkdir, mv, rmdir). Things like touch, chmod, chown, and permissions are in the works. Fuse-dfs currently shows all files as owned by nobody.
If tested more... Maybe, you might find something weird(not sure why?).
-bash-4.1$ cat /mnt/user/cloudera/file2.txt
TEST1
TEST2
-bash-4.1$ echo "TEST" > /mnt/user/cloudera/file2.txt
-bash-4.1$ cat /mnt/user/cloudera/file2.txt
-bash-4.1$-- no data -- 
-bash-4.1$ hadoop fs -cat file2.txt
-bash-4.1$-- no data -- 
Maybe, someone can help to explain :)... So, just learn and test before use.


No comments: