Wang Hoi (wkai): Portable Meta-Information again…
| March 31st, 2009
Portable meta-information has been discussed twice recently:
http://www.kdedevelopers.org/node/3923
http://zwabel.wordpress.com/2009/03/29/portable-meta-information/
I also have something to say that doesn't fit into one comment, so i post it :)
My thought is realistic:
1. Will nepomuk really be more usable with portable/distributed stored meta information.
I don't know much of nepomuk's server implementations, afaik it seems use a central database to index/store these informations, so i've a question to ask : when user move his file to another local folder, will the file's meta infomation get lost , updated later(next time to index) or updated instantly (use inotify )? The first result(lost) is totally un acceptable, the second is ok, but nepomuk's value/power will be limited, the third is good, just stick to central storage.
2. How to implement it.
someone has suggested to use side-car file, other suggested xattr.
There is a seems better way to implement it : file forks/resource forks (not process fork),
http://en.wikipedia.org/wiki/Fork_(filesystem)
http://en.wikipedia.org/wiki/Resource_fork
It's invented by Apple , store text file's encoding, app's icon ...
Quote:
Apple's HFS, and the original Apple Macintosh file system MFS, were designed to allow a file to have a resource fork to store metadata that would be used by the system's graphical user interface (GUI), such as a file's icon to be used by the Finder or the menus and dialog boxes associated with an application.
It tightly binded to a file unlike sidecar file, but also bypass the size limit of xattr, the size limit is the largest file you can create.
This is a practise proof method,but only modern filesystem support it:
Only Apple's HFS, M$'s NTFS, Solaris's ZFS has full support.
And extensively used by Apple and M$.
Something need to mention, that Mac OS X's unix command line utilities (cp, mv, ....) can handle file with resource/file forks correctly. M$ name it alternate data stream (ADS). You can refer their dev docs to see how they design the api.
Oops the main linux filesystem, ext2/3/4, xfs doesn't support it well.
Seems not pay much attentions too.........
Now we move to the next question, is it hard to implement it under kernel, or which level should the support goes to ?
In my personal opinion, not hard indeed, i want :) and asume it to be implement in VFS level,
the simplest way is to add a dentry to each general inode, that from the filesystem's view(not user visible), a regular file can be associated with a "directory" too, pointed to the metadata files.
like this:
/home/xx/xxx/a.jpeg ----> nepomuk.xml
encoding
acl
....
and make getfattr/setfattr related system call to lookup that dentry too, this method also remains backward compatible with xattrs, but remove the size limit put on them.
Of cause, we need more analysis/profiling to say sth. on the memory/time performance.
From my predictation, the extra memory requirement it introduces(an extra dentry pointer) is affordable, and there's no/very subtle extra time needed with open/write ..regular system call, and it will be fast than create side-car files, since we needn't to create/open the side-car files from user space, kernel handled it ..
Anyway, to implement portable meta information, i think we need support from underline library/filesystem/architecture.
Just my own murmurs.
Thanks for reading..![]()
http://www.kdedevelopers.org/node/3923
http://zwabel.wordpress.com/2009/03/29/portable-meta-information/
I also have something to say that doesn't fit into one comment, so i post it :)
My thought is realistic:
1. Will nepomuk really be more usable with portable/distributed stored meta information.
I don't know much of nepomuk's server implementations, afaik it seems use a central database to index/store these informations, so i've a question to ask : when user move his file to another local folder, will the file's meta infomation get lost , updated later(next time to index) or updated instantly (use inotify )? The first result(lost) is totally un acceptable, the second is ok, but nepomuk's value/power will be limited, the third is good, just stick to central storage.
2. How to implement it.
someone has suggested to use side-car file, other suggested xattr.
There is a seems better way to implement it : file forks/resource forks (not process fork),
http://en.wikipedia.org/wiki/Fork_(filesystem)
http://en.wikipedia.org/wiki/Resource_fork
It's invented by Apple , store text file's encoding, app's icon ...
Quote:
Apple's HFS, and the original Apple Macintosh file system MFS, were designed to allow a file to have a resource fork to store metadata that would be used by the system's graphical user interface (GUI), such as a file's icon to be used by the Finder or the menus and dialog boxes associated with an application.
It tightly binded to a file unlike sidecar file, but also bypass the size limit of xattr, the size limit is the largest file you can create.
This is a practise proof method,but only modern filesystem support it:
Only Apple's HFS, M$'s NTFS, Solaris's ZFS has full support.
And extensively used by Apple and M$.
Something need to mention, that Mac OS X's unix command line utilities (cp, mv, ....) can handle file with resource/file forks correctly. M$ name it alternate data stream (ADS). You can refer their dev docs to see how they design the api.
Oops the main linux filesystem, ext2/3/4, xfs doesn't support it well.
Seems not pay much attentions too.........
Now we move to the next question, is it hard to implement it under kernel, or which level should the support goes to ?
In my personal opinion, not hard indeed, i want :) and asume it to be implement in VFS level,
the simplest way is to add a dentry to each general inode, that from the filesystem's view(not user visible), a regular file can be associated with a "directory" too, pointed to the metadata files.
like this:
/home/xx/xxx/a.jpeg ----> nepomuk.xml
encoding
acl
....
and make getfattr/setfattr related system call to lookup that dentry too, this method also remains backward compatible with xattrs, but remove the size limit put on them.
Of cause, we need more analysis/profiling to say sth. on the memory/time performance.
From my predictation, the extra memory requirement it introduces(an extra dentry pointer) is affordable, and there's no/very subtle extra time needed with open/write ..regular system call, and it will be fast than create side-car files, since we needn't to create/open the side-car files from user space, kernel handled it ..
Anyway, to implement portable meta information, i think we need support from underline library/filesystem/architecture.
Just my own murmurs.
Thanks for reading..