I just bought a new mobile phone to replace my aging HTC Hero, a brand new Google Nexus S device, one of the latest offerings in the mobile arena.

However, this model being new and all, means it lacks some basic(for me) stuff, like a proper rooted custom ROM with the tun.ko module included. The tun.ko module is used, among other things, for VPN connections. Which I need.

While installing a custom ROM would normally solve this, there are no stable or feature-full ROMs out there for this phone including the needed module, at this point in time anyway[1] [2] [3] [4]. They seems to exist only as proof of concept. I installed about 10 different customs ROMs today and they were all pretty much identical to the original ROM, apart from a couple of included apps that one can install by themselves anyway. Thus, I sacrificed my whole day off to get this working. I had lots of fun doing it… NOT!

(MoDaCo ROMs do exist with TUN included but I had so many issues with their HTC Hero one, that I passed…)

VPN capabilities in android devices require a) root access, b) tun.ko linux kernel module and c) VPN client binaries.

The client binaries I had, so that was crossed out pretty quickly.

Rooting the phone was easy as well, thanks to xda-developers.com as usual.

Tun.ko, however, is another story.

There are, like me, lots of people asking for a tun.ko module(or any other linux kernel module) for a specific android version and build. There aren’t so many Linux kernels running android out there(I counted about 10), so what’s stopping anyone building a tun.ko for each one and be done with it?

The problems seems to be that the linux kernel used on android devices is not allowed to load any module not matching the version of the kernel. While the concept is sound, to prevent kernel crashes, the implementation is not. But first, some background.

The linux kernel, and android in general, is managed by git. Each snapshot of the repository, snapshot being the state of the source code after every commit, is marked by a unique SHA-1 hash string as identifier.

Every kernel and its modules are tattooed by “vermagic”, a string value that identifies the kernel version. This includes a substring of the git unique hash identifier of the git snapshot at the time of the compile. Thus, a kernel will only be able to load compiled modules of the same git snapshot at that time. If another commit is made to the repository, the git snapshot identifier will change, thus any new compiled code will be incompatible with the previous one, even if wasn’t affected by the commit.

Now the fun part. Since the vermagic only includes a small part of the actual identifier, although all of the source code repositories are public, one can never know which snapshot was used to compile a specific kernel, because no search engine will ever return results on a substring. You can only search on full strings.

Although only a handful of Linux kernel versions are used for android devices, the number of different releases with different vermagic combinations, seems infinite. Every different tweak in the source results in different vermagic values. Pretty much every release is unique. If you want to share compiled code, you need to track the originating repository snapshot of point in time. Good luck with that.

(Actually, you can get a list of all the identifiers in the history of a source repository and you can grep it for any substring. However, you would need to do than on every android source repository you can find. Assuming of course, that the repository you are searching is public somewhere and not inside a manufacturer’s intranet.)

I was forced to, at the very least, rebuild my own linux kernel, only to extract the compiled tun.ko module. If I couldn’t, I would need to create my own custom ROM, only to include my own kernel with my own vermagic value and my own modules. For the record, before starting this exercice, I never ever even touched android source. All of a sudden, I was forced to learn how to build ROMs. Just because I can, doesn’t mean I want to. Fortunately, this helped a lot. Apart that /proc/config.gz is apparently now a luxury that we cannot afford to have, either on stock kernels or custom ones. *Sigh*…

My phone has a vermagic value “2.6.35.7-g7f1638a” and I failed to find my kernel’s original source repository. I did find the kernel source for my phone specifically, from Samsung itself, but it didn’t seem to include this snapshot.

Although I did reach a point of almost rolling out my own ROM, I decided to cheat. I was able to produce a tun.ko with my phone’s vermagic, even if my source produced a different one. All I had to do was rig “scripts/setlocalversion” to return whatever vermagic value I needed.

After that, I loaded the module to my phone and it worked. Flawlessly.

Any here it is, hopefully it will be useful. It applies to:

Model number: Nexus S
Android version: 2.3.2
Baseband version: I9020XXJK8
Kernel version: 2.6.35.7-g7f1638a android-build@apa28 #1
Build number: GRH78C

The work around is to create a hard link to the table’s .ibd file, drop the table, and then remove the link.

Just a cool little trick I learned from our DBA that, for some reason, never saw written down anywhere.

Say you want to share the encrypted data somehow. What do you do?

- Using NFS to serve the encrypted dir(and then unencrypting on the client-side) proved very unstable. After some heavy data transfers, the mountpoint simply disappeared.

- Using NFS to serve the unencrypted dir(after being mounted by ecryptfs) proved… impossible, at least using the NFS server in Ubuntu.

- Using sshfs to serve either the encrypted or unencrypted dir proved unstable as well.

- Using smbfs/cifs was out of the question, as I depended on some unix permissions for some things. And I am not keen on sharing data between linux computers using smb/cifs.

- Ecryptfs, when used to encrypt file names, has issues with long file name paths. Some of my deep structured dirs and files were lost because of this.

- Not exactly a fair reason to judge this, but double ecryptfs encryption(that is, having ecryptfs encrypted data inside an ecryptfs unencrypted partition) proved a disaster(yes, I am a moron). After a week of usage my double encrypted data were corrupted as hell. Although I didn’t try to investigate it further, it seems like it was happening slowly as well, bit by bit.

After thinking it through I decided to go ahead and migrate everything to a pure block based encryption. I chose luks.

Yes, block based encryption is not so flexible as ecryptfs, but apart from the impression I have that it feels faster, it is network friendly as much as any other block based storage device.

If you plan on sharing somehow that encrypted data of yours, perhaps it’s best to avoid ecryptfs.

I was reading a technical article offering some insights about a technology I am unfamiliar with. It had many unknown, for me, keywords, but fortunately each one of them was linked to other appropriate articles around the web.

Since the article was old, while clicking some of the links I discovered that most of them were dead, either because they expired or because they were hosted on a currently unreachable server. So I went on to search for the keywords by myself, using Google.

I’m sure the above doesn’t sound too interesting but bare with me.

The web, with its ever-changing nature, is even now filled with dead links. Keeping the links updated is no small task when one has to maintain lots of information. Wikipedia seems to be doing a pretty good on this topic but they are powered by an army of volunteers. What about the rest of us?

Entities that know all of the current best links and information pages for any given topic do exist. They are called search engines.

So what could change? Why not always link to a search, instead of a direct article?

For example, if someone would use an article about the economic crisis of Greece as an argument to their blog, why not link to a specific Google query and let Google sort out the best article on the topic?

I don’t know if what I’m proposing is viable, but I do know that with the web evolving faster and faster each day, the dead link problem becomes more obvious as time goes by.

The consequences of using dynamic links are enormous.

First of all, SEO rank information based on static links between them. If one would replace static links with dynamic ones, then there would be no way, with current methods, to rank information. So while dynamic links can be useful, they can only be partly implemented, as static links must always be there, somehow. Either this or search engines need to find another objective way to rank information.

Second, if instead of linking directly to a site we start linking to search engines queries instead, then we give over even more power to search engines to decide about the information flow. That can’t be good as diversity should always be available. Perhaps a dynamic link could compare results from different search engines and combine them to make an educated guess about the best possible result. This could mean that meta-searching will be back in some form.

Third, and last, is the context matter. The economic situation of Greece is bad and any article currently will reflect that. Tomorrow, however, is another day and the same query will probably fetch different results, depending on the situation of that time. The link could become obsolete and although it will fetch information, it may not be the intended information at all.

Search engine personalization is all about fetching information that we specifically want, not something most of the people want. If dynamic links become the norm, search engines will need to analyze the source article of a query and perhaps tweak the results to be as relevant possible.

The dynamic link approach may actually ridden the world from dead links and that can only be a good thing. Most of us technologists imagine the future as an era where information will always be available effortlessly and instantly when we needed, being both accurate and in context.

Well, does anyone imagine struggling with overcoming dead links in this picture? I don’t think so.

After that, I just created symbolic links in /etc/cron.d/ pointing to the NFS share and everything seem to be working perfectly.

Or so I thought.

It seems that cron is a bit sensitive on file availability. If, for some reason, a symblic link becomes dead temporarily, because the NFS mount lost temporary connection to the NFS server, then cron will ignore that configuration file, even after the connectivity of the link is restored.

So basically, cron started skipping some jobs. Restarting it solves the problem, but only until the next time it happens.

Weird. I reverted to the old scenario, where are of the cron files are local to the server using them.

It happens on Ubuntu 10.04 for sure.

This not new stuff, anything you read here will probably strike as obvious to any fairly current linux system administrator.

However, this was done in 2003 and nothing similar then, nor puppet or cfengine was available at that time. As far as I know, it was an original and unique concept. Please correct me if you know better.

I kept it for myself, not for profit but of pure laziness. Hopefully, it can still do some good, although I doubt it.

Configuration Unification Project
Initial Project goal: Provide a common storage medium for all common settings between several different and commonly disconnected Linux servers.

Problem: I had several different servers, each dedicated to specific tasks. However, most of them had common settings. If I wanted to change something network wide, I had to manually replicate the change by hand to each server separately.

Project requirements:

• The common set of files should be accessible even when the server is offline
• Multi-master configuration. Every server with a copy of the common set of files should be able to replicate any change made to any one else.
• No common or shared physical storage between servers. Each server would be located in different geographic locations
• The set of technologies used should be accessible to any fairly recent Linux distributions. Compiling and custom kernel modules should be avoided.
• The method should not require separate physical partition on the server. It should  be deployed on current production servers without much difficulty.
• Lightness. The method should only replicate the changes and nothing else.

1st stage: Evaluate available technologies

Evaluation was done on technologies such as NFS, rsync, DRBD, GFS and other filesystems but none of them satisfied all of the requirements.

Note: This was done 2003 at a time when where no puppet, cfengine or other similar free technologies existed or were stable enough for enterprise use. As far as I know, no similar project existed at that time.

The implementation was finally based on three separate and freely available programs:

• DSH stands for distributed shell. It’s a system for sending identical commands to many Linux servers at once. It’s secure, as it can be based on SSH.
• Subversion is a Revision Control System. Its job to store a file hierarchy along with any changes committed to it in the future. Since it stores only the changes, it is perfect for distributing changes very efficiently.
• Cron is the system task scheduler found in any Linux distribution.

2nd stage: Implementation

There was a central SVN repository somewhere, reachable by every other server. The location of the server was found by the clients via a DNS alias, so moving the repository elsewhere on demand(for example after a server failure) was very easy to do and no client reconfiguration was necessary at all, other than a simple DNS alias update.

After each new Linux server was introduced to the network, it run a script where it replicated the common set of files from the SVN repository locally, so each new server had each own copy of the current state of the common set of files in a simple local directory.

On regular configurable intervals, but specifically random for each client to avoid conflicts, each client would check via cron its version of the common files. Depending on the change found, it would either replicate the local change to the main SVN repository or introduce the change found on the repository to the local copy.

After the repository was changed, each client would replicate the change locally in due time. If the change was urgent, there was an option where the client, using dsh, could instruct all the others to wake up and update their copy as soon as possible.

Some changes to configuration also required the restart of their respective services, network wide. With DSH and the proper grouping techniques it offers, I was able to massively restart simultaneously at the same time every related service in the network with just one command.

3rd stage: Configuration Versioning and Service Redundancy

After the above was completed, I realized that I accidentally, as a consequence of using an RCS for replication, I also had a full history of all the changes of every common configuration file in the network.

In short time, I added every server’s configuration, either common or private to the repository. So at any point in time I was able to revent any kind of change that introduced a problem. Specifically, I was able to troubleshoot any new problem more easily, since I was able to know at once whether the problem could be related to a recent change of a neighbouring service.

Furthermore, moving a service from one server to another was then a matter of minutes. All one had to do is install the service’s software and then create a symbolic link for its configuration files to the common file repository.

4th stage: Cross-platform Configuration Backup.

Given the benefits of such a system, I started writing modules that would dump configuration of the common filesystem of other devices as well, non Linux ones, like access points, switches, routers, network printers, network cameras, windows services, and even the Active Directory itself.

I wrote modules that would export configuration of any kind of a network device or service that would run periodically and dump information as files to the common filesystem. Since the system keeps every change made, I ended up having the configuration of hundreds of different network devices at literally any point in time.

Extra attention was given to the format of the file, as it should be something easily imported back to the device, in case it needed a configuration restore.

5th stage: Alerting

After having a common repository for every change, it was easy to monitor for changes in just about anything. I wrote a module where it would monitor for changes in critical files, such the as the one representing the configuration of a router. If a change was detected, it would sent an e-mail with a side by side comparison of the old and the new configuration file, highlighting the change made. This way, I and other administrators could monitor the changes of every critical device in the network and act in the event of an unauthorized change and just be informed about the authorized ones.

6th stage: Migration to Puppet

I’m keeping the system up to date, by writing scripts than can dump configuration information from any kind of network device. The language of this script varies, depending on the situation. Up until now I’ve used technologies such as SNMP, LDAP, expect, Perl, Python, Bash, PHP and so on.

I am currently in the process of evaluating puppet for integration into the system, for the part of system configuration enforcement and deployment.

It seems that the National Bank of Greece, the largest financial organization of the country, has a VERY STRICT password policy, in regards to their Internet Banking.

And by strict, I mean that one is not allowed to choose a password with MORE than 5 characters.  In addition, one is not allowed to use symbols at all, only latin characters and numbers! This is a screenshot:

Your new Password should have 5 characters and consist of numbers, letters (only Latin) or a combination of both. In addition, we inform you that if your password remains unchanged for a period of two months, the system will ask you to change it.

No, these are not the minimum password complexity requirements, these are the limits!

*sigh*…

If only my account there wasn’t mandatory by my employer…

Under normal circumstances, one would download Oracle’s InstantClient with the SDK component, and then install the OCI8 php module via PECL.

However, this isn’t the case, thus the reason for me posting this. Oracle’s InstantClient 11 doesn’t support connecting to such an old database version:

ORA-03134: Connections to this server version are no longer supported.

The oldest client Oracle currently distributes is version 10, which, although I found some references to the contrary, doesn’t work either.

So what I did is install IIS and PHP on the Windows Oracle Server. Since Oracle’s DLL’s where there for the taking and since windows PHP is distributed in binary form(including the oci8 module), all would be OK.

However, the only remaining problem was that the latest PHP was compiled against the latest Oracle Client too. So, I downgraded my PHP version by version, until I found the one that worked. It seems that PHP 5.0.4 is the winner.

So, the bottom line is that if you want PHP to connect to an old Oracle database, use an also old binary PHP distribution that includes the binary oci8 module. How old probably depends on your database.

Three times it failed me, leaving my servers unable to boot, stuck just after readeahead’s messages, without any error message whatsoever.

At first it didn’t like my nfs fstab entries. So I removed them and mounted them after the boot.

Then it considered that my external usb hard disk(which I had auto mounted in fstab) missing is something so critical, that it didn’t continue booting.

Today, after a bios reset(and therefore bios datetime reset), it still wouldn’t boot. I found the message “mount time of this partition is in the future” so I corrected the time and only after that did my system boot.

In all of the above occasions, which happened on three different servers, the boot process would halt, without any explanation.

Way to go, Ubuntu…!

So, if you find your system unable to boot, do yourself a favor and strip your fstab to the absolute minimum and debug the boot process around mountall…

(Πληροφοριακός) Κομμουνισμός: Το καθεστώς όπου όλες οι δικτυακές συσκευές έχουν περίπου το ίδιο uptime. Και αν κάποιοι πάμε να ξεφύγουμε, έρχεται η ΔΕΗ να επιβάλει την ισότητα και μας γυρίζει όλους στο 0…