smetana.name

letsencrypt.sh on CentOS 6

2016-02-25T00:00:00+01:00

Let's Encrypt was difficult not to notice of. It's a new certificate authority providing free certificate signing for everyone. And on top of it the whle process of registration, certificate signing, renewal and revoking can be scripted and automated by the CA provided client tools. The tools can even automatically reconfigure the most popular web servers (Apache and Nginx). It has a drawback though: The client itself is quite a big pile of Python scrips with heavy dependencies. Moreover the Python 2.6 support is marked experimental. Not quite perfect match to CentOS 6.

This server actually runs on CentOS 6. It's not only because it was the latest CentOS available when I first installed it. I still keep the system since RHEL-7 and therefore CentOS 7 dropped the 32bit x86 support. Given the machine is a cheap rented VM with little disk space and memory, the 64bit architecture would be a really wasteful choice.

The good news is that Let's Encrypt builds on open ACME protocol and the smart people on the Internet came with alternative clients requiring much fewer dependencies and resources.

The one that looked to be the one suitable for my underpowered ancient server seemed to be letsencrypt.sh. It's a Bash script requiring only curl and openssl. And the good news is: it really works and getting it to work is actually pretty simple.

For those who would want to give it a try on CentOS/Fedora: I've build rpm packages for myself in COPR (COPR is quite broken these days: the builds might show as fail though the packages are there). Thepackage contains just a few files:

/etc/letsencrypt.sh
/etc/letsencrypt.sh/config.sh
/usr/bin/letsencrypt.sh
/usr/share/doc/letsencrypt.sh
/usr/share/doc/letsencrypt.sh/LICENSE
/usr/share/doc/letsencrypt.sh/README.md
/usr/share/doc/letsencrypt.sh/domains.txt.example
/usr/share/doc/letsencrypt.sh/hook.sh.example

Obviously, the main /usr/bin/letsencrypt.sh script and its configuration /etc/letsencrypt.sh/config.sh. There are several configuration steps required. First, edit the /etc/letsencrypt.sh/config.sh. First, the default CA URL is set to https://acme-v01.api.letsencrypt.org/directory. For the testing purposes it's wise to point it to a staging API instance at https://acme-staging.api.letsencrypt.org/directory. This will issue invalid certificates but doesn't have that strict quota. Choose a BASEDIR value: this is the directory where the domains configuration file would have to be created, where you would store your private key and also the script would store the ceritificates obtained from the CA. So /etc/letsencrypt.sh/ is a reasonable choice.

The other important thing is the WELLKNOWN directory. That's where the ACME server would look for the HTTP challenge files (see the How it Works document). The script should have write permissions to the directory.

Then set a path to the private key (PRIVATE_KEY) used to identify your Letsencrypt account: you may need to remove it when switching from the staging API to the production one.

The last mandatory field in the config.sh is the self-explanatory CONTACT_EMAIL...

Now for the domains you want to have the certificates issued for: create a BASEDIR/domains.txt file with a simple format:

domain.tld server.domain.tld another.domain.tld
nextone.tld andanotherserver.nextone.tld

The important thing: each domain has to be accessible over HTTP and serve the challenge file: make sure to set up the web server accordingly: unlike the oficial client, letsencrypt.sh will not do that for you. The staging API is really great to test your setup.

Once you think you're done:

letsencrypt.sh -c

...and watch the logs. If things work, just create a monthly cron job and enjoy. If not... well, you wouldn't be doing this not knowing the drill already...

All in all: letsencrypt.sh rocks. Thanks to the author.

Lenovo Carbon X1 and Fedora

2015-12-14T00:00:00+01:00

My employer wants us to re-new out-of-warranty hardware. This includes laptops. Till the last week I was using Lenovo T520: heavyweight, battery sucking beast that could have been used as a melee weapon... When it went out of warranty and I was supposed to have it replaced..., I waited. It was the time when some of the more misantropic designers at Lenovo succeeded with the idea of removing physical buttons from the trackpoint (the only pointing device I'm able to use on a laptop -- touchpads are simply out of my motoric abilities). And I didn't like the big laptop anyway. I liked my old X61. I wanted that. Just faster, newer.

Then there was the Carbon X1: razor-thin, light, powerful. And even more unusable because apparently removal of the physical mouse buttons didn't make the users suffer enough. The most expensive Lenovo ultrabooks line had also one of the keyboard rows replaced by some software-controlled touch strip that most of the marketing victims never figured out how to use. Brilliant move by Lenovo. I either don't like people much.

When guys at Lenovo had enough of their fun, they decided to go back to doing business and producing laptops people wouldn't want to kill them with (the only reasonable thing those machines could have ever been used for).

So... Thinkpads x50 looked much better. Obviously some of the business-sustaining, cost-cutting decisions had to be made: soldered RAM (memory slots are obsolete), clickable touchpad (I know, Apple knows how to make them, sadly this is about Lenovo), new RealTek NICs (RealTek is not that bad as PowerVR or Mali, but still...) and of course: firmware! Still this stuff seemed to be usable at least.

But I was tired of heavy T500s and their big dust-catching docking stations with so many connectors. All in all, I need two monitors, two USB ports, microphone and headphone and that's it. It's 2015, these things Just Work these days. And I wanted the pretty and shiny stuff: Carbon X1 in its third generation. Third! They had to perfect-out that thing. It's going to be awesome, trust me. Let's unpack it.

When it came to the laptop itself, it looked OK(ish...). Apart from the fact the screws are some 0.5 mm longer than the holes, so the case is loose and rattles. Later I found out everybody I knew using this machine has tried to tighten the screws and all of them failed. The screws are simply too long. Haha! My admiration of Lenovo designers should be probably extended also to their QA. Of course, the RAM is soldered on the mainboard (8 GB in my case) and of course there is the Apple-like keyboard (the old Thinkpad keyboard I have on this old, wonderful, sturdy, perfectly designed X61 I'm typing this post on, was the best thing ever; even the Apple folks could envy you!, fools at Lenovo; I know... it would spoil the fun). And of course, there was some "invention" put in the keyboard layout, so people have something to get used to (and again: X61 is thiiiis tiny... and still... so much better keyboard). My almost 20 years old habit of Alt-F1 to Alt-F4 combination to switch between the four desktops is ... not possible. Had been turned to Alt-Fn-F1 to Alt-Fn-F4!? Thankfully, the trackpoint is there and moves the cursor down really well... I want it to move up too sometimes though... Let's use the force... OK, that works. Hm. Whatever. I drive Citroen, I'm used to rattling and violence. But otherwise, everything is comfy and never breaks. (And my Citroen was CHEAP!)

Installing Fedora 22 is a boring business. I had been warned not to install F23 to avoid problems with Wi-Fi stability... Anaconda can't install over wireless anyway. Never did, even on X61 or T520. Just won't connect. Hidden SSID or enterprise WPA... All wrong. Must have been designed that way (fun!). There's some dongle where one can plug the RJ45 connector. Fits here, yes. No surprises. Fedora installed.

It boots, I can log in. With minor violence applied on the trackpoint thingie I can choose "Shut down" from the XFCE menu. Almost like the old ugly T520. Just more silent and slightly faster (though I expected more from the SSD). Most of the time the laptop would be docked connected to much more convenient LCD panels (two of them: I got used to write longer texts on a rotated screen).

Let's plug in the dock connector (Yes. The evolution: the ThinkPad OneLink Pro dock is not a dock, but yet another box with a thick cable.). It boots and I can see the bootloader prompt on the laptop LCD but nothing at all on the external monitors. Turned out it takes several seconds for the external screens to turn on. There is even an option in EFI/BIOS to configure delay on boot to give the docking station chance to turn the monitors on. All the systemd violence that saved few precious seconds on boot in vain. Lenovo doesn't like even the systemd developers. At least, no misantropist conspiracy here.

The image is mirrored using the lowest common resolution of the detected monitors on both of them. Odd. Let's log in and tell xrand to fix this: turn the laptop LCD off, and put the two external monitors next to each other. Now there is no image on the larger panel and the smaller one flickers beyond readability. No fiddling with xrandr fixes this. The only possible combination is one external monitor plugged to the dock and the laptop screen. I'm getting really upset.

It turns out the "dock" is not what one would expect: It's basically an USB 3.0 hub with some devices hard-wired to some of the ports. So with the dock one gets and USB graphic card, USB NIC and USB sound card. Unplugging it makes the devices disappear: very funny with the soundcard for example.

And the USB NIC: marvellous piece of hardware: took me quite some time that when the thing decides to stop detecting carrier on the ethernet, the only thing that helps is reloading the r8152 module. No, even reboot won't help. Reading the other poor "ThinkPad OneLink Pro" users comments on Lenovo site I suspect this is simply crappy hardware/firmware. NFS has become an unexpected adventure with the new Lenovo hardware. Bet your data on it.

In the end the only undeniable improvement is much longer battery life (some 10 hours). It's not that big concern to me though... There power plugs in the meeting rooms and I travel really rarely. The rest is just pure downgrade in every aspect with the "dock" being definitely the biggest disappointment.

The length of this post is proportional to my frustration. Given the official price tag on the Carbon X1 and the unbelievably crappy "dock"... Just keep your old ThinkPad x20/x30 and just ignore this marketing scam. My next laptop is going to be Mac. And Microsot has also produced very good hardware, they make even laptops nowadays..

Agilia Conference Day 2

2015-03-27T00:00:00+01:00

I was quite happy about the first day of the Agilia conference as you might already know. And was looking forward for the second one.

I'm not sure what is your experience with keynotes. Quite frequently those are the least interesting talks at any conference. And I'm afraid to report Agilia was not an exception to the rule. The keynote's title was "Attracting talent" however most of the talk was about building office spaces. I was not really convinced this was related to the title in any way... And I found the talk unusually boring. Perhaps I was not the right audience.

However the second one proved to be quite different: Zsolt Kulcsar shared his views about building trust within companies. Mr. Kulcsar, albeit graduated psychologist, works for another small web development company. And this one might be pretty unusual. It was the first time I've heard someone saying the company's top priority is their employees. The talk was focusing on the evolution of the organizations from the tribal gangs ruled by a single despot through modern profit-oriented corporations and post-modern people and innovation driven companies to the organizations resembling a living creature where each member functions as a vital organ. He talked about how their firm tries to capture moods of the employees, lets them get to understand their customers as perfectly as possible and maintains very high level of transparency both internally and also towards the clients. Interesting and even inspiring talk.

Jordi Falguera and Victor Firén talked about the Behaviour Driven Development methodology and demonstrated practically part of the development cycle as they use it in GTech. This was the most technical and practical talk of all the conference. The speakers talked not only about the BDD supporting framework and the tool GTech developed for themselves but also about the practises that worked for them the best: like do things right from the beginning to prevent the technical debt to pile up and become unmanageable later.

Roland Tiefenbrunner spoke about his experiences as a founder of two startups and scrum master in other one. Here's some of the ideas I found interesting or important:

Use hypothesis instead of requirements; invest in experiments
The customer should pay even for the minimum viable product
Get first feedback ASAP
Focus on solving the problem, not the solution itself
Use cohort analysis and actionable metrics
Decide what to build before building it

We're almost at the end: Arne Åhlander's Becoming a successful product owner was a set of good advices to the product owners -- what tools and techniques can help them in their quests. Arne suggested to use a playbook to keep information about the product, stakeholders, roadmap, backlog, but also the product vision and definitions of "ready" and "done". The important part for presenting ideas is their visualization (canvas). The products should make customers happy, therefore it is important to to find out what are the customer's jobs, pains... It is also important to realize what is my "unfair advantage", what do I have the others can't. And making use of it.

The conference's grand finale was Angel Medinilla's Change management presentation. Even though it was not too new for anyone who went through some related soft-skills training (like me): it reiterated the idea that to make a change one has to first convince the others that status quo is unbearable, make them realize the pains the change will relief and move slowly step by step to the goal. Angel is a showman and turned the talk into a wonderfully entertaining event. And it was also a very suitable topic for the last talk.

The Takeaways

One big thing: agile is not a software development method. At least not only. It is an organization method: of anything. Most notably it has become an standard for not only the small and quickly moving start-ups but also for large companies who realize they need to adapt to survive.

I have heard several recurring themes in the talks:

Companies are here for the people not the other way round
Focus on customer, employees; during the whole conference I don't remember hearing the words "shareholder" or "investor"
Repetition and routine are necessary for doing things right; it must be a habit, it must make people uncomfortable breaking it
Get to know the people you work with: learn about their goals, motivations, fears; and help them to achieve what they want

And again: much of those may look like clichés any corporate employee hears and sees in their internal company communications way too often. The context of agile organizations however emphasises why is it necessary to keep them on mind. "The people are our greatest asset." Which of the corporations doesn't trump something like this? I would guess only minority of those company employees really feel that way and perceive such statements as propaganda that somewhat belongs to the territory. The agile organization cannot afford such sentiment to grow among the people. The agile methods bring the results at a cost: they put more stress on people -- they have to operate with more frequent deadlines, the requirements and environment change more often, even the teams themselves can re-arrange more often... To sustain the conditions people have to believe the employer, the work, the colleagues... Creating the environment where agile methods have chance to succeed then takes much more than just train few scrum masters.

Open Source?

Of course I had also some opportunity to chat with random people during the breaks and in the evening party. And of course I have asked about things I work on at job, in my spare time... I live in a bubble where FOSS is standard and the weird thing is actually not to use it. Note one: open source is in some environments only becoming accepted. And quite notably: it is happening "from the bottom": the admins, developers, use Linux and FOSS for their own needs and get it in production by selling it upwards. This means losing mindshare might be quite dangerous these days. The desktop matters even though it is never going to be for the masses.

Agilia Conference Day 1

2015-03-26T00:00:00+01:00

I am involved in agile developed projects at my job. Only very indirectly though: I manage a team working on several projects and two of them use Scrum. And it is a rather new thing around so there was something to learn in the past few months for me. I must say that the Scrum---or perhaps the way I saw it implemented---quite captured me. That much that I'm actually trying it out with the project I'm directly involved in... So, I decided to visit the Agilia conference.

I was hoping to learn about various implementation details of agile methods for different types of projects and teams. And Agilia did not disappoint me in this regard. However I had not realized how much could be transfered from the work organization to the company orgainzation. I have seen some of this in, e.g. Steve Denning's Forbes column and even though his observations fit quite well what I see around myself, I was rather sceptical about the solutions Mr. Denning presented: and quite often it was some form of agile organizations. At Agilia I had a chance to meet people who already implemented some of the ideas and who were able to describe the change they brought and obstacles along the way. This was the biggest (pleasant) surprise: I learned not only how do the modern companies approach the product development but more imporantly what actually makes them "modern".

The conference was opened by a keynote by Mario Almondo, the COO of Scuderia Ferrari (the F1 racing team). Mr. Almondo spoke about culture and its necessity for a success. The talk was of course attractive largely by the fact the speaker was part of a true legend: only few people can refer to Jackie Icks or Michael Schumacher as colleagues. And yet another difference: How many sport or automobile analogies have you heard in software development? Now, this speaker was really talking about cars and races. And software and IT companies popped up as the "other part" of the analogy:

Winning a race is not enough to win the championship: one time success doesn't matter that much as the ability to repeat it.
Doing things right must be a habit, a routine. Realizing what things should be like is not enough. Doing them right once is useless.
To build a team one must know the people: their past, their ambitions, their goals.
We don't want just to win over the competition. We want to innovate for the customer.
If the top management does not care, the team cannot win.

I realize how these may sound as clichés but really it was the speaker's personality and the different context that augmented their actual importance. And clichés become clichés quite often for a good reason...

The other talk I have attender the first day was presented by Anna Danes Boix who was working for a completely distributed web development company and spoke about the challenges it brought and ways to overcome them. Distributed in this case means 228 people living in 25 countries, who never meet in person: the company effectively has no travel budget (some 400 Euros spent on travelling in the past 6 years) and still the company is profitable and maintains really low turnover rate. Anna spoke about the necessity to focus on the people: First the clients, second the employees. And of course not everybody is capable of working in such a team: the technical skills are necessary but not enough, it really needs people who are able to communicate, appreciate quality and strive for it. And of course the company needs to treat the people so they feel important and engaged: the company exists not only to make the clients to reach their goals but also for the employees to reach theirs. It is necessary for everybody to understand their role, people communicate frequently and openly, status reports need to receive feedback and another interesting thing: leaders are never hired from outside. And again: doing things right must become a habit and habits are developed by repetition.

Bjarte Bogsnes' Beyond Budgeting was perhaps the "highest-level" talk. There is quite a lot of theory behind how to allocate company budgets and how not to do that. The traditional way is to sit down once per year and put some money in this bucket, some money in that bucket... The speaker works for Statoil: one of the biggest oil mining companies in the world. (And he works there since 1983.) Still he is driving changes in the company: like budgeting. Statoil approaches the problem in an unusal way (for a corporation): dynamically as the requirement from the projects come and change. Yes, "predictability" is apparently not a priority for the company. However creating environment that allow the employees to succeed is: so they are given right to decide what to spend the company money on... For example the travel budget is totally open---anyone can take as much they want from it---but also totally transparent: everybody sees the others' spending. No, the employees didn't ruin the company, the overall travel spendings went actually down...

The next one was Philip Eisbacher's talk of how to use agile technique to develop the company itself and how to actually organize the teams to be able to do so. The core of the idea was to allocate some sprints just for the company's needs: improving processes, tooling, implement new ideas..., and be actually transparent about it to the customers.

And maybe I'm overly sensitive to this but it was not the first time I witnessed people saying something like: "It was open source so it was not that good...". It feels odd especially when the other tools the same person refers to (inherently as "good") are open source too; in those cases the openess apparently doesn't matter.

Jaroslav Prochazka then talked about the transformation of the industry, about the shift from creating products to providing services. As all the other talks this one emphasised the necessity of delighting the customer and building long-term relationships. This requires all the developers and all the team members to be aware of the overall product and how does their contribution fit into the final outcome and why it's necessary. Jaroslav suggested a way to organize the teams more efficiently and what roles were necessary for this success. I have to admit that much of the talk I felt like there was too much fill-in and the core idea itself didn't seem that novel to me.

And the day was closed again by the stories from Ferrari F1 team: their former product manager, Enrico Lombardo, talked about breaking the resistance when introducing new ideas to the actually quite conservative team. It is not possible to actually capture the atmosphere of the talk. Mr. Lombardo is really great story teller and I really think inviting the Scuderia Ferrari people to talk to the mostly IT guys was a great move from the Agilia organisers.

Overall my impression of the conference was very positive. Given I came there with a lot of scepticism and the conference was actually almost non-technical, I was after all quite impressed. And there were several perpetuating themes in most of the talks. I'll get to them next time.

Comments!

2014-12-11T00:00:00+01:00

As and excercise I've decided to implement some way to add comments to this site's content. I know how many visitors do I have here so I don't expect much interaction with the readers... However the experience with Flask and JQuery was definitely worth it. And I was surprised how little code was needed for creating something actually usable.

Next on the list is a gallery. I'm not sure when will I get to it but I do have quite solid idea of what I want to implemet. And even know how.

Sharing DVD drive over iSCSI

2014-10-20T00:00:00+02:00

The old SCSI target utils shipped with older Fedoras and RHEL-6 never really worked for one of my use-cases: sharing optical drives. I don't use them too extensively however sometimes I need to burn a CD/DVD or watch a DVD video. This was not really possible before LIO and it's pSCSI plugin (the old iSCSI target daemon required to know backend block device's size and didn't suport removable media). I didn't feel like equipping each of my computers with an optical drive. It was not even possible for all of them.

These days it's easy to set up. There is a good documentation on the LIO wiki. There are slight modifications required with Fedora 20.

Please note: The pSCSI backstore is generally unsafe and not recommended to use. There are features that won't work in all scenarions and things will break (the backend device does not implement all the SCSI commands that the initiator may issue).

Target configuration

On the box where the optical drive to be exported is physically present (the target) install the targetcli utility:

[root@localhost ~]$ yum install targetcli

Assuming the DVD drive is at /dev/sr0, here's the complete guide for "exporting" it as an iSCSI device to the network.

There is not much difference from the iSCSI guide on the LIO wiki. As root, start targetcli and then:

Create the pSCSI backend:

/> cd backstores
/backstores> pscsi/ create name=dvd_backend dev=/dev/sr0
Note: block backstore recommended for SCSI block devices
Created pscsi storage object dvd_backend using /dev/sr0

Now instantiate the iSCSI target:

/backstores> cd pscsi/dvd_backend
/backstores/pscsi/dvd_backend> /iscsi create
Created target iqn.2003-01.org.linux-iscsi.zaphod.x8664:sn.a5f5c643cc7e.
Created TPG 1.
Created default portal listening on all IPs (0.0.0.0), port 3260.

Note that the default portal has been created along with the iSCSI target. By default it listens on all the interfaces and port 3260. This is actually what I want (and check the iptables settings...).

Now I need to export the LUN:

/backstores/pscsi/dvd_backend> cd
/iscsi/iqn.2003-01.org.linux-iscsi.zaphod.x8664:sn.a5f5c643cc7e/tpg1/
/iscsi/iqn.20...c643cc7e/tpg1> luns/ create /backstores/pscsi/dvd_backend
Created LUN 0.

Configure authentication/authorization for the portal. I'm on a local network, trusting everyody, so I open the unrestricted access for everybody:

/iscsi/iqn.20...c643cc7e/tpg1> set attribute authentication=0 \
demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
Parameter demo_mode_write_protect is now '0'.
Parameter authentication is now '0'.
Parameter generate_node_acls is now '1'.
Parameter cache_dynamic_acls is now '1'.

The resulting configfs tree should look like this:

/iscsi/iqn.20...c643cc7e/tpg1> cd /
/> ls
o- / ............................................................................. [...]
  o- backstores .................................................................. [...]
  | o- block ...................................................... [Storage Objects: 0]
  | o- fileio ..................................................... [Storage Objects: 0]
  | o- pscsi ...................................................... [Storage Objects: 1]
  | | o- dvd_backend .............................................. [/dev/sr0 activated]
  | o- ramdisk .................................................... [Storage Objects: 0]
  o- iscsi ................................................................ [Targets: 1]
  | o- iqn.2003-01.org.linux-iscsi.zaphod.x8664:sn.a5f5c643cc7e .............. [TPGs: 1]
  |   o- tpg1 .......................................................[gen-acls, no-auth]
  |     o- acls .............................................................. [ACLs: 0]
  |     o- luns .............................................................. [LUNs: 1]
  |     | o- lun0 ....................................... [pscsi/dvd_backend (/dev/sr0)]
  |     o- portals ........................................................ [Portals: 1]
  |       o- 0.0.0.0:3260 ......................................................... [OK]
  o- loopback ............................................................. [Targets: 0]
  o- sbp .................................................................. [Targets: 0]
  o- vhost ................................................................ [Targets: 0]

Now, save the configuration so it doesn't get lost.

/> saveconfig
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json

The machine now "exports" its /dev/sr0 DVD drive over iSCSI with complete SCSI command pass-through.

There is one catch however: On Fedora 20 there is no unit file that would ensure the configuration is reloaded automatically on boot. This means the iSCSI target will be inaccessible after reboot until you run:

[root@localhost ~]$ targetcli restoreconfig

So... TODO: create a systemd unit file that would run that on boot.

Initiator configuration

Now I have the DVD device accessible over iSCSI. Let's configure the initiator (the machine which wants to connect the remote drive). Again, install the necessary packages first:

[root@localhost ~]$ yum install iscsi-initiator-utils

Assuming the iSCSI target machine's IP address is 192.168.122.1, here's how to discover and login to the target:

[root@localhost ~]# iscsiadm --mode discoverydb --type sendtargets \
    --portal 192.168.122.1 --discover
iscsiadm: Could not scan /sys/class/iscsi_transport.
192.168.122.1:3260,1 iqn.2003-01.org.linux-iscsi.zaphod.x8664:sn.a5f5c643cc7e
[root@localhost ~]# iscsiadm --mode node --login
Logging in to [iface: default, target: iqn.2003-01.org.linux-iscsi.zaphod.x8664:sn.a5f5c643cc7e,
portal: 192.168.122.1,3260] (multiple)
Login to [iface: default, target: iqn.2003-01.org.linux-iscsi.zaphod.x8664:sn.a5f5c643cc7e,
portal: 192.168.122.1,3260] successful.

This created /dev/sr0 device node as well as /dev/cdrom symlink to it. Now I can play video DVDs and audio CDs remotely, even burn optical media and do whatever a "normal" optical drive would let me to.

Please note: the initiator configuration is actually persistent. That means the system will try to connect the iSCSI device on the next boot.

In the case one decides to remove the remote device, just logout of the session and remove the node. Again this is a persistent change:

[root@localhost ~]# iscsiadm -m session -u
Logging out of session [sid: 1, target: iqn.2003-01.org.linux-iscsi.zaphod.x8664:sn.a5f5c643cc7e,
portal: 192.168.122.1,3260]
Logout of [sid: 1, target: iqn.2003-01.org.linux-iscsi.zaphod.x8664:sn.a5f5c643cc7e,
portal: 192.168.122.1,3260] successful.
[root@localhost ~]# iscsiadm -m node --delete

Extracting subtitles from Matroska video files

2014-10-02T00:00:00+02:00

Subtitles embedded in a video file are essentially a sequence of images overlaid over the video frames during the playback. To convert them to a text file one has to first extract the subtitle images themselves and then use some OCR mechanism for the image to text transformation.

On Fedora the procedure requires to install some packages. First, enable the rpmfusion repositories if you haven't done so yet. (Side note: livna repository is not dead yet and you may need it for some DVDs. Enable livna too.) Then install the following packages using yum or dnf:

mplayer
mkvtoolnix
subtitleripper
tesseract

The following should work on Matroska containers. I have a video in the MyFilm.mkv file: there are several tracks in it: video, several audio tracks and subtitles. I found MPlayer to be the simplest tool to identify which track to work with:

[tom@localhost ~]$ mplayer MyFilm.mkv
MPlayer SVN-r37150-4.8.3 (C) 2000-2014 MPlayer Team
Playing MyFilm.mkv.
Cache fill:  0.00% (0 bytes)

libavformat version 55.19.104 (external)
libavformat file format detected.
[lavf] stream 0: video (mpeg4), -vid 0
[lavf] stream 1: audio (vorbis), -aid 0, -alang jpn
[lavf] stream 2: audio (vorbis), -aid 1, -alang fre
[lavf] stream 3: audio (vorbis), -aid 2, -alang eng
[lavf] stream 4: audio (vorbis), -aid 3, -alang ger
[lavf] stream 5: audio (vorbis), -aid 4, -alang jpn, Audio-Kommentar
[lavf] stream 6: subtitle (dvdsub), -sid 0, -slang fre
[lavf] stream 7: subtitle (dvdsub), -sid 1, -slang eng
[lavf] stream 8: subtitle (dvdsub), -sid 2, -slang dut
[lavf] stream 9: subtitle (dvdsub), -sid 3, -slang ger
[lavf] stream 10: subtitle (dvdsub), -sid 4, -slang pol
[lavf] stream 11: subtitle (dvdsub), -sid 5, -slang fre, Audio-Kommentar
[lavf] stream 12: subtitle (dvdsub), -sid 6, -slang eng, Audio-Kommentar
[lavf] stream 13: subtitle (dvdsub), -sid 7, -slang dut, Audio-Kommentar
[lavf] stream 14: subtitle (dvdsub), -sid 8, -slang ger, Audio-Kommentar
[lavf] stream 15: subtitle (dvdsub), -sid 9, -slang pol, Audio-Kommentar
...

Alternatively you can use mkvmerge -i MyFilm.mkv to obtain the list of tracks. The advantage of MPlayer is that it also tells you what language is a particular subtitle track for. With mkvmerge you have to guess.

Assume I'm interested in the English subtitles from the example above. MPlayer tells me it's the track (stream) ID 7. Let's extract them using mkvextract:

[tom@localhost ~]$ mkvextract tracks MyFilm.mkv 7:subtitles.sub

This produces two new files: subtitles.sub (VOBSUB format) and subtitle.idx (separate text index file). Now I can feed the vobsub2pgm utility with the two files. The program takes only the base name without the .sub and .idx suffixes.

[tom@localhost ~]$ vobsub2pgm subtitles title

This produces a lot of PGM bitmaps named like title????.pgm and a title.srtx file that comes handy when putting the final result together.

The next step is to run the images through some OCR. There exists a pgm2txt utility that uses gocr for the task and I found the results very unsatisfactory [1]. I was much luckier with tesseract:

[tom@localhost ~]$ for i in *.pgm; do tesseract $i $i; done

Now we have a .pgm.txt file for each image and we're almost done. The last step is to replace the file references in the .srtx file with the referenced files content. Before I knew srttool I used awk for this, however it's much simpler like this:

[tom@localhost ~]$ srttool -s -i title.srtx -o subtitles.srt

That's it. The subtitles.srt is a text version of the embedded subtitles and you can do whatever you want with them.

[1]	And the sad thing is that I'm the gocr Fedora maintainer...

Starting with a backlog

2014-10-01T00:00:00+02:00

It took several unsuccessful attempts with some CMS systems to actually decide that static web is perhaps the best thing to use for a site with one writer and one reader. Especially since it's the same person.

There are more unsuccessful attempts of various kinds I made in the past year: the "in progress" part of the Coding section is largely incomplete.

TODO

As for the stuff that's on my plate right now and I consider useful/interesting (this list is about to grow I suppose):

Find out how to treat the images and galleries with Pelican
MPDroid still doesn't have multiserver support and the upstream seems to go the way I don't like at all (adding closed-source automatic crash reporting...)
The game I've been writing for ages...
Get Pelican running on Android
Read book list, possibly in Czech

There is another dozen of git repositories I don't plan to ever touch again. Looking at all of what I tried and failed it seems the problem must be in the approach. So, dear myself, will you be able to really get to do anything every day? This is actually important.

Lots of boring fun.