Thinkscape perspective

Apache FLV streaming done right!

2009-02-04T06:42:00.000-08:00

After 2 days of researching, I've just found how to do FLV streaming the right way!
Let's wrap up current (I.2009) situation:

Why?

FLV is still very common container format for videos. h264 is more efficient but is supported by Flash since version 9.115
FLV streaming means you can seek to any position during video, and browser (flash player) will buffer only from this position to the end.
Streaming allows people to skip boring parts or see video ending without loading the whole file (saves bandwidth).
Lighttpd has FLV streaming built in.
Apache doesn't.
There is number of PHP and Perl scripts that allow to implement streaming. These work by loading the given flv file, and pushing it's content from given position to the end.
It's bloated! PHP uses memory, Apache PHP module uses memory, every single streaming connection is a running php script (try empty php file with just <?=memory_get_usage(true)?> )

Behind the scenes

Flash player requests the file, starts loading i.e.: http://host/video.flv
When user clicks any position on the "scrubber" (seek bar) player stops buffering and starts buffering from url: http://host/video.flv?start=1503320
The parameter is byte count from where buffering should start.
Server fetches the file, truncates it and rewrites FLV header.

Solution: flv streaming in apache:

Download apache mod_flv module by Paul Querna or my tuned version.
Install apxs tool (also avaliable in RPMs for most systems as httpd-devel)
Compile and install your module with the following command:

apxs -c -i ./mod_flvx.c
Add the following 2 lines to your httpd.conf or create a dedicated /etc/httpd/conf.d/mod_flvx.conf (path depends on your distribution):

LoadModule flvx_module modules/mod_flvx.so
AddHandler flv-stream .flv
Restart Apache (i.e. with service httpd restart)

Tuning:

I've modified the module to fix content-length and add last-modified headers. I've also added a fix when offset is set to wrong position (i.e. past video length).
You can download the tuned module here.

Things to remember:

To use flv streaming on the web, you need to use a pseudo-streaming compliant Flash player. Flowplayer does the job quite well.
Streaming requires that your FLV has embedded keyframe markers (meta-data). You can inject any FLV with this data using flvtool2:

flvtool2 -U video.flv
Generated streams will work briliantly in Flash player, but you won't be able to play them back in standalone player (VLC, WMP etc.). So don't be suprised when you "wget" a video with ?start=1000 and the file doesn't play. That is because they don't have starting meta-data and players will have hard time deciding which codec to use.

Have fuN!

Troubleshooting mailman - receiving posts, but not sending messages

2007-06-14T10:48:00.000-07:00

After some struggle, I've just solved quite strange problem with mailman. I will share my new knowledge with you.

So the symptoms are:

You or other people are sending emails to the group
They get through your SMTP server (Exim here, with a line in log similar to:
=> list <list@domain.com> R=mailman_router T=mailman_transport
It goes directly to mailman script via pipe - you can verify that (i.e. exim -d -v list@domain.com)
Mailman doesn't send a bit, and nothing appears in it's logs from the moment it receives new message. It doesn't start sending it out, and nothing new is written in smtp.log, post.log, nor vette.log.

We have the following environment:

Fedora 7 system
Exim 4.62 as the mail exchanger (installed from RPM) - probably doesn't matter at all
Mailman 2.1.9 (from RPM)
On Exim, we use mailman_transport, working on pipe driver, executing mailman command /var/mailman/mail/mailman
Local dns server, everything above on this very machine

Resolution:
It turned out to be a problem with localhost dns name resolving. When I tried to use ping localhost it started to ping wrong address. The problem was in /etc/hosts - you have to make sure, that localhost is pointed to 127.0.0.1. It might be a problem in Mailman itself, because no problem was logged. Still - it is quite important to set up proper localhost (loopback) binding.
After fixing hosts, I could ping myself with ping localhost, and mailman magically started to work.

Other steps worth trying when mailman is dead:

Check if your exim routing is set properly: exim -bt list@domain.com - this should result in printing something with "mailman" in it, for example:
list@domain.com
router = mailman_router, transport = mailman_transport
use the delivery debug in exim: exim -d -v list@domain.com This will tell you if the message actually arrives at mailman router and then mailman transport. After running the command, it will stop near Recipients: - there you have to press CTRL+D.
You should look for something similar to:
mailman_transport transport entered
direct command:
argv[0] = /var/mailman/mail/mailman
argv[1] = ${if def:local_part_suffix {${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} {post}}
argv[2] = $local_part
direct command after expansion:
argv[0] = /var/mailman/mail/mailman
argv[1] = post
argv[2] = list

....

mailman_transport transport yielded 0
This means the command executed successfully, and mailman actually gets the message.

Useful pages:

If you still have problems, write me and I'll gladly help.

Best RSS tools for a professionals

2007-06-11T11:47:00.000-07:00

Here is my list of essential RSS tools that will simplify your life. They are supplementary, every one has it's purpose in this world of information :)

Firefox RSS Ticker

small and sexy
non-distracting
does exactly what you expect it to do

Google Reader
So all around these desktop, windowed small RSS readers, here comes web-based RSS reader. It's a little by the wind, but amazingly efficient! It's google... and they did it again. It's advantages are:
- It's on the web, always there when you need it
- It tracks what you have read, no matter if you're in your office, working at home computer or in internet cafe.
- It even comes with you when you are offline in subway or on a plane.
If you happen to work on several machines during a day, it's like a blessing. Give it a try... You might also take a look at some cool firefox addons for Google Reader.
Thunderbird News & Blogs
If you have just found yourself reading 100+ feeds every day, you need a heavy-duty tool! Here comes the RSS functionality of well known "birdy". Just click file - new account and pick RSS News & Blogs.
Suprisingly it has no more functionality than Google Reader above, but is a little faster and smoother when dealing with a large amount of articles and feeds. And it also does a pretty good job as an email client;)
Feedblitz
If you're too lazy or busy to keep configuring, monitoring and playing with the above toys, here's the easiest way to stay up-to-date.
You give them your e-mail address, they send you the news every few hours. It's free, it's convenient and is working like a charm (unless you don't know what email is ;)

What are your favorite RSS toys ?

Why would I need RSS ?

2007-06-09T03:05:00.000-07:00

It's incredible to find out, how many Internet professionals and engineers don't use RSS readers. Many of them don't use RSS at all. While it's not a crime, it is extremely inefficient at a long term.
For the past 10 years news sites like "Anandtech", "Slashdot" and others have been primary source for news and buzz in the industry. With a group of professional editors, and "human-Internet-spiders" (E-Spiderman ?), these people crawled the net searching for information, stories and press releases.
What happened? I believe that most important changes are:

SPEED of information.
For an IT guy like me, it was about ~20 news / day on my favorite sites, all taking about 30 minutes to read... But I have to admit, that the next say 3 hours were spent on visiting a sequence of speciality websites, manufacturers, developers and corporate sites. Slashdot does a great job, so does Gadgeteer and others, but it was simply not enough. I remember those geeky news readers, which given a list of urls, grabbed HTML and ran it through regexp parser (if you don't know what I mean, that's good :) I even had this cute little tools that read my Netscape bookmarks, checked page modification times and coloured them, so I knew which pages have been updated.
Here comes syndication - they take from us, those take from them and these here keep leeching from them and everyone has it :)
There is also a natural trend of changing publishing scheme from pull to push - it is developer or manufacturer's job to update major news sites and syndication points. Publisher are having a good time now.
44 new feeds have appeared in my Google Reader as I write this paragraph...
AMOUNT of information
Hey - I remember 1996 - I remember the "far end of Internet".... I remember the listing of websites in Poland fit on one page! I remember that time spend on one website was under a minute, not because I was bored, but because there was little more to see. Number of RSS feeds has passed 5 million (try this and this). Editors publishing for my favorite sites have to pick only the best ones out of the crowd, but it is well known, that the pieces create the whole. There is number of blogs and tech resources you wouldn't ever know about, but these can sometimes give you a unique perspective.