Lessons from a 20 year-old web app

After I read "Why billing systems are a nightmare for engineers", I thought it would be fun to write about my own experience. Specifically lessons I've learned from building and supporting a 20 year old (and counting) "ERP" web app. But first, let me explain a bit how this came to be.

The web app is a general ledger, project management, inventory management, HR, and general "ERP" system (including billing, invoicing, foreign currency purchases, etc.) for 20-30 users at a small interior finishing supplies company which my grandfather founded. It has been used every day for at least 20 years.

Building iteratively for years

My Dad decided to give me a chance at building software for the family business before I was out of high school. I had shown an aptitude for programming and was making good money with friends building local websites. He built a COBOL system for the family business when he was my age, so the legacy was passed on to me (which I am very grateful for). My Dad made it a bit easier for me and bought a GUI code generator that would create CRUD pages in PHP using a MySQL backend. The LAMP (Linux Apache PHP and MySQL) stack was all the rage back then. My Dad's role was to design the relational database schema while I made pages (mixed PHP and HTML) to create, update, and delete entities from the database.

We started small with a simple project management app. We trialed it with a few users in the company, got feedback, and kept going. One summer I setup a door desk in the office. Literally a door on top of two filing cabinets. Unlike the Amazon story, I did not have braces or legs, just filing cabinets. I had an old Toshiba laptop and spend hundreds of hours for at least 2 months build the core of the application.

Our goal that summer was to replace all the paid software at the company, apart from operating systems, and Microsoft Office. It was cheaper to pay me by the hour, after all, than ongoing license costs for the alternative ERP systems everyone hated. By February 2003, we had fully transitioned to our own makeshift LAMP stack with 7 sub-systems. While I still run some occasional maintenance, the system has been stable for years requiring no oversight. I feel lucky for the chance to build it, but remain constantly paranoid about it's up-time, security, and potential for catastrophic failure.

App Stats:

- 2 developers (me & my Dad)

- Running on a single m3.xlarge instance in AWS (with a failover), after it ran on-prem for a decade

- 19GB used on disk (code, DB, and jobs)

- 15 cron jobs

-  2,543 PHP files

- 3.5 million lines of PHP (find . -name '*.php' -print0 | wc -w --files0-from=-)

- 1.9GB MySQL storage (du -sh /var/lib/mysql)

- 9 MySQL DBs, largest table has 1.5M rows - not that big!

- 23 active users this week (cat access_log* | grep "/login*" | awk '{print $1}' | sort | uniq | wc -l)


Lessons Learned

- Worse is better. The app feels like a giant hack and always has. But its beautiful in its own insane way. It grew as needed and that was better than something built really well launched in big version releases. I cut my teeth on it and am grateful to the poor users who were forced to put up with my bugs -- I know now how much of a privilege that was.

- No libraries, just global functions. This is bad advice, but it's served me well as a lone developer over the years. I have a single globals.php script that every application in the system uses, and for the main sub-system (called "Distribution Master") I have a single functions.php script. Only the navigation elements are shared.

- Create single purpose features. Whenever a new feature was requested, I would create a new PHP script. A few pages -- like the Invoicing routine -- I would version & keep multiple copies around. This makes finding anything in the system very easy. Whenever something would break & I would ask the user to report it -- they could give me a URL and I'd instantly know which file to go and fix.

- Use cron and stop worrying. It took me a few years to realize this, but time-based automation is easy to grok, easy to debug, and easy to explain to others. More complex triggering automation based on events sounds better, but crontab -l is a godsend when I've forgotten how everything works in the background system routines. I don't even bother with daemons anymore.

- Test in production. Heck I've edited code live on the production server a few times. The production deployment is tied to our Git repo so I can pull and push from our main branch. I know, it's insane and kind of stupid! But I didn't have time to properly test in another environment and then deploy. Since it's just been me -- I know how to manage risks of breaking something, undoing it. Plus having single-purpose-pages means I can create a copy of a script, test it in prod, and relink when it's working.


Final Thoughts

The system never became the foundation of a startup (which we dreamed of at one point). We may have missed an opportunity. However, it was the bread & butter of a business that put food on the table for dozens of people for decades. If you want to fast, go alone; if you want to go far, go together. The system won't make it another 20 years, but it surprises me how simple, "hacky", not-following-the-trends gave me opportunity and worked so well for years.

Comments

Post a Comment

Popular posts from this blog

Fluent NHibernate - Incorrect syntax near the keyword 'Group'

I have to share this. I spent a good amount of time searching around Fluent NHibernate for a way to escape columns in the generated SQL while still using AutoMapping. My initial configuration looked like: Fluently.Configure() .Database(MsSqlConfiguration.MsSql2005.ConnectionString( c => c.Is(ConfigurationManager.ConnectionStrings["Something"].ConnectionString))) .Mappings(x => x.AutoMappings.Add(AutoPersistenceModel.MapEntitiesFromAssemblyOf () .WithSetup(convention => { convention.FindIdentity = p => p.Name == p.DeclaringType.Name + "Id"; convention.GetComponentColumnPrefix = type => type.Name + "Id"; } ) .BuildSessionFactory() This works great, but I had a property in my class (SomeType) that is called Group. This was causing the SqlException: Incorrect syntax near the keyword 'Group' (others would be like: Incorrect syntax near the keyword 'User' or Incorrect syntax near the keyword ...
Read more

Fluent NHibernate + Encrypting Values

I recently created two CLR User Defined Functions for SQL Server 2005 (written in C#) to decrypt & encrypt a given string (based on a hard-coded key). select dbo.EncryptString('Testing') -- Returns: Fjfds243qcm43fsdj2343== select dbo.DecryptString('Fjfds243qcm43fsdj2343==') -- Returns: Testing Pretty simple stuff. This will allow the user with enough access to pull encrypted data out of the database and provide the ability to restrict decryption to other SQL users. Now to implement this in code... Because it's awesome, Fluent NHibernate has a method called FormulasIs that assists in building out values using SQL. public class User { public virtual int Id { get; set; } public virtual DateTime DateOfBirth { get; set; } } public sealed class UserMap : ClassMap { public UserMap() { WithTable("dbo.[User]"); Id(x => x.Id, "[ID]"); Map(x => x.DateOfBirth, "DOB").FormulaIs("dbo.DecryptString(DOB)"); ...
Read more

MapReduce in C# using Task Parallel Library

Back in August I starting playing with a C# implementation of Google's MapReduce algorithm. The implementation was based on something Stephan Brenner did, although I completely refactored it. Today I added a little bit of logic to split up the actual execution of Map & Reduce in this implementation using the Task Parallel Library in .NET 4.0. Check out the source code for MapReduce in C# on GitHub. Below is an excerpt from the Tests on how to implement the library. Counting Words in Files public static List<KeyValuePair<string, int>> Map(FileInfo document, string text) { var items = text.Split('\n', ' ', '.', ',','\r'); return items.Select(item => new KeyValuePair<string, int>(item, 1)).ToList(); } public static List<int> Reduce(string word, List<int> wordCounts) { if (wordCounts == null) return null; var result = new List<int> { 0 }; foreach (var value in wordC...
Read more