Good vs. Bad & Ugly: Comprehensive vs. General Design

Let's assume a simple categorisation of business into trade and manufacturing.
Most of the companies that are active in a category, arrange and manage their processes - despite different fields of business- inspired by a similar and close to optimal pattern.  For example all trade companies share the same processes and concepts such as sales region, sales commission, petty cash and etc.  The pattern is called "best practice".

Good

Now, imagine that you have designed and implemented information software for 25 trade companies so far; naturally resulting in your extensive knowledge of trade firms' processes and requirements -put simply, best practice.  In other words, you do know what the customer wants and needs.

What happens if you decide to design a comprehensive information system for trade firms? As you know their requirements and best practices, supported by the trade business knowledge and understanding which you have already earned during last 25 implementations, you design a model that, with minimum complexity and maximum clarity, covers most of the a typical trade firm' processes. Not only your model but also your user interface are efficient and understandable by the customer.

This is what I call a "Comprehensive Design".

Obviously the model doesn't cover all processes in all trade firms but, those non-covered are a small fraction compared to what the model already covers.  In fact, your model embraces about 80 to 85% of a typical trade firm's processes which leaves you 15 to 20% customisation during an implementation.  The percentages I just mentioned are very famous in ERP world. 

Bad & Ugly

Now, imagine a totally different situation: you have designed and implemented information systems for 2 trade companies so far and all of a sudden you decide to design an information systems for trade firms in general. Let's see how your design is.

Clearly, the knowledge of a typical trade firm requirements and best practices which you've acquired during your 2 implementations -even if both were successful- is not extensive. So when designing you have take care of all the possible requirements -which you're not aware of yet- by designing facilities to easily change the behaviour of the model. This is the key characteristic of your design, happening all over your model since you don't know the real world's requirements and need to prepare for meeting them.

Actually, your model is a "General Design".

It doesn't have a skeleton -the extract of your experience and knowledge- rather it has lots of lots of features that help users customise all the aspects of it and all because you don't know what you'll be facing in a typical trade firm.
There is a design principle stating that "The design and usage complexity of a software is proportional to how much the software is generalised." In other words, the more general your design the more difficult its development and usage.

Because it is complex and doesn't have a real structure, the result is not a software that has a shape but such a flexible set of -sometimes inconsistent- customisation features that it hardly has any shape. Practically, you have to create a new software out of all the customisation features, In every implementation using the general design. 

Conclusion

• Comprehensive design is based on business knowledge, understanding the target market requirements and best practices and experience.
• Comprehensive design is easily grasped by users and it is easy to use.
• General design is based on a considerable technical knowledge, little business knowledge and incomplete understanding of the target market requirements and best practices.
• General design is complicated and not friendly from the user's perspective.

The following statement is a concise and precise summary:

"Do not generalise.  Generalisations are generally wrong." -Butler Lampson

Amailyser - The E-mail Analyser

Recently Ed Daniel (a friend of mine) needed to produce a time analysis report out of his e-mails and asked if I can help him.  Well, obviously I said yes.


I thought to myself, he needs charts and graphs and data querying from many aspects that I may not know.  So to me the best option is to load his e-mail important fields into an RDBMS (such as SQLite or PostgreSQL).  Later he can do whatever he wants with the data using reporting tools such as JasperReports and produce nice charts and graphs as he requires.


That was the motivation to write Amailyser (on GitHub).

• The language is Python 2.x; I could have done it in Perl but I'm learning Python and it seems like a wonderful practice to me. 

• For DB interaction I used SQLAlchemy a nice ORM mapper for Python.

• To read the e-mail messages I used Python standard library mailbox.

After downloading and extracting, you can run the Amailyzer (after you have modified config.py to suite your needs)  in a terminal:

$ cd amailyser/src

$ python main.py


The big picture is to read mailboxes, load each message, extract important fields and persist them in the database.

Below is the structure of the source:

Amailyser source tree

• config.py: This is where you can change the behaviour of Amailyser like which mailboxes to process and type of each of them.
• model.py: This is the model of database described in SQLAlchemy.
• workhorse.py: Contains two calsses MailBox and MailMessage.  MailBox opens a mailbox in mbox or maildir formats and processes each message inside using MailMessage which (optionaly) persists them to the database.
• main.py: Actually as a main is supposed to be, it doesn't anything special.  Just calls model setup routines and iteraties over items in the list of mail boxes, passing each one to MailBox.

To install SQLAlchemy please visit the tutorial on its website.