What Just Happened? Why is Everything so Slow?

Keeping the house clean sure can put a damper on all of the other work we all need to get done every day.  The same can be said in computer environments.  The tools we use to keep our PCs and networks clean from foreign programs and network traffic can have a dramatic impact on overall performance.  This exact scenario played out recently here at Escape.

In an effort to protect ourselves and our customers from malicious attacks, Escape implements a strict network security and antivirus policy.  Recently a major update to our anti-virus software had an extremely negative impact on our Escape Online 5 software product.  The scanning engine of the new anti-virus software was scanning all of the data as it was transmitted between the application servers and the clients.  This resulted in a 200% decrease in performance.  Once the culprit was identified as the system security software, action was taken to eliminate the issue.

The first step is to add the Online5.exe to the safe processes exclusion list in your security/anti-virus software.  This is normally done from a centralized security management server and pushed/pulled down to the clients.  Second, you will want to add the following processes that run on the application servers themselves:

EscapeTech.CrystalWindowsService.exe

EscapeTech.Online5MgrSvc.exe

EscapeTech.OnlineCoreService.exe

EscapeTech.ServerServices.ServiceGroupManagerHost.exe

Online5.exe (for the client application)

The Escape Online 5 application (both server and client) only has the ability to pass information to and from itself so there is no real risk of the application contracting and/or passing along viruses to other systems.  The exception to this is if you upload an infected file from your PC to a server via the client.  This being said, if you don’t have system security software installed on all of your systems you really should consider it.

*It is very important to not only add these files to the file exceptions list, but also to the process exceptions list.  This will ensure that work is being sent and received over the network and locally on the system without the interference of antivirus software scanning.

Read Full Post »

Online5Server – The First Step to Easier Upgrades

A re-occurring question we hear from the IT folks out there is, “Can’t we make the upgrade and patch process any easier?  We have multiple application servers to deal with.”  We hear ya!

Mike and I are always trying to think of new ways to make everyone’s lives a little easier (including our own), so we’ve been throwing around ideas to help make the application server upgrade and patch process more smooth and simple.  I don’t want to get you too excited, but we’ve come up with a plan to help you out.  The first phase of this plan is to get everyone on the same System Name.  The System Name is derived from the path to the software on each of the application servers.  In the past, the System Name was normally the customer name.  In recent years, we’ve used the same System Name for new implementations, so phase 1 will be to get all Escape Online 5 application servers on the same nomenclature; Online5Server.  Phase 1 will only affect our friends that have been live on Escape Online 5 earlier than 2010.  Phase 2 will include adding registry entries to track Centralized Storage and software drive installation locations.  I will be contacting some of you in the coming weeks to discuss what needs to be done, so don’t go anywhere.

Image courtesy of jscreationzs/FreeDigitalPhotos.net

Read Full Post »

Streamlining document management with Escape & docStar

Aside from providing technical support to customers and Escape Customer Care, I sometimes find myself on various special projects that come my way.  I am in the final phases of a project that I’ve been working on for Modoc County Office of Education and wanted to share some of the cool things we’ve accomplished.  This is actually my second project in which we have integrated Escape Online5 with docStar; the first project was with Tehama County Department of Education.  On both projects, I have worked closely with Jeff White at Coastal Business Systems, and both times it has been a pleasure working with him.

Here’s how it works:  each time the County staff creates a payment batch within the Escape Online5 system, the data for that batch is automatically transferred to the County’s docStar system where it is then parsed and matched to files that the County has scanned into the docStar system.  This alleviates the manual work of hand entering data into docStar, allowing resources to be allocated to other tasks.  The data that is getting input into docStar is more accurate since we have removed the chance of human error/mistyped keys.

Jeff tells me that people like the fact that when they have an audit, they can retrieve everything they need by a funding source code.  Auditors also have access to files remotely instead of having to be onsite.

I recently reached out to Jonna Turek, Business Systems Support Specialist at Tehama County Department of Education, to find out what benefits they have experienced since integrating Escape Online5 and docStar systems.  Jonna tells me that everything she hears from her districts is that “everyone loves it!” and they would never want to go back to the old days of manually scanning and indexing documents.  One district CBO just recently mentioned to her that they have saved so much time in the indexing process by having the key words from AP payments imported to docStar from Escape on a nightly basis.

“The indexing has become merely a matter of scanning the backup documentation into docStar, entering the check number and all the other relevant reference numbers are attached to the Payment document automatically. Data entry is one field, the check number, for each payment, as opposed to multiple fields.  So, we benefit in time saved, timeliness of the indexing and, with fewer manually entered fields, fewer errors.  It’s win-win all the way.”

It’s always good to hear how our hard work has paid off and is making things easier for others.  If you have any questions, let me know.  I’m sure Jeff wouldn’t mind discussing a future project with you either.

Read Full Post »

SQL Tips – Finding space in a pinch with sp_cycle_errorlog

I can’t tell you off the top of my head how many times I’ve received customer calls reporting Online5 errors which end up being caused by lack of space on the SQL Server’s C drive.  Luckily, we document incidents here at Escape Technology and it looks like I’ve had 6 occurrences in the past 2 years regarding errors due to drive space.  Often times there is not even enough drive space to open SQL Server Management Studio or view a Log file.

When I do need to free up space on a SQL Server, I check the usual suspects first:

• Old backup files that can be deleted and/or moved to another location
• Old stack dumps, crash dumps, mini dumps in the \LOG directory that can be deleted
• Large files saved to User Desktops or Document folders that can be removed
• The Recycle Bin can also provide a goldmine of large files hogging up disk space; empty it to free up some space.

If you have exhausted your search for files to delete and are still low on drive space, the next step you might consider would be to reboot the SQL Server and perform some emergency maintenance while you have a couple spare megabytes available before SQL eats them up again.  A SQL Server reboot will free up some drive space, but it will also create a group of angry users who cannot access the system for several minutes while services come back up.  Before you reboot your server, take a look at cycling the SQL Error logs.

It is not uncommon to find SQL Error Logs that are over 2GB in size, and when you have 5 or more logs, they can fill a drive very quickly.   I have found the sp_cycle_errorlog command very helpful to free up some space so that I can perform other emergency maintenance.  Executing the sp_cycle_errorlog command closes the current error log file and cycles the error log extension numbers just like a server restart.  Every time SQL Server is started, the current error log is renamed to errorlog.1; errorlog.1 becomes errorlog.2, errorlog.2 becomes errorlog.3, and so on.  sp_cycle_errorlog enables you to cycle the error log files without stopping and starting the server.

Now that you have some breathing room, try freeing up some space on the C drive when you are able to stop SQL Services.

1. Move data\log files to another drive.  NOTE: It is best practice to put Data and Log files on separate physical drives to optimize the I/O performance.
2. Place SQL Error Logs on another drive.

The following Microsoft article also provides Storage Top 10 Best Practices.

If you have any handy hints, please let us know!  We’d love to hear from you.

Read Full Post »

XCOE’s Upgrade in the Cloud

In an effort to help facilitate the needs for small counties, Escape Technology has offered a cloud solution called XCOE.  This project began in mid-2008 and has been going strong ever since.  There have been a few minor upgrades to various hardware elements over the years but for the most part the systems have remained the same.

Recently, however, Escape evaluated the systems and network for XCOE and decided to overhaul the entire design.  The first step was to replace the database server infrastructure.  We replaced the database server with the latest and greatest hardware which included massive amounts of processing power, memory, and solid-state disk drives.  In addition to the hardware upgrade, we also upgraded the operating system and SQL Server versions to the latest software available.

While this was a good first step, Escape also decided to replace all of the application servers with much more powerful systems.  The processing power of the servers was more than doubled and system memory was quadrupled.  A more fault tolerant design was put into place to help ease the possibility of failure and system downtime.  All of the server operating systems were reinstalled with fresh software and configured for optimal performance for our customers.

In addition to all of this, Escape replaced all of the networking equipment and installed a state-of-the-art SAN solution to house all of the mission critical data for our customers.  By implementing this solution, Escape Technology now has the ability to bring on new customers without having a negative impact on existing systems.  New servers, databases, and document stores can be created on the fly in a fraction of the time it normally takes.  All of these changes were a major undertaking, but it was worth it as our customers and support staff will certainly feel the benefits of all the changes.

Read Full Post »

An Eye on the Future and a Foot in the Past

With ongoing software development, there are always new technologies to evaluate and test.  This is no different for Escape Technology.  As new operating systems and database versions come to market, it is important for us to leverage the need to take advantage of new features, as well as evaluate the implications of this with our existing customers.  This is no more apparent than with the database infrastructure we use.

For many years Escape utilized Microsoft SQL 2000, which was a very solid platform for our initial implementation of Escape Online4 and Online5.  As time progressed Microsoft released SQL 2005, which had some exciting new features and was a much more stable platform for us and our customers.  We began testing SQL 2005 very early on and as we felt comfortable with it we began doing all of our development on this system.  We took advantage of new technologies and processes (such as the Service Broker) that were only available in SQL 2005.   This required all of our current customers to migrate to the newer version.  As new customers came online with our software this was the baseline for their infrastructure.

Time moves on and so does the development cycle at Microsoft.  Since moving to SQL 2005 three newer versions of Microsoft SQL Server have been released (Microsoft SQL Server 2008, 2008 R2, and now Microsoft SQL Server 2012.)  Here at Escape we have also moved on, but with careful considerations.  Our primary development platform currently is SQL 2008 R2, but the one major difference from our move from SQL 2000 to SQL 2005 is that we have been careful not to introduce any new SQL 2008 R2 only functionality that required our customers to migrate to this newer version of SQL.

We will continue to evaluate new versions of SQL and the functionality therein, always keeping our customer’s best interests at hand.  Please always refer to the System Requirements on our website to see what we currently recommend our customers implement.  If changes are made we will keep our customers informed and give them plenty of time to make changes as necessary.  As a side note, it is a good idea to purchase Software Assurance which will allow you to carry over the current licensing model to newer versions of software as it becomes available.

Read Full Post »

Windows Tools to Help Us Help You

Having been in software support for many moons, it is safe to say that the hardest part of a support rep’s job is understanding the issue that a user is experiencing and determining if that issue is caused by the software, setup, or the user.  The best way to get to the bottom of an issue is to see it in action and try to reproduce it.  After we have a better understanding of the steps involved to reproduce the behavior in the software, we can then perform our sleuthing magic and help users resolve the problem so they can continue on with their daily lives.

There are several tools built in to Windows that are available to users that can help us understand the issue.  The first tool, one which I personally rely on daily, is the Snipping Tool (Windows Vista and 7).  I’m sure you’re all aware of this fantastic feature, but there’s nothing better than being able to select any portion of your Windows desktop and simply copy/paste that image into an email or document.  A high-quality error message can really get us pointed in the right direction on our road to resolution.  It is also very helpful when providing step by step instructions in which users will need to follow; “Click this button, highlight this row…”  Windows XP users will need to rely on 3^rd party Snipping-Tool addons, or the old [Alt]&[PrntScn] option which screen caps the entire window in focus.

Pro Tip: If you want to use the Snipping Tool to take a screen shot of a drop down menu:

1. Open the Snipping Tool and click ESC to hide the white over-layer.
2. Open the menu you want to capture.
3. Press [CTRL]&[PrntScn].
4. Click the arrow next to the New button, select a snip type from the menu, and then use your mouse or tablet to capture the menu screenshot.

Drop down menus will normally close when you focus on another window…like the Snipping Tool, so this can be helpful in bypassing this.

The 2^nd tool that can be very helpful is the Problem Steps Recorder – type “prs” into your start menu to locate the prs.exe (Win7 only). This tool will take a continuous run of screen grabs as you navigate around on your computer and is excellent for when your ECC rep asks “What steps did you perform to get this software behavior?”  Simply start up Problem Steps Recorder, launch Online5, and do what you did to recreate the issue.  Each mouse click will be recorded and a description and screenshot of each of the user’s actions is created.  Click Stop and you will be asked to save the output file in a .ZIP folder automatically.  Although this is a great tool, each mouse click creates an image of your screen, so the final output file can get very large.  In a test I ran, a 6-click PSR file was 1MB; not extremely large, but large enough to cause issues with email filters or attachment size limits.

Below are links to these nifty tools that I hope will help us all in future troubleshooting tasks.

Snipping Tool

Problem Steps Recorder

If you have any handy tools that you can’t live without, leave a comment below.  I’m sure we’d all like to hear about them!

Read Full Post »

Get Those Checksums in Check

As computer components and software increasingly get better and faster, it just makes sense to virtualize some of the internal infrastructure of your business.  The ability to run ten separate servers on a single physical box is just too good to pass up, right?  It is if you put the right amount of planning and resources into the project.  We have seen many customers move to a virtual environment over the past few years and we expect more to do so in the future.

In an effort to help increase the performance of the Escape Online5 application servers running in virtual environments, Escape is always listening to our customers and identifying known issues which can affect the system.   Recently, we have come across a very common issue among Windows servers running on VMware VSphere 4.0 and 4.1.  This issue involves the offloading of tasks to the network interface card.  While this may be a good thing on physical network adapters, it may not be so good on virtual servers that have virtual NICs bridged to real physical NICs.

Escape used Wireshark (a well known packet monitoring tool) to monitor the network interface on many of our virtualized application servers running on VSphere 4.1 and Windows 2008 and Windows 2008 R2.  We noticed that within a three minute period, there were over 4000 checksum errors on most of these systems.  Checksum errors occur when the packets sent over the network don’t match from one end to the other.  This was shocking, especially since the systems we monitored were not very busy with client requests.  After some investigation, it was discovered that VMware is aware of this issue and has a fix noted in the following KB article. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006619

This article describes how to disable the Task Offloading on the TCP network interface.  Once this is done, most, if not all, checksum errors should be eliminated.

Read Full Post »

SQL T.L.C. – Defrag your database for a better experience

In recent weeks, I’ve received a handful of calls regarding diminishing performance on various reports.  I was unable to reproduce the issues internally, and I could not find fault with the stored procedures themselves, so my conclusion was data-related.  After scrambling the data and hosting the databases internally, I was still unable to reproduce the issues.  After some thought, I realized that our scramble process also includes re-indexing the data!  A ‘quick’ scramble without re-indexing allowed me to reproduce the slow reports.

Similar to your workstation’s hard drive, fragmentation occurs on SQL Server tables as data is inserted, deleted, or updated.  As the indexes are modified, the information is scattered across the data files, and the logical ordering of the data no longer matches the physical ordering; this can lead to a decline in query performance.  Again, like your workstation’s hard drive, it is a good practice to defrag the data to keep things running in tip-top performance.

There are 3 ways to do this:

1. Use the maintenance plan wizard in SQL Server Management Studio – Microsoft includes a simple wizard to automate this process, but it has its limitations; it rebuilds every index regardless of fragmentation level, some editions can’t rebuild indexes online and holds locks on tables, etc.
2. Write your own T-SQL scripts to do more custom logic – Using T-SQL commands gives you the flexibility to see which indexes are fragmented and only fix those tables.
3. Buy a third party index maintenance program

I prefer to use T-SQL to identify the level of fragmentation for each table and either rebuild the index, or simply reorganize them.  Reorganizing an index defragments the data by reordering the pages to match the logical order (left to right) which improves index-scanning performance, releases additional available disk space, uses minimal system resources and does not hold long-term blocking locks.  Rebuilding an index drops the index and creates a new one, removing all fragmentation, reclaiming disk space, and the index rows are reordered in contiguous pages.

A good rule of thumb is to reorganize indexes that are 5-30% fragmented, and rebuild indexes that are over 30% fragmented.  The following website provides good information regarding Reorganizing and Rebuilding Indexes, including the advantages and disadvantages of each method.  The fragmentation of your database should be checked weekly and, if needed, rebuilt/reorganized at a time when users will not be accessing the system.

Read Full Post »

Power to the People

For the past year, we here at Escape Technology have been working hard on making the Escape Online 5 upgrade process much easier.  With the help of Installshield, we have been able to cut the upgrade process time in half (or less) with a couple setup.exe files which do most of the work for you.  You simply need to know the location of your Escape Online 5 application software and your Documents folder (Centralized Storage) and you’re good to go.

I have been using the upgrade installers for the past 5 versions, making changes and modifications to help provide a great product.  Several Counties have been using them as well, and the feedback we have received so far has been very positive.  We now feel comfortable distributing the installers to the rest of you that have been requesting an easier method of upgrading your Escape Online 5 systems.

Ask no more my friends.  Starting with the v3.12.01 software release in March ’12, you too can take ownership of your Escape Online 5 system as we will be providing the Upgrade Installers via a secure login on the Escape Technology website.  No longer must you coordinate and schedule with us as to when you can have your system upgraded.

As we get closer to the v3.12.01 release date, we will provide the details and location of the installers.

Read Full Post »