Monday, September 12, 2011

Phantom Disaster Recovery Site at Long Island City

In 1997, after NYCERS received independent budgeting authority, we began developing a computer disaster recovery plan. In conjunction with IBM and its Sterling Forest site we constructed a process where we were able restore our computer systems at Sterling Forest along with our key operations. This plan only supported about 50 operations employees.

In February, 2000, we moved NYCERS from 220 Church St. to 335 Adams Street in Brooklyn. This new office site had 3 general office floors (21 to 23) and a mezzanine site where we located our customer service center, our call center, and medical board division for handling disabilities.

After of 9/11, everyone in the New York metropolitan area became very aware of the potential for wide spread disasters.

In early 2004, NYCERS experienced two minor business interruptions at our new office site. One was a broken sprinkler pipe on the 24th floor and the other was a fire in an electrical closet on the 8th floor. The water damage was minor but the fire closed the 3 main office floors for a week. We were lucky that the mezzanine still had power. We had installed a second data line from Sterling Forest to the mezzanine along with the main data line on the 23th floor. We also had installed a backup PBX in the mezzanine which allowed the call center to operate after two days.

While the recovery was a great success, it was clear to me that any long term outage would be unacceptable with respect to agency’s work load and the management our staff of 370 employees.

I proposed to the trustees as part of the FY-2005 budget package that NYCERS should explore a possible alternate work site in the event of a local disaster at 335 Adams Street. We would be looking for site within the city, on a different power grid, with decent mass transit access, capable of housing at least 250 workers at one time, capable of being linked up to Sterling Forest, 335 Adams Street, and other city data centers, and finally not too expensive.

The site would have to have the potential to be fully outfitted but we would start with only a minimal kernel for immediate operations. This would include a full technology infrastructure. We would slowly install non-critical equipment over time. In the event of a disaster, we would expedite that process. The site would have the added advantage of allowing us more frequent and realistic disaster testing.

I was looking for a 5 year lease with an option to renew. I was hoping the city would have developed a local DR site for general city agencies within that time frame. That was wishful thinking. I had told the trustees that after we had identified a site, I would present them with a total cost proposal and plan for their final approval.

The Deputy Director of Administration, Niki Browne, was in charge of the search. Late in 2004 Ms. Browne identified a potential site in Long Island City. It was, however, warehouse space and had many issues to be resolved. Unfortunately in June of 2005, Ms. Browne was removed from the project and it was turned over to Karen Mazza. Needless to say, this project is above her pay grade. In fact, Mazza, a lawyer, has serious problems with filling out time sheets, handling her emails, and analyzing legislation.

In October, 2005, Martha Stark and the trustees appointed Diane D’Alessandro as the new executive director of NYCERS.

On of May 31, 2006, D’Alessandro signed a lease with 30TH Place Holdings, LLC for space on the 10th floor of The Factory Building, 30-00 47th Avenue, Long Island City, NY (LIC). This was the site that Ms. Browne had originally identified. The term of the lease was ten years with an option to renew for 5 years. The dimensions of the space were not specified in the lease. NYCERS began paying rent circa 2/1/2007 ($141,905 for FY-2007).

The annual rents for the ten years are listed at the end of this posting. These amounts do not include taxes, HLtP, or water charges.

In her February 25, 2011 letter to the trustees presenting the FY-2012 budget, D’Alessandro notified the trustees that the development of the disaster recovery facility was continuing. She also stated that the LIC site will have the capacity to house two shifts of 150 people. NYCERS currently has 396 employees and a large number of consultants.

I have recently commented on the fact that as of this past summer basic construction of the LIC site had not yet been started. This is over five years since the lease was signed.

D’Alessandro makes no mention to the trustees of this enormous delay in getting the site operational. She also does not mention the fact that, even if the site is made operational, it will not be able to handle the full current workforce. This is beginning to fit the CityTime pattern as far as the loss of time and money. NYCERS has budgeted over $8.7M from FY-2007 to FY-2012 for the LIC site.

NYCERS actually spent over $2.2M between FY02007 and FY-2009 on the LIC site. As of FY-2010, NYCERS stopped publicly reporting the ongoing rent payments and the associated costs for the LIC site.

This lack of disclosure is not unique to the LIC site. It applies to all NYCERS paid administrative expenses. The trustees no longer have any way of checking on actual spending at NYCERS. You can be sure the Comptroller is not auditing the financial controls at NYCERS. He has enough trouble managing the investment expenses for the pension funds. And, I wouldn’t count on the outside accounting firm either.

My professional opinion is this site is seriously flawed and will never be able to function at any acceptable level as a business disaster recovery site for NYCERS. I suspect that at the time of the lease signing NYCERS had not solved all the operational problems connected to the site and that some of those problems are intractable. Let’s hope NYCERS has the good sense not to extend the lease in 2016.

Annual rents for each year of the ten year lease for 30-00 47th Ave, LIC, NY:

  1. $362,076
  2. $370,222
  3. $378,552
  4. $387,070
  5. $479,335
  6. $490,120
  7. $501,148
  8. $512,423
  9. $523,953
  10. $535,742

No comments: